ArticlePDF Available

Standing on the Shoulders of Giants: Challenges and Recommendations of Literature Search in Information Systems Research


Abstract and Figures

The “standing on the shoulders of giants” metaphor is often used to acknowledge the work of others when undertaking research and, in particular, stresses the importance of literature reviews in scientific inquiry. Though the significance of literature reviews has never been in doubt, researchers, especially novice researchers, still struggle with developing effective strategies for reviewing literature. An important reason for this difficulty is the rapidly increasing number of potentially relevant publications—not all of which necessarily add value to a literature review. As such, avoiding standing on the shoulders of dwarfs literature search emerges as a major issue in crafting an effective literature review. In this paper, we discuss challenges of literature searches in the increasingly dynamic context of information systems (IS) research and make recommendations for how to deal with them. We present practical guidelines and a checklist to help researchers with planning and organizing their literature searches. Download:
Content may be subject to copyright.
www.uni.l i
This is the author’s version of a work that was published in the
following source:
vom Brocke, J., Simons, A., Riemer, K., Niehaves, B., Plattfaut,
R., & Cleven, A. (2015). Standing on the shoulders of giants:
Challenges and recommendations of literature search in
Information Systems research. Communications of the
Association for Information Systems, 37(1), Article 9, 205-224.
The final publication is available at
Notice: Copyright © 2012 by the Association for Information Systems. Permission to
make digital or hard copies of all or part of this work for personal or classroom use is
granted without fee provided that copies are not made or distributed for profit or
commercial advantage and that copies bear this notice and full citation on the first
page. Copyright for components of this work owned by others than the Association
for Information Systems must be honored. Abstracting with credit is permitted. To
copy otherwise, to republish, to post on servers, or to redistribute to lists requires
prior specific permission and/or fee. Request permission to publish from: AIS
Administrative Office, P.O. Box 2712 Atlanta, GA, 30301 -2712, Attn: Reprints; or via
e-mail from
Communications of the Association for Information Systems
Volume 37 Article 9
Standing on the Shoulders of Giants: Challenges
and Recommendations of Literature Search in
Information Systems Research
Jan vom Brocke
University of Liechtenstein,
Alexander Simons
University of Liechtenstein
Kai Riemer
e University of Sydney
Bjoern Niehaves
University of Siegen
Ralf Pla$fault
European Research Center for Information Systems
See next page for additional authors
Follow this and additional works at: h=p://
<is material is brought to you by the Journals at AIS Electronic Library (AISeL). It has been accepted for inclusion in Communications of the
Association for Information Systems by an authorized administrator of AIS Electronic Library (AISeL). For more information, please contact
Recommended Citation
vom Brocke, Jan; Simons, Alexander; Riemer, Kai; Niehaves, Bjoern; Pla=fault, Ralf; and Cleven, Anne (2015) "Standing on the
Shoulders of Giants: Challenges and Recommendations of Literature Search in Information Systems Research," Communications of the
Association for Information Systems: Vol. 37, Article 9.
Available at: h=p://
Standing on the Shoulders of Giants: Challenges and Recommendations
of Literature Search in Information Systems Research
Jan vom Brocke, Alexander Simons, Kai Riemer, Bjoern Niehaves, Ralf Pla=fault, and Anne Cleven
<is special section article is available in Communications of the Association for Information Systems: h=p://
ommunications of the
ssociation for nformation
Research Paper ISSN: 1529-3181
Volume 37 Paper 9 pp. 205 – 224 August 2015
Standing on the Shoulders of Giants: Challenges and
Recommendations of Literature Search in Information
Systems Research
Jan vom Brocke
University of Liechtenstein
Alexander Simons
University of Liechtenstein
Kai Riemer
The University of Sydney, Australia
Bjoern Niehaves
University of Siegen, Germany
Ralf Plattfaut
European Research Center for Information Systems
(ERCIS), Germany
Anne Cleven
A2 Research, Switzerland
The “standing on the shoulders of giants” metaphor is often used to acknowledge the work of others when
undertaking research and, in particular, stresses the importance of literature reviews in scientific inquiry. Though the
significance of literature reviews has never been in doubt, researchers, especially novice researchers, still struggle
with developing effective strategies for reviewing literature. An important reason for this difficulty is the rapidly
increasing number of potentially relevant publications—not all of which necessarily add value to a literature review. As
such, avoiding standing on the shoulders of dwarfs literature search emerges as a major issue in crafting an effective
literature review. In this paper, we discuss challenges of literature searches in the increasingly dynamic context o
information systems (IS) research and make recommendations for how to deal with them. We present practical
guidelines and a checklist to help researchers with planning and organizing their literature searches.
Keywords: Literature Review, Literature Search, Systematic Literature Review, Dissertation Review, Information
The manuscript was received 31/03/2014 and was with the authors 6 months for 2 revisions.
206 Standing on the Shoulders of Giants: Challenges and Recommendations of Literature Search in Information
Systems Research
Volume 37 Paper 9
1 Introduction
Literature reviews are essential in any type of research (Webster & Watson, 2002). They often lead to
ideas for new studies because they help to identify research problems and gaps and justify the relevance
and timeliness of addressing them. They serve as foundations and frameworks for research projects
because they help to develop an understanding of a domain and explain the topic under study. Moreover,
because they guide data collection and analysis, they also inform theory development and testing. Done
correctly, they can significantly improve the outcome of research projects.
Innovations in IT and the Internet have provided researchers with new ways to collect and organize
scholarly literature in crafting a literature review. The Internet and specialized online databases provide
ready access to more publications than ever before, and several software products for managing
bibliographies, citations, and references allow one to organize publications effectively. Based on these
developments, researchers should be much better equipped to conduct literature reviews today than they
were 20 years ago. At the same time, however, IT advances have made literature searches far more
complex. While the Internet allows fast and easy access to publications, it has also led to a mushrooming
of publication outlets. With only a few mouse clicks, researchers are able to retrieve more literature than
anyone would be able to read and evaluate in a lifetime. For example, a straightforward search for
“knowledge management” with Google Scholar produces more than 950,000 hits (Google Scholar, 2015).
Where researchers 20 years ago had to spend considerable time just to compile a reasonably
comprehensive bibliography, today they face the opposite problem of “literature overload”. The challenge
of performing literature reviews has shifted from collecting literature to interpreting it because our technical
capacities to publish, distribute, find, and retrieve research information have outgrown our individual ability
to read, evaluate, understand, and synthesize the resulting body of work.
As a result, today’s researchers are required to develop new approaches to conducting and reporting their
literature reviews, a task that inexperienced researchers in particular often find difficult, even
overwhelming (Boell & Cecez-Kecmanovic, 2014). This is particularly the case for the literature search
task, when the researcher must make several important decisions, including selecting databases and
journals, defining search terms, selecting criteria for including and excluding papers, and developing
strategies for citation analysis (Levy & Ellis, 2006). These and related decisions are neither
straightforward nor easy, especially in the context of IS research.
First, IS research is a highly diverse discipline that covers a wide variety of research themes (Benbasat &
Weber, 1996). Therefore, the search for IS studies typically requires one to use several databases and a
combination of search strategies, which makes the process of identifying and collecting literature all the
more complex and challenging. Second, IS research is a social science that deals with the phenomena of
IT design, adoption, and use in various contexts. Researchers have argued that social phenomena are
naturally tackled by different fields, which makes searches fragmented and challenging because they
have to venture across disciplinary boundaries (Helmericks, Nelsen, & Unnithan, 1991). Third, the number
of journals and conferences that publish IS research is rapidly increasing, as is the average length of
research papers (Peffers & Hui, 2003). Because the number of academic books published every year is
also growing quickly, IS researchers must deal with unprecedented amounts of research information in
their reviews. Fourth, IS research often deals with concepts that emerge and disappear quickly
(“buzzwords”) and topics that are constantly evolving (Bandara, Miskon, & Fielt, 2011), so identifying
related works is a special challenge in the IS field. Similar phenomena are often studied under different
headings, so there is a risk of overlooking relevant literature.
Given these challenges, IS research has not invested much effort into developing guidelines for
conducting literature reviews compared to the efforts that have gone into developing other research
methods (Wolfswinkel, Furtmueller, & Wilderom, 2013). This paper addresses this gap by 1) identifying
typical challenges that IS researchers are likely to experience in their literature searches and 2) providing
recommendations in the form of detailed guidelines on how to deal with such challenges.
The paper is structured as follows. In Section 2, we explain a literature review’s general nature and
characteristics. In Section 3, we outline several approaches to searching for literature and the activities
involved in this process. In Section 4, we address the challenges that typically confront IS researchers in
their literature searches, and, in Section 5, we presents recommendations on how to deal with these
challenges. In Section 6, we summarize the results in the form of a checklist, and, in Section 7, we
conclude the paper.
Communications of the Association for Information Systems 207
Volume 37 Paper 9
2 Literature Review: Outcome and Method
The term literature review can refer to both an outcome and a method (Boell & Cecez-Kecmanovic, 2014).
The outcome perspective refers to different kinds of literature reviews that serve different purposes, while
the method perspective refers to the ways in which one actually reviews the literature.
2.1 Outcome Perspective
From an outcome perspective, literature reviews most often 1) frame and provide background for a
research study, 2) review a field as a standalone paper, or 3) provide the review section for a research
thesis (Boell & Cecez-Kecmanovic, 2014; Okoli & Schabram, 2010).
Background reviews are usually presented as parts of primary empirical papers, such as surveys and
case studies (Okoli & Schabram, 2010). For example, background reviews typically inform the
development of research questions and hypotheses in quantitative, theory-testing studies, while they
provide an orienting framework for data collection and analysis in qualitative, theory-generating studies
(Creswell, 2009). In design-oriented studies that deal with creating and evaluating IT artifacts (Hevner,
March, Park, & Ram, 2004), background reviews help to justify the novelty of design ideas against related
works and to identify existing methods and models that can inform the design process1.
Standalone review papers are research publications that are not grounded in any original empirical data
but exclusively draw from extant theoretical or empirical studies. This type of literature review is usually
much longer than the background review—often longer even than primary empirical papers. Review
papers typically develop new theory, identify potentially interesting avenues for future research, close
research areas where enough has been done (Webster & Watson, 2002), and classify and categorize
extant research studies to analyze the progress in a field (Okoli & Schabram, 2010).
Researchers have argued that background literature reviews often fail to display the same rigor and
structure than the empirical parts of a research manuscript do because “authors have little motivation to
push themselves to higher standards for the theoretical background literature review” and peer reviewers
“rarely ask for any explanation of how the relevant articles in the review were identified, screened for
quality, or analyzed” (Okoli & Schabram, 2010, p. 6). In contrast, standalone review papers tend to be
much higher quality, which often makes them even more challenging to write compared with reports of
empirical research (MISQ, n.d.).
Review sections of research theses are located somewhat between background reviews and review
papers. They often have similar purposes as background reviews, but they are more like review papers
when it comes to rigor and length. On the one hand, they review the literature in a field or subfield to
provide a solid foundation for the research presented in the thesis and, therefore, strive for coverage. On
the other hand, a review section’s main purpose is to motivate the research study and establish and
demonstrate clearly a research gap that allows the research candidate to claim and justify the originality of
their work (Boell & Cecez-Kecmanovic, 2014).
2.2 Method Perspective
From a method perspective, two basic types of literature reviews are commonly distinguished: narrative
reviews and systematic reviews (Collins & Fauser, 2005).
A narrative review, sometimes referred to as a “conventional” review (Kitchenham & Charters, 2007),
typically draws from the reviewer’s experience in a field. With a long tradition in IS research, these reviews
are written from the vantage point of the author and, thus, provide a (sometimes critical) overview from the
expert’s point-of-view. Conversely, they have been criticized for not being comprehensive or balanced in
their inclusion of source material or objective in their viewpoints (Boell & Cecez-Kecmanovic, forthcoming;
Petticrew & Roberts, 2006).
A systematic literature review, on the other hand, takes a structured approach to identifying, evaluating,
and synthesizing research (Kitchenham, 2004; Kitchenham et al., 2009). Such reviews address a specific
research question that guides the data collection, extraction, and aggregation process (Turner,
1 The term “background review” might be misleading in that literature reviews can play diverse roles in empirical research studies. In
grounded theory studies, for example, literature reviews are often conducted after the theory generation process to engage with
existing theory (Glaser & Strauss, 1967).
208 Standing on the Shoulders of Giants: Challenges and Recommendations of Literature Search in Information
Systems Research
Volume 37 Paper 9
Kitchenham, Budgen, & Brereton, 2008). Systematic reviews have gained momentum in IS research for
offering an explicitly scripted and reproducible review process (e.g., Bandara et al., 2011; Levy & Ellis,
2006; Okoli & Schabram, 2010; Wolfswinkel et al., 2013). At the same time, systematic literature reviews
have been criticized for being mechanistic and too focused on identifying literature at the expense of
scholarship in performing the actual review (Boell & Cecez-Kecmanovic, 2014; Boell & Cecez-
Kecmanovic, forthcoming).
While some researchers use the systematic review notion to refer narrowly to a specific method that has
its roots in the cumulative knowledge-creation tradition of the medical disciplines (Boell & Cecez-
Kecmanovic, 2014), others use the notion more widely as a qualitative attribute and consider all reviews
as systematic to a certain extent (Okoli & Schabram, 2010). Between these views, this paper uses the
systematic review notion to refer to a class of literature review that is more structured in its approach than
conventional reviews but that does not advocate any specific method: if a literature review follows an
explicit method, regardless of the kind of method, we consider it a systematic literature review; if not, we
consider it a narrative literature review. This distinction is independent of the outcome, and we do not
compare these approaches or argue in favor of one or the other. However, the narrative review style is
more suited for experienced scholars who are able to synthesize the literature in a particular field based
on their experience and expertise, while less experienced researchers are likely to benefit from more
systematic, guided approaches.
Therefore, Fink’s (2010, p. 3) definition of a research literature review as “a systematic, explicit, and
reproducible method for identifying, evaluating, and synthesizing the existing body of completed and
recorded work produced by researchers, scholars, and practitioners” corresponds to what we understand
as a systematic literature review. This definition emphasizes the methodological part of literature reviews,
the three key activities of searching, appraising, and synthesizing the literature, and the variety of
potentially relevant publications with which researchers must deal in their reviews. It also suggests the
need to describe the review method in sufficient detail to make reviews more explicit and reproducible for
3 Searching the Literature: Role and Process
Literature reviews can serve many purposes. They help researchers to understand a domain, to uncover
gaps in research and develop an agenda for their research project, to justify research problems by
showing their importance, to learn about the evolution of ideas, to identify the research methods and
strategies that are common in an area, to identify authors who are active in a field, to synthesize existing
research, and to overcome conflicts in the literature (e.g., Hart, 1998; Helmericks et al., 1991; Levy & Ellis,
2006; Okoli & Schabram, 2010; Webster & Watson, 2002). The literature search contributes to achieving
all of these goals.
One can broadly divide the process of conducting literature reviews into the literature search, selection,
and synthesis phases. Each of these phases involves several detailed tasks, such as retrieving, reading,
mapping, classifying, and interpreting publications and presenting and writing up the findings (Boell &
Cecez-Kecmanovic, 2014; Fink, 2010; Levy & Ellis, 2006; Okoli & Schabram, 2010; Webster & Watson,
2002). Because many of these activities are intertwined, this tutorial focuses not only on the search
process but also on the two closely related activities of selecting and retrieving publications to review.
The literature search is a salient part of any literature review but particularly of systematic literature
reviews. The literature search refers to the process of collecting data in literature reviews, and, much like
collecting empirical data for surveys and case studies, it is important to thoroughly plan the procedures for
collecting data in the literature search. Failures in the activities related to the literature search (e.g.,
overlooking relevant publications, excluding relevant publications, and using flawed metadata in reference
management) can have a considerable impact on the resulting review’s quality.
Depending on whether the review process is organized sequentially or iteratively, the literature search’s
role will also be interpreted differently. The “sequential review approach” uses step-by-step processes,
with the literature search among the first of these steps. Literature reviews of this type are basically three-
phased processes of input (searching), processing (analyzing and synthesizing), and output (writing)
(Levy & Ellis, 2006). There are many descriptions of sequentially organized review processes, each with
its own focus and level of detail, but these models all expect the researcher to complete the literature
search (to a greater or lesser degree) before analyzing and synthesizing the literature.
Communications of the Association for Information Systems 209
Volume 37 Paper 9
Literature reviews can also undertake an iterative process; that is, the intertwined activities of searching
and reading literature. Boell and Cecez-Kecmanovic (2014) use hermeneutics to explain how the activities
of searching/acquiring and analyzing/interpreting research studies engage with each other in a literature
review. The “iterative review approach” expects researchers to start their reviews with a literature search,
to learn about new themes and concepts during the reading process, refine and extend the initial search
to collect additional related works, and repeat the process until the researcher is satisfied with what was
Which review and search process is better suited for a particular literature review depends largely on the
type of the review. Research review, research synthesis, literature analysis, meta-analysis, state-of-the-
art-analysis, bibliometric analysis, and related terms are often used synonymously, although some of them
carry a broader meaning than others. Perhaps most common is the distinction between the literature
synthesis and the meta-analysis (e.g., Baumeister & Leary, 1997): while a literature synthesis describes,
summarizes, and integrates academic literature conceptually (Webster & Watson, 2002), a meta-analysis
uses more formal, statistical techniques to combine and aggregate data from various sources (Fink,
2010). Also, meta-analyses in IS research often collect publication metadata (e.g., publication year and
outlet) and analyze and classify the literature in terms of the type of research performed (e.g., research
strategies and methods) (e.g., Poeppelbuss, Niehaves, Simons, & Becker, 2011; Wareham, Zheng, &
Straub, 2005).
No single approach to conducting a literature search for a specific type of review exists. However,
literature reviews such as meta-analyses that take a quantitative stance often apply sequential procedures
to collecting and analyzing the academic literature. In contrast, the iterative approach to searching and
interpreting literature tends to guide theoretical, concept-centric reviews and syntheses.
In summary, literature reviews can be either an outcome, such as with review papers and background
reviews, or a method. If the review method is emphasized and an explicit method (organized as either a
sequential or an iterative process) is applied, we refer to it as a systematic review in this paper; otherwise,
we call it a narrative review. Regardless of which process is chosen, the literature search is always an
important activity even though there is no single approach to doing it. In more quantitative reviews, such
as meta-analyses, the literature search is often a single activity at the outset of the review that is
completed before the analysis begins, while the concept-centric literature synthesis often requires the
researcher to search for literature several times and to engage with the literature in an iterative learning
4 Literature Search Challenges
Independent of the review type and method, several challenges can make literature reviews and searches
in IS research difficult. Scholars in disciplines other than IS have developed guidelines, but, as a multi-
paradigmatic and highly diverse discipline (Benbasat & Weber, 1996) that spans the boundaries of several
academic areas and faculties and one that is located at the intersection of the social and the technical
sciences, IS research poses unique challenges.
4.1 There is No One-Size-Fits-All Approach to Searching the Literature
Compared to other disciplines, IS researchers pursue differing research objectives in their studies, and
delivering answers to these questions requires similarly differing research designs. Differing research
designs, in turn, often involve differing forms of literature reviews, so the role and structure of the literature
search also differs significantly. Just as there is no single approach to writing a review paper (Schwarz,
Mehta, Johnson, & Chin, 2007), there is also not one approach for literature search.
For literature reviews, the focus of interest (e.g., research outcomes, research methods, or theories) and
associated goals (e.g., integration or criticism) vary, as do coverage (e.g., representative or selective) and
organization (e.g., conceptual or historical) (Cooper, 1988; Fettke, 2006). While there are only few
guidelines for literature reviews and searches in IS research, the ones that exist are not universally
applicable but must always be assessed for their applicability to a particular study because they rarely
meet specific research objectives. Compared to primary empirical research methods, common standards
for conducting literature reviews (and searches) are lacking, and the diverse nature of research studies in
IS makes developing individual strategies for searching the literature particularly difficult.
210 Standing on the Shoulders of Giants: Challenges and Recommendations of Literature Search in Information
Systems Research
Volume 37 Paper 9
4.2 The Number of Research Publications is Increasing Rapidly
IS research results are created and published at a higher rate than ever before (Peffers & Hui, 2003). The
ever-increasing number of books and book chapters, journal papers, conference and workshop papers,
and so on often prevent researchers from including all available sources in their literature reviews, even
though the goal of systematic reviews has traditionally been “to comprehensively identify, appraise, and
synthesize all the relevant studies on a given topic” (Petticrew & Roberts, 2006, p. 19; emphasis added).
Calls for comprehensive or exhaustive literature reviews cannot be answered in many of today’s reviews;
depending on the topic, much work has to be omitted from the review, which is why inexperienced
researchers in particular tend to worry about the scope of their literature reviews. They often hope to use
their literature reviews to understand and conceptualize the topic they plan to study, to find research
ideas, and to refine them into research problems. However, knowing that there is much more literature
available than they can ever include in their reviews, they often feel anxious and confused about the
originality of their ideas and the conceptual completeness of their research.
4.3 Literature Searches Often Lead to Unexpected Results, so it is Difficult to Plan
the Search
Conducting a literature review, no matter what type it is or which process it follows, requires the
researcher to plan the search. However, developing a search strategy is not straightforward, especially
not in IS research. Even strategies that are thoroughly thought out often do not work as expected. In
particular, the number of publications that a predefined set of search terms will produce can hardly be
estimated at the outset of a review. Certain search terms might sound promising in the beginning but, in
the end, might lead to a limited number of publications that do not justify doing a review or to an
overwhelming number of results that make a review virtually impossible. The number of retrieved
publications can be small when new types of technologies (“buzzwords”) are studied, which is not
uncommon in an IT-oriented discipline such as IS, and it can be large when it turns out that these new
technologies have already been already studied under different labels.
In addition, IS researchers have to collect their literature from various databases (Levy & Ellis, 2006),
which further complicates how one selects search terms. It is sometimes difficult to know whether search
terms are too broad or too narrow because they often do not work for all databases in the same way.
While some databases might deliver only a few publications for a given set of terms, others may produce
endless result lists. What’s more, search terms are only one out of several parameters to be considered in
a literature search, so it can be difficult to predict how many sources a search strategy will produce.
4.4 Online Databases Differ Regarding Coverage and Functionality, and They are
Highly Volatile
Using several databases for retrieving literature leads to another issue. Depending on the review’s scope
and objectives, it can be time-consuming to decide which databases to use when considering that IS
research currently has more than 700 active journals (Lamp, 2004). In addition, many of these journals
are not indexed in the most common databases. However, even if a database offers access to a specific
journal, it may not offer all of the journal’s volumes and issues (as a matter of coverage) or the opportunity
to browse and read the journal (as a matter of subscription). Moreover, in fast-moving fields such as IT
and IS, conferences typically play an important role in the research dissemination process. Still, most
databases only index journals, so any literature search that relies exclusively on databases (e.g., to
ensure replicability) can be implicitly biased against conferences and, thus, more recent ideas and novel
Databases also differ in terms of their functionality, which includes not only basic features such as search
operators (e.g., quotation marks) and search fields (e.g., publication types) but also the underlying search
algorithms, which are often not so obvious. For example, some databases allow full-text searches even if
one does not have a full-text subscription, while others do not. Even the use of stop words differs
significantly, and many keywords in IS research involve terms such as “IS” and “IT” that are often treated
as stop words. Therefore, using several databases in a literature search can require the researcher to
conduct the same search in more than one way, and, considering the high volatility of databases in terms
of their coverage and functionality and the many available subscription models, the same search can
produce different results if it is conducted at different points in time and/or place.
Communications of the Association for Information Systems 211
Volume 37 Paper 9
4.5 It can be Difficult to Evaluate the Quality and Relevance of Publications
Based on an analysis of 37 literature reviews including meta-analyses, Allen and Olkin (1999) calculate
that completing a systematic review takes an average of 1,139 hours (as cited in Petticrew & Roberts,
2006, p. 49). Since the number of publications that may be relevant to a review is usually high, it is easy to
find an enormous number of relevant publications in very little time via the Internet. However, one must
wonder about the usefulness of having hundreds of publications if there is not enough time to read. As
time typically becomes a serious issue in the later stages of the review, the up-front screening of
publications retrieved from the literature search is key to effective time management.
Screening the literature for applicability requires identifying publications that would not add value to the
review and excluding them from further analysis. Such screening is usually done by reading the
publications’ abstracts, which can save time in the later phases of the review that require thorough reading
of the full text. However, even the screening phase can be time-consuming, which is why searching and
screening the literature are deeply intertwined activities: search parameters that are too broad produce
exhaustive results, so selecting the literature for the review is likely to take considerable time, while search
parameters that are too narrow can save time but risk overlooking relevant literature.
However, IS research does not provide common guidelines for screening literature. The criteria that are
most often used are quality and practical relevance (Okoli & Schabram, 2010), but it can be difficult,
especially for inexperienced researchers, to assess the relevance and quality of research papers at the
outset of a study. What’s more, quality criteria from other disciplines (some of which prefer quantitative
over qualitative approaches, for example) are often not applicable to IS research due to its multi-
paradigmatic nature, and practical criteria are typically arbitrary (Okoli & Schabram, 2010).
4.6 The Retrieval and Storage of Research Literature is Often Messy and Time-
Not too long ago, researchers had to physically travel between libraries and archives to collect literature to
build their studies’ bibliographies. Today, the Internet allows researchers to search for and retrieve
publications quickly and easily. Even so, acquiring some publications can still be challenging. Since
database subscription costs can be expensive, a researcher’s local library might not provide access to all
of the sources that carry IS research, and, even if it does, it still can be difficult to access papers published
in less popular journals, papers presented at conferences or workshops, or books and book chapters free
of charge. Some universities offer a central point of access for the literature search, a “meta-search” or
“cross-search” (Boell & Cecez-Kecmanovic, 2014). While using such portals is convenient, it risks
overlooking relevant literature. Inexperienced researchers in particular are often unaware of their
university’s online libraries’ coverage.
Storing and managing retrieved publications has its pitfalls, too. Reference management software helps to
collect publication metadata quickly and to assign file names automatically, but metadata downloaded
from the Internet is often not standardized and can be inconsistent. As a result, retrieving and storing
literature for review might be both time consuming and lead to citation errors in later publications.
4.7 It is Not Easy to Tell When a Literature Search is Finished
A literature review can quickly become massively complex, and the same is true for the literature search.
While it has to stop at some point, it is not easy to tell exactly when the search is finished. IS researchers
typically strive for a certain level of saturation before they conclude their reviews (e.g., Helmericks et al.,
1991), but it is not easy to achieve or even to measure saturation, which depends on the review type. In a
review paper, saturation often means that including more publications would not reveal any new concepts,
ideas, or arguments that add value to the review (Levy & Ellis, 2006; Webster & Watson, 2002), while the
saturation level of a background review is often determined by the degree to which it establishes
confidence in the novelty and importance of the research problems, questions, and hypotheses (Boell &
Cecez-Kecmanovic, 2014).
A literature search usually reaches saturation if it does not produce any new citations that are of potential
relevance, or, in other words, if further searches either deliver only irrelevant publications or publications
that have been found already or those that do not add to what is known from the ones already found (Levy
& Ellis, 2006). However, it requires considerable knowledge and understanding of the subject matter to
determine whether the coverage of a literature search is satisfactory, which also depends on how
212 Standing on the Shoulders of Giants: Challenges and Recommendations of Literature Search in Information
Systems Research
Volume 37 Paper 9
“coverage” is understood. Since it is often the review’s objective to establish just such knowledge and
understanding of a domain, measuring saturation is especially challenging if the search and review
process is sequentially organized and does not follow an iterative process of searching, reading, and
4.8 Search Results Often Differ, so Teamwork can be Challenging
Although there are exceptions, such as dissertation reviews, most of the literature reviews in IS research
are crafted in teams because literature reviews require various methodological and subject-related skills
and take considerable time, which makes effectively dividing labor and coordinating work an important
reviewing skill. Since researchers’ readings and interpretations of research literature can vary, teamwork
is another challenge in compiling literature reviews (Okoli & Schabram, 2010; Turner et al., 2008).
During the literature search, reviewers might share the work and take responsibility for individual tasks
according to mutually agreed-on procedures and criteria to ensure consistent and replicable search
results. However, even a well-defined search strategy can come with collaboration issues that may
compromise the results of the literature search. For example, databases or publications are often
assigned to individual researchers to save time, but databases differ in functionality, so the reviewers
often have to adjust predefined search strategies. Some search fields may not be available and the
search strings may even have to be changed (e.g., regarding the use of stop words).
In addition, screening publications for relevance is necessarily a subjective task, especially if no
appropriate evaluation criteria are in place, so dividing the screening work can lead to inconsistent results.
Collaboration issues can also arise when reviewers conduct their searches from different locations (e.g.,
jurisdictions that are serviced under differing licensing conditions by publishers) or under different
subscription levels or if they manage the references differently or name and store the retrieved files
differently. Working on search engines with different coverage of the search space can be confusing for
team members and hamper any claims of coverage and the decision concerning when to stop the search
5 Recommendations for Conducting Effective Literature Searches
Although literature reviews come with several challenges, many of which are clearly IS related, the
literature review method has not received as much attention in IS research as other research methods
have (Wolfswinkel et al., 2013), and the few approaches that have received attention tend to focus on
activities other than the search process (Boell & Cecez-Kecmanovic, 2014). As Bandara et al. (2011, p. 1)
point out, “While the importance of literature studies in the IS discipline is well recognized, little attention
has been paid to the underlying structure and method of conducting effective literature reviews”. For their
part, Okoli and Schabram (2010, p. 1) observe “an abundance of guides to conducting such reviews in
other research fields, (but) none entirely meet the unique needs of IS researchers”. As a result, IS
researchers lack guidelines and often have difficulty searching the literature in a way that suits their
research objectives.
The following guidelines address the identified challenges to support IS researchers in planning and
conducting their literature searches.
5.1 Develop an Understanding of the Subject Matter at the Outset of the Search
Researchers often face a dilemma at the start of a literature review. In many cases, they conduct literature
reviews to develop an understanding of a subject matter or domain, especially at the outset of larger
research projects such as dissertations, but conducting the search requires the researcher to understand
the subject area to a degree, especially if the review process is organized sequentially. For example,
defining keywords, evaluating publications’ relevance, and assessing the literature search’s coverage
require solid domain knowledge.
While novice researchers especially face the task of developing a thorough foundation for the review
before it starts, examples of high-quality literature reviews in IS demonstrate this issue is a common one.
For instance, Leidner and Kayworth (2006) grounded their review of culture in IS research in non-
empirical manuscripts and leading books to broaden their understanding of the empirical studies in the
field, Melville, Kraemer, and Gurbaxani (2004) collected studies on IT and organizational performance
based on keywords from a definition of IT business value they developed at the outset of their review, and
Communications of the Association for Information Systems 213
Volume 37 Paper 9
Glass, Ramesh, and Vessey (2004) began analyzing the computing science literature by developing a
classification scheme that later guided their coding process.
These examples demonstrate that researchers can employ any of several strategies to build a foundation
for a review and to increase their familiarity with a subject area in a review’s early phases. Thus, novice
researchers will find it useful to begin a review by reading sources that are likely to provide topic
summaries and overviews of the key issues in the area they wish to research (Baker, 2000), such as
extant review papers in their own or adjacent areas (Boell & Cecez-Kecmanovic, 2014). Review papers
are among the most frequently cited papers in most fields, so finding them is not usually difficult, and most
databases allow researchers to search for publications of a specific type, including literature reviews.
Other sources that help a researcher engage with a topic are encyclopedias, textbooks, and edited books
if they are sufficiently current. These sources provide good introductions and list a field’s most important
publications. IS research does not yet have an online encyclopedia, but there are encyclopedias in book
form (e.g., Khosrow-Pour, 2014). Another good way to gain an overview of the current status of a field or
subject area is to follow the calls for papers for journals’ special issues and for conference tracks and
workshops. This approach is particularly helpful in identifying relevant search terms that cover one’s
subject area. Of course, recommendations from supervisors also help novice researchers understand a
subject before they start their reviews.
5.2 Justify the Literature Review and Literature Search
Students at the graduate and undergraduate levels in particular often lack a proper understanding of a
literature review’s role. As with any research, literature reviews do not present an end in themselves but
are conducted to serve a certain purpose (Okoli & Schabram, 2010). A literature review is both an
outcome and a method, and, from a method perspective, literature reviews are guided by research
problems, which provide their justification.
Research problems take diverse forms, and, while many of them require empirical modes of inquiry, some
require literature reviews. For example, developing new theory often requires one to conceptually
synthesize the literature, and research agendas can hardly be developed without engaging with existing
literature (Webster & Watson, 2002). Moreover, quantitative literature studies such as meta-analyses are
often conducted if large numbers of publications on a subject have made developing an orienting
framework necessary to categorize past studies and guide future research (Okoli & Schabram, 2010) or if
the quantitative aggregation of past results allows an increase in statistical power (Cohn & Becker, 2003).
For example, Ahuja (2002) explores the role of women in the IT profession based on an integrative
literature synthesis because a number of contemporary drivers had made it necessary to explore barriers
to promoting women in IT. Gerwin and Barrowman (2002) conducted a meta-analysis on the
characteristics of integrated product development based on a substantial critique of empirical studies in
the field. Wareham et al.’s (2005) meta-analysis explored the major domains, themes, and methods in the
e-commerce literature because the sheer number of e-commerce publications had made a survey of the
research field necessary. Each of these authors’ justifications for their reviews differed from the others,
and their literature search processes differed as well. While Gerwin and Barrowman (2002) grounded their
meta-analysis in a database search, a search in fifteen journals from a variety of disciplines, and a review
of the reference lists of papers retrieved from these searches, Wareham et al. (2005) searched in the
mainstream academic and professional journals in the IS area to track divergences between research and
Therefore, the reasons for carrying out a literature review directly affect the literature search, which is why
quantitative literature studies often have more references than qualitative, conceptual literature syntheses
do. For instance, Glass et al.’s (2004) analysis of the computing science literature covered 1,485
publications, and Wareham et al.’s (2005) meta-analysis on e-commerce research was grounded in 582
studies. In contrast, Eierman, Niederman, and Adams (1995) conducted a conceptual literature synthesis
to develop a theoretical framework of decision support systems research based on a review of only 15
carefully selected papers, and Pateli and Giaglis (2004) developed a conceptual research framework for
analyzing e-business models based on 29 available studies in the field. In fact, classifying and
categorizing papers usually takes less time than reading, interpreting, and summarizing them does.
Researchers should also be aware of a background review’s purpose because it substantially affects the
review’s structure and presentation. For example, in design-oriented studies, background reviews can
help to illustrate the novelty and practical relevance of design ideas or they can identify related works that
214 Standing on the Shoulders of Giants: Challenges and Recommendations of Literature Search in Information
Systems Research
Volume 37 Paper 9
can inform the design process—two reasons for conducting the review that also affect how it will be
organized and written up. Therefore, researchers should make clear why they conduct the literature
review before they start their literature search.
5.3 Define the Search Scope
While justifying a review provides a clear account of why it needs to be conducted, it is similarly important
to clarify what will be done. Defining the search scope helps determine the basic features of the literature
search before it begins, so it provides an orienting framework for the search and helps the researcher
make the decisions that will determine the structure and organization of the search process. Cooper
(1988) developed a taxonomy for categorizing literature reviews, an idea we adapt to clarify the search
scope (Figure 1). Each of the rows of the framework guide researchers in specifying the nature of the
search process.
Figure 1. Definition of the Search Scope
1. Process: the search process is based on the literature-review goals. Just as a review can be
organized either iteratively or sequentially, so can a literature search. In a sequential review, the
literature search is a defined step at the beginning of the reviewing process, although it often has
to be refined and updated during the course of the review. Iterative reviews require the researcher
to search for literature in a continuous, repeated process that is intertwined with reviewing and
synthesizing the literature.
2. Sources: the sources of the search refer to the origins of the data included in the study. Literature
searches can draw on databases, publication outlets such as journals, and citation indexing
services. Again, the choice to base a search on one or more sources depends on the purpose and
goals of the literature review. For example, a review might include only conference publications if
the researcher’s intent is to uncover the emergence of new issues and trends. Examples of
available integrated sources include free services such as CiteSeerX, getCITED, and Google
Scholar and subscription-based services such as Elsevier’s Scopus and Thomson Reuter’s Web
of Science (Bandara et al., 2011). There are also other sources for literature searches, such as
the search portals that many university libraries offer.
3. Coverage: literature searches can either be comprehensive to collect as many relevant
publications as possible or focused to center on the seminal works in a field or subfield. However,
it is usually impracticable and sometimes even impossible to collect all publications on a subject
area in a comprehensive literature search, so the coverage of a search must be justified.
Researchers might also claim that their literature searches are representative by collecting a small
number of papers that typify a larger body of publications (Cooper, 1988). For example, a
literature review might explore how a certain subject is covered in IS research in particular and so
collect publications from only a small sample of top IS journals that represent the discipline.
4. Techniques: finally, the most common techniques of a literature search are keyword searches and
backward and forward searches. Backward searching basically refers to collecting appropriate
publications by screening the reference lists of the papers retrieved from the keyword search
(“going backward in time”), while forward searching refers to collecting appropriate publications
that have cited these papers (“going forward in time”) (Levy & Ellis, 2006; Webster & Watson,
Researchers can use this framework to position and plan their literature searches and to organize how
they write about what they have done. To provide an example for illustration, we highlight some fields in
gray in order to categorize an exemplary literature search from IS research (see Figure 1). Gerwin and
Barrowman’s (2002) literature search was part of a review paper on integrated product development.
Communications of the Association for Information Systems 215
Volume 37 Paper 9
Their search followed a (1) largely sequential process and (2) was grounded in the ABI/INFORM online
database and fifteen academic journals (sources). The search resulted in 28 publications on integrated
product development, so its coverage (3) was of the seminal works on the subject. The results from the
(4) keyword search were complemented with reference-based backward searches (techniques).
5.4 Test and Apply Combinations of Search Parameters
Defining search parameters involves selecting search terms, combining them with the help of search
operators, and using appropriate search fields (see Boell and Cecez-Kecmanovic (2014) for a more
detailed account of how literature searches are conducted online).
Search terms and research concepts are not the same thing. Searching by search terms is
inherently limited because many terms can be used to describe the same concept.
Consequently, defining search terms can be difficult unless one has established a solid
understanding of the subject area. A preliminary search for seminal publications can help
researchers to identify and select search terms. Reading the seminal papers on a subject
helps researchers to learn about the topic and related fields, and the keywords these papers
contain will often be applicable to their own searches. In addition, some databases support a
thesaurus that can contribute to defining search terms.
Several search operators can be used to improve results from the use of the search terms by,
for example, combining them with Boolean operators into search phrases (e.g., AND, OR).
While the exact use and interpretation of search phrases can be determined with the help of
brackets, the precision of the search can be influenced with quotation marks (for exact phrase
searching) and truncations (for less restrictive searches). Proximity operators (e.g., NEAR,
ADJ) can be used to search for terms that are closely related, although they must not appear
adjacent to each other as quotation marks would require.
The search fields are the parts of papers or publication metadata that are searched for the
defined search phrases. The most frequently used fields are publication titles, key words,
abstracts, and full texts. Selecting search fields has a considerable impact on the search
results; for example, a title search will retrieve far fewer publications than a full-text search.
Many other search fields can be used that require search terms that are not subject related,
such as author names, publication types, and publication years.
Most literature reviews in the IS discipline are conducted with the help of search phrases, which can be
broad or narrow depending on the review’s scope and objectives. Gerwin and Barrowman’s (2002) review
on integrated product development, for example, used a comparatively specific search string
(“(performance OR schedules OR speed OR time) AND (projects OR sample OR teams OR tests) AND
(product)”), while Ngai and Gunasekaran (2007) applied broader descriptors in their review of mobile
commerce research (“mobile commerce”, “m-commerce”), as did Leidner and Kayworth (2006) (“IT
culture”, “information systems culture”, “IT values”) and Wareham et al. (2005) (“eCommerce”, “electronic
commerce”, “electronic business”, “e-Business”).
Defining and using the search parameters has a significant impact on the search results, so researchers
often have to experiment with combinations of search terms and operators (Kitchenham, 2004). Tests of
search phrases and their application to various databases also help the researchers to improve their
feeling for their subjects. Such tests can also help them refine their search strategy because it can be
difficult to predict the results a keyword search will produce. As such, testing the search phrases helps the
researcher assess the feasibility and relevance of the literature review and search.
5.5 Use Seminal Sources to Build the Backbone of the Literature Review
Techniques other than keyword searches for finding the publications required for a literature review
include backward and forward searches. Researchers should evaluate these techniques’ applicability and
usefulness against their research objectives and, if they are applicable, use them in combination. The
coverage of literature searches can vary, but every literature search should at least cover the most
important publications on a topic, for which purpose backward and forward searches are especially helpful
(Webster & Watson, 2002). Backward and forward searches help to uncover the seminal publications on a
subject because these publications are usually heavily cited by other studies in the field.
216 Standing on the Shoulders of Giants: Challenges and Recommendations of Literature Search in Information
Systems Research
Volume 37 Paper 9
Backward and forward searching can be reference based or author based and can be conducted at any of
several levels (e.g., second-level or third-level searches) (Levy & Ellis, 2006). For example, a search for
“knowledge management” using Google Scholar shows that Alavi and Leidner’s (2001) literature review is
an authoritative source in the field. The papers provides a suitable foundation for a backward search
because its bibliography covers several other relevant knowledge-management studies and for a forward
search because it has more than 7,000 citations. Some of these citations are themselves important,
heavily cited studies in the knowledge management field, so they may be useful sources for further rounds
of backward/forward searching. However, if Alavi and Leidner recently published another paper on
knowledge management, it would probably not be on top of the Google Scholar results list because it
would not yet have many citations. (Google does not reveal how it sorts its results.) Therefore, neither
backward nor forward searches would reveal this more recent work if they are grounded only in citations,
while searching for the authors in bibliographic databases or screening their webpages would reveal the
new publication. This approach is an author-based approach to backward and forward searching (Levy &
Ellis, 2006).
As backward and forward searches are key to identifying the seminal papers on a subject, they have been
part of several literature reviews in IS research. For example, Melville et al. (2004) used the citations of
papers they identified in their keyword search as sources for their review on IT and organizational
performance, and Piccoli and Ives (2005) collected—as part of their review on IT-dependent strategic
initiatives—a large number of papers from journals other than those they searched with keywords in a
citation-based backward search.
Backward and forward searches help to identify publications that the keyword search did not retrieve, and
they can also be used to assess the extent to which the coverage of a literature search is satisfactory
because researchers can compare their search results with the bibliographies of the seminal papers in the
field. What’s more, citation analyses help the researcher to identify important contributions published in
outlets other than journals, such as books, which are not covered by many online databases. For
example, Alavi and Leidner’s (2001) review cites several important knowledge-management books that
are not indexed in many bibliographic databases.
Finally, colleagues’ recommendations can substantially inform the literature search process. Sharing the
search results and resulting bibliographies with colleagues early in the process helps the researcher to
collect feedback on the coverage of the literature search and often leads to including other important
publications that may have slipped through the net of the keyword search. For example, Krishnan and
Ulrich’s (2001) review on product-development decisions involved an electronic mail survey of
approximately fifty researchers to identify the most influential papers in the field. Contacting colleagues
and other experts may also help to identify yet unpublished sources (Kitchenham, 2004). A common
mistake of inexperienced researchers is to adhere stubbornly to a defined search strategy and to
disregard other potentially relevant publications, but there is no reason to exclude a relevant publication
from a literature review if the researcher came across it by means other than the keyword search, even by
5.6 Weigh Feasibility Against Coverage
With the help of techniques such as keyword searches and backward/forward searches, researchers can
collect large numbers of research publications for their literature reviews. Still, they may not be satisfied
with their search results when an overwhelming number of publications causes them concern that they
have overlooked important contributions (Boell & Cecez-Kecmanovic, 2014). It is difficult for new
researchers to understand that it is often impossible to collect and read all of the publications on a subject.
With the ever-increasing numbers of publications, coverage—not exhaustiveness—has become the
relevant measure of saturation.
Some literature reviews in IS research have managed to collect and analyze large numbers of studies. For
example, the meta-analyses by Wareham et al. (2005) and Glass et al. (2004) included 582 and 1,485
publications, respectively. There are also examples of conceptual literature syntheses with high levels of
coverage. Fjermestad and Hiltz (1998) analyzed 200 studies in their review of experimental research on
group support systems, and Melville et al. (2004) and Krishnan and Ulrich (2001) each collected and
analyzed around 200 publications for their literature reviews on IT business value and product
development decisions. However, because it is virtually impossible to collect all publications on a topic,
even the coverage of these reviews was not exhaustive. For example, Wareham et al. (2005) had to
restrict their analysis to a set of 28 IS journals due to the breadth of the e-commerce field, and Glass et al.
Communications of the Association for Information Systems 217
Volume 37 Paper 9
(2004) also selected a representative set of journals for their analysis of research in the computing
Coverage is always a matter of relevance, so researchers should not focus on collecting everything that
exists so much as a reasonable number of relevant publications (Petticrew & Roberts, 2006). However
coverage is judged or measured, it is likely to increase as the researcher invests more time, but
researchers must weigh coverage against decreasing returns on the investment of time. A search with
general search parameters retrieves most of the relevant studies but also many irrelevant ones, while a
focused search retrieves few irrelevant studies but comes at the risk of missing important publications.
Petticrew and Roberts (2006) used “sensitivity” (a high proportion of relevant studies) and “specificity” (a
low proportion of irrelevant studies) as measures of relevance, so the more specific a search is, the less
time-consuming the later tasks of selecting and retrieving the publications (there is typically a trade-off
between specificity and sensitivity). In their review of the decision support systems literature, Eierman et
al. (1995) reduced 200 potentially relevant publications to a set of 15 papers that finally become part of
their review on decision support systems.
Balancing available time with an appropriate level of coverage can be a challenge. Okoli and Schabram
(2010, p. 6) observe that “experienced researchers are familiar with a frequent tendency during the
research process to quickly finish the theoretical background section, and then ‘go on to the good stuff,’
meaning the data collection and analysis”, yet even researchers who are willing to spend more time on
their reviews may not have the opportunity to do so. In fact, many literature searches, including those that
are parts of review papers, are conducted under time pressure (Bandara et al., 2011). Accordingly,
researchers who conduct their reviews in a restricted period of time must weigh coverage against
feasibility in their literature searches.
5.7 Justify all Decisions Made During the Literature Search Process
A literature search requires the researcher to make several important decisions that concern the selection,
combination, and application of search parameters during the keyword search; the scope of citation
tracking and analysis; and the appraisal and selection of publications for the review. These and related
decisions must be justified and documented to establish confidence in the search results.
Although the choice of search terms should be grounded in a clear understanding of the subject,
keywords from other papers may also be used. Existing frameworks and concept mapping can further
justify how one selects keywords (Rowley & Slack, 2004). However, using parameters other than search
terms, such as publication years and types, should be justified as well; that is, a review must explain the
reasons for restricting a literature search to a certain time period or for considering only papers published
in certain outlets. For example, Wareham et al. (2005) set the starting date of their meta-analysis of e-
commerce research to 1997 because they wanted to have a representative sample of literature from the
years before and after the market crash of 2000, while Ngai and Gunasekaran (2007) consider
only studies published between 2000 and 2003 because most of the research in their field (m-commerce)
began during that period.
While certain events can justify the time horizon for a literature search, it is more difficult to explain why a
search is so narrow as to include only a small number of publication outlets. As an anonymous reviewer
cited by Webster and Watson (2002, p. xvi), observes, “studies of the IS literature have consistently been
limited by drawing from a small sample of journals. …I just can’t see the justification for searching by
journal instead of searching by topic across all relevant journals”. Books, book chapters, and conference
papers are promising sources for literature searches, so there is no reason for excluding them from
reviews. (There are several examples of literature reviews in IS that draw from sources other than journals
(e.g., Pateli & Giaglis, 2004).) Even so, if a topic must be studied specifically from an IS perspective, the
literature search can be grounded in a justifiable set of IS journals and conferences (Bandara et al., 2011).
Studies on the core and boundaries of the IS discipline (e.g., Sidorova, Evangelopoulos, Valacich, &
Ramakrishnan, 2008), such as many meta-analyses in IS research (e.g., King & He, 2006; Poeppelbuss
et al., 2011; Wareham et al., 2005), are good examples.
In such cases, especially in concept-centric literature syntheses, backward and forward searches, which
are based on citations rather than on publication outlets, become even more essential to finding relevant
literature that the keyword search did not produce. The structure and organization of citation analysis and
tracking should be justified not only by explaining which publications provided the foundation for the
218 Standing on the Shoulders of Giants: Challenges and Recommendations of Literature Search in Information
Systems Research
Volume 37 Paper 9
backward/forward searches and why, but also by justifying the search range and explaining the reasons
for including or excluding papers retrieved from the secondary searches.
Finally, the screening process must be justified and documented. Some general criteria are usually used
to exclude papers (e.g., language, availability), and many searches also exclude non-research
publications such as editorials and comments. However, it is more difficult to evaluate the quality of
publications, especially for inexperienced researchers, and relevance assessments depend largely on the
subject. For example, Eierman et al. (1995) included studies of group decision support systems, executive
information systems, and expert systems and excluded from the resulting sample papers that did not have
a sound theoretical basis and/or the goal of testing or extending existing theory in the field. Lin, Standing,
and Liu (2008) initially included in their review a set of 251 virtual-teams studies that discussed the factors
that affect virtual teams’ effectiveness, compared face-to-face-communication and computer-mediated-
communication, or discussed decision support systems in that context. Then they excluded publications
that did not provide correlation coefficients and those that used dependent/independent variables that did
not relate to the terms identified in the literature.
Several criteria can be used to evaluate the quality of publications (e.g., data collection, analysis, and
interpretation) and their relevance (e.g., research method, study context, participants) (Okoli & Schabram,
2010), but the applicability of these criteria varies from review to review, so this search activity is often a
highly contextual task, which makes justification even more important. Many scholars suggest using
journal rankings to judge the quality of publications (e.g., Levy & Ellis, 2006), but there are many
examples of seminal literature that are not published in highly ranked journals. Researchers should keep
in mind that a paper’s relevance or quality cannot be directly inferred from the ranking of the journal in
which it appears.
5.8 Develop a Protocol to Guide and Document the Search Process
A search protocol is a useful way to guide and organize the literature search and to document the
decisions made in the process. Several authors have suggested using protocols for systematic literature
reviews (Brereton, Kitchenham, Budgen, Turner, & Khalil, 2007; Kitchenham et al., 2009; Okoli &
Schabram, 2010), but all types of reviews can benefit from protocols, and they can also support the
literature search. A search protocol, as part of the review protocol, should cover all activities involved in a
literature search, including selecting and retrieving publications.
As such, search protocols can provide a background and justification for the literature search and describe
the search scope, including its objectives. The protocol should also explain how the keyword search will
be performed, including the search phrases, databases, and publications involved; whether and how
backward and forward searches will be used; how the literature retrieved will be screened for relevance
and quality; and how the relevant publications will be acquired and stored. Search protocols also describe
when these activities will be performed and, in the case of collaborative reviews, who is in charge of what
parts of performing the search. Search protocols should also reveal which software tools are used in the
process (e.g., for reference management).
As part of a literature review on systematic literature reviews in the software engineering field, Kitchenham
and Charters (2007) provide an example of a protocol that demonstrates how the search process can be
described and documented. Their protocol shows who searched which journals and conferences (e.g.,
Information and Software Technology, Journal of Systems and Software); explains the criteria of inclusion
(e.g., systematic literature reviews, meta-analyses) and exclusion (e.g., informal reviews, no peer review),
measures for quality assessment (e.g., description of inclusion and exclusion criteria), and the procedures
for data extraction (e.g., source, publication year, research question) and analysis (e.g., publication
activity, research topics); and identifies the researchers who were contacted in the search and review
Search protocols can help researchers plan the literature search in advance, so they should be created as
early in the review as possible. However, literature search strategies often change, such as when search
terms do not work as expected. Therefore, version management, which documents changes to the
literature search and tracks revisions and updates of the search protocol, helps to ensure the search is
explicit and reproducible (Okoli & Schabram, 2010). Anonymized search protocols can be submitted with
literature reviews to demonstrate academic rigor in the review process, and they can be provided in the
appendix of review papers or shared online.
Communications of the Association for Information Systems 219
Volume 37 Paper 9
If more than one researcher contributes to the review, the protocol helps to clarify responsibilities and
supports research training (Okoli & Schabram, 2010). For example, the protocol explains and defines
what criteria will be used for including and excluding papers, so it helps to ensure that all researchers
have a common level of understanding for the analysis. In addition, the protocol defines the databases,
techniques, and software tools that will be used during the search, which can also indicate the need for
training. Even naming files and using basic metadata (e.g., abbreviation of author names) can cause
inconsistencies, so a search protocol should explain how to store publications in a standardized way.
6 Conclusion
Researchers face several challenges in planning and conducting their literature searches that range from
a lack of guidelines to collaboration issues. Some of these challenges are especially important in the
context of IS research, yet most can emerge in any type of literature search, although their significance
will differ with the procedures applied. As a response to these challenges, we presented eight
recommendations that researchers may find helpful when planning and organizing their literature
We do not present these guidelines as a “step-by-step recipe”, nor do we advocate one approach to
literature searches, but the guidelines address the identified challenges that are associated with literature
searches in the IS field. For example, the recommendation to define the search scope considers that there
is no common, standardized approach to conducting a literature search in IS research, and the
recommendation to assess coverage and feasibility helps researchers deal with potentially overwhelming
numbers of publications. The recommendation to test alternative search techniques can support
researchers in dealing with unpredictable search results and in consistently using databases that differ in
terms of functionality and coverage. The recommendation to create search protocols supports
collaboration, helps researchers define the criteria used to include and exclude publications, and sets
standards for storage and retrieval. For their parts, the recommendations to justify decisions made during
the literature search and to collect seminal publications through backward and forward searching help
researchers proceed to higher levels of saturation and to have confidence in the decision to finish the
search. Finally, the recommendations to develop an understanding of the subject at the outset of a
literature review and to justify it are important preconditions for a literature search and contribute to
overcoming many of the challenges identified.
Identifying the challenges and developing guidelines are grounded in several other IS research
publications (e.g., Bandara et al., 2011; Boell & Cecez-Kecmanovic, 2014; Brereton et al., 2007;
Kitchenham, 2004; Levy & Ellis, 2006; Okoli & Schabram, 2010; Turner et al., 2008; Webster & Watson,
2002; Wolfswinkel et al., 2013). These and related sources discuss additional issues, but our account
presents a fair representation of the challenges that IS researchers typically face in any literature search,
regardless of the specific method. We focused on the search process as a specific early-stage element of
a literature review because of the increasing number of potentially relevant publications. As Pateli and
Giaglis (2004, p. 304) explain, “the selection phase is critical, since decisions made at this stage
undoubtedly have a considerable impact on the validity of the literature review results”. While there are
exceptions (e.g., Boell & Cecez-Kecmanovic, 2014), the literature search has not yet received much
attention from IS researchers, so this tutorial can support researchers in their endeavors to conduct
effective literature reviews and searches.
Table 1 presents the challenges and guidelines in checklist format.
Table 1. Literature Search Checklist
1. Before the literature search, did I…
…develop an understanding of the topic?
…justify why the literature review is necessary?
…define an appropriate search scope?
…assess the feasibility and coverage of the search?
220 Standing on the Shoulders of Giants: Challenges and Recommendations of Literature Search in Information
Systems Research
Volume 37 Paper 9
Table 1. Literature Search Checklist
2. During the literature search, did I…
…test alternative approaches to searching the literature?
…use justifiable search techniques and parameters?
…apply appropriate criteria for inclusion and exclusion?
…discuss my search strategies in the team?
3. After the literature search, did I…
…assess the sensitivity and specificity of my search?
…rigorously document the search process and results?
…compare my results with those of other literature reviews?
…collect feedback from colleagues?
As with the challenges and guidelines, the checklist in Table 1 does not cover all of the issues that are
relevant to literature searches, and some of the questions may not apply to all reviews (e.g., “did I discuss
my search strategies in the team?”). However, researchers may find the checklist helpful in ensuring that
they continuously reflect on their search strategies in terms of assessing and describing whether they
addressed these issues and how they did so.
7 Summary
A thoroughly planned literature search can help novice researchers make their way through the
increasingly dense publication jungle. It can increase researchers’ awareness of the available literature
and help them create an orienting map of promising research directions to ensure that they visit areas that
are worth studying and avoid less promising ones. The literature search process is important because it
determines the coverage of the literature review. Just as surveys require the researcher to calculate
sample sizes based on population, the literature search helps researchers define the boundaries in which
their review is conducted and position it in the increasingly vast landscape of research publications.
Few publications in IS research have discussed the structure and process of literature reviews. Today’s
researchers face several challenges in performing literature searches. The search for literature often
produces overwhelming numbers of potentially relevant publications, the screening of literature can cause
serious collaboration issues, and managing references can be difficult when it comes to storing and
retrieving documents. We present several such challenges and recommendations for dealing with them in
the course of a literature search. A checklist developed from the challenges and guidelines can help
researchers in carrying out their literature searches.
There are several approaches to conducting literature reviews and many possible review outcomes, and
the role of the literature search varies in IS research. Because there is no standardized approach to
crafting a literature review and search in the IS field, researchers must evaluate the applicability and
completeness of the guidelines presented here against their individual research objectives. However, the
challenges we discuss are relevant to most literature reviews, so the guidelines should be applicable to
most as well
This paper builds on ideas presented at the 17th European Conference on Information Systems (ECIS
2009) in Verona, Italy (vom Brocke et al., 2009). We thank the ECIS and CAIS reviewers for their
constructive comments and suggestions.
Communications of the Association for Information Systems 221
Volume 37 Paper 9
Ahuja, M. K. (2002). Women in the information technology profession: A literature review, synthesis and
research agenda. European Journal of Information Systems, 11(1), 20-34.
Alavi, M., & Leidner, D. E. (2001). Review: Knowledge management and knowledge management
systems: Conceptual foundations and research issues. MIS Quarterly, 25(1), 107-136.
Allen, I. E., & Olkin, I. (1999). Estimating time to conduct a meta-analysis from number of citations
retrieved. Journal of the American Medical Association, 282(7), 634-635.
Baker, M. J. (2000). Writing a literature review. The Marketing Review, 1(2), 219-247.
Bandara, W., Miskon, S., & Fielt, E. (2011). A systematic, tool-supported method for conducting literature
reviews in information systems. Paper presented at the 19th European Conference on Information
Systems, Helsinki, Finland.
Baumeister, R. F., & Leary, M. R. (1997). Writing narrative literature reviews. Review of General
Psychology, 1(3), 311-320.
Benbasat, I., & Weber, R. (1996). Research commentary: Rethinking “diversity” in information systems
research. Information Systems Research, 7(4), 389-399.
Boell, S. K., & Cecez-Kecmanovic, D. (2014). A hermeneutic approach for conducting literature reviews
and literature searches. Communications of the Association for Information Systems, 34(12), 257-
Boell, S. K., & Cecez-Kecmanovic, D. (Forthcoming). On being “systematic” in literature reviews in IS.
Journal of Information Technology.
Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., & Khalil, M. (2007). Lessons from applying the
systematic literature review process within the software engineering domain. Journal of Systems
and Software, 80(4), 571-583.
Cohn, L. D., & Becker, B. J. (2003). How meta-analysis increases statistical power. Psychological
Methods, 8(3), 243-253.
Collins, J. A., & Fauser, B. C. (2005). Balancing the strengths of systematic and narrative reviews. Human
Reproduction Update, 11(2), 103-104.
Cooper, H. M. (1988). Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in
Society, 1(1), 104-126.
Creswell, J. W. (2009). Research design: Qualitative, quantitative, and mixed methods approaches (3rd
ed.). Thousand Oaks, CA: Sage.
Eierman, M. A., Niederman, F., & Adams, C. (1995). DSS theory: A model of constructs and relationships.
Decision Support Systems, 14(1), 1-26.
Fettke, P. (2006). State-of-the-Art des State-of-the-Art: Eine Untersuchung der Forschungsmethode
“Review” innerhalb der Wirtschaftsinformatik. Wirtschaftsinformatik, 48(4), 257-266.
Fink, A. (2010). Conducting research literature reviews: From the Internet to paper (3rd ed.). Thousand
Oaks, CA: Sage.
Fjermestad, J., & Hiltz, S. R. (1998). An assessment of group support systems experimental research:
Methodology and results. Journal of Management Information Systems, 15(3), 7-149.
Gerwin, D., & Barrowman, N. J. (2002). An evaluation of research on integrated product development.
Management Science, 48(7), 938-953.
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative
research. Chicago, IL: Aldine.
Glass, R. L., Ramesh, V., & Vessey, I. (2004). An analysis of research in computing disciplines.
Communications of the ACM, 47(6), 89-94.
222 Standing on the Shoulders of Giants: Challenges and Recommendations of Literature Search in Information
Systems Research
Volume 37 Paper 9
Google Scholar. (2015). Knowledge management. Retrieved from"knowledge+management"&btnG=&lr=
Hart, C. (1998). Doing a literature review: Releasing the social science research imagination. London, UK:
Helmericks, S. G., Nelsen, R. L., & Unnithan, N. P. (1991). The researcher, the topic, and the literature: A
procedure for systematizing literature searches. Journal of Applied Behavioral Science, 27(3), 285-
Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research.
MIS Quarterly, 28(1), 75-105.
Khosrow-Pour, M. (2014). Encyclopedia of information science and technology (3rd ed.). Hershey, PA: IGI
King, W. R., & He, J. (2006). A meta-analysis of the technology acceptance model. Information &
Management, 43(6), 740-755.
Kitchenham, B. (2004). Procedures for performing systematic reviews (technical report). Keele University
and National ICT Australia Ltd. Retrieved from
Kitchenham, B., Brereton, O. P., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009). Systematic
literature reviews in software engineering—a systematic literature review. Information and Software
Technology, 51(1), 7-15.
Kitchenham, B., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software
engineering (EBSE Technical Report EBSE-2007-01). Retrieved from
Krishnan, V., & Ulrich, K. T. (2001). Product development decisions: A review of the literature.
Management Science, 47(1), 1-21.
Lamp, J. W. (2004). Index of information systems journals. Retrieved from
Leidner, D. E., & Kayworth, T. (2006). Review: A review of culture in information systems research:
Toward a theory of information technology culture conflict. MIS Quarterly, 30(2), 357-399.
Levy, Y., & Ellis, T. J. (2006). A systems approach to conduct an effective literature review in support of
information systems research. Informing Science Journal, 9, 181-212.
Lin, C., Standing, C., & Liu, Y.-C. (2008). A model to develop effective virtual teams. Decision Support
Systems, 45(4), 1031-1045.
Melville, N., Kraemer, K., & Gurbaxani, V. (2004). Review: Information technology and organizational
performance: An integrative model of IT business value. MIS Quarterly, 28(2), 283-322.
MISQ. (n.d.). Objectives of the MISQ theory and review department. Retrieved from
Ngai, E. W., & Gunasekaran, A. (2007). A review for mobile commerce research and applications.
Decision Support Systems, 43(1), 3-15.
Okoli, C., & Schabram, K. (2010). A guide to conducting a systematic literature review of information
systems research. Sprouts: Working Papers on Information Systems, 10(26). Retrieved from
Pateli, A. G., & Giaglis, G. M. (2004). A research framework for analysing ebusiness models. European
Journal of Information Systems, 13(4), 302-314.
Peffers, K., & Hui, W. (2003). Collaboration and author order: Changing patterns in IS research.
Communications of the Association for Information Systems, 11(10), 166-190.
Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide. Malden,
MA: Blackwell Publishing.
Piccoli, G., & Ives, B. (2005). Review: IT-dependent strategic initiatives and sustained competitive
advantage: A review and synthesis of the literature. MIS Quarterly, 29(4), 747-776.
Communications of the Association for Information Systems 223
Volume 37 Paper 9
Poeppelbuss, J., Niehaves, B., Simons, A., & Becker, J. (2011). Maturity models in information systems
research: Literature search and analysis. Communications of the Association for Information
Systems, 29(27), 505-532.
Rowley, J., & Slack, F. (2004). Conducting a literature review. Management Research News, 27(6), 31-39.
Schwarz, A., Mehta, M., Johnson, N., & Chin, W. W. (2007). Understanding frameworks and reviews: A
commentary to assist us in moving our field forward by analyzing our past. The DATA BASE for
Advances in Information Systems, 38(3), 29-50.
Sidorova, A., Evangelopoulos, N., Valacich, J. S., & Ramakrishnan, T. (2008). Uncovering the intellectual
core of the information systems discipline. MIS Quarterly, 32(3), 467-482.
Turner, M., Kitchenham, B., Budgen, D., & Brereton, P. (2008). Lessons learnt undertaking a large-scale
systematic literature review. Paper presented at the 12th International Conference on Evaluation
and Assessment in Software Engineering, Bari, Italy.
vom Brocke, J., Simons, A., Niehaves, B., Riemer, K., Plattfaut, R., & Cleven, A. (2009). Reconstructing
the giant: On the importance of rigour in documenting the literature search process. Paper
presented at the 17th European Conference on Information Systems, Verona, Italy.
Wareham, J., Zheng, J. G., & Straub, D. (2005). Critical themes in electronic commerce research: A meta-
analysis. Journal of Information Technology, 20(1), 1-19.
Webster, J., & Watson, R. T. (2002). Analyzing the past to prepare for the future: Writing a literature
review. MIS Quarterly, 26(2), xiii-xxiii.
Wolfswinkel, J. F., Furtmueller, E., & Wilderom, C. P. (2013). Using grounded theory as a method for
rigorously reviewing literature. European Journal of Information Systems, 22(1), 45-55.
About the Authors
Jan vom Brocke is Hilti Chair of Business Process Management and Director of the Institute of
Information Systems at the University of Liechtenstein. His research has appeared in, among others, MIS
Quarterly (MISQ), the Journal of Management Information Systems (JMIS), and Information and
Management (I&M). He serves on the editorial review boards of several journals, including the Journal of
the Association for Information Systems (JAIS), Business & Information Systems Engineering (BISE), and
the Journal of Information Technology Theory and Application (JITTA). He is author and editor of seminal
books, such as the International Handbook on Business Process Management. Jan has taught at a
number of universities, including the University of Muenster (Germany), the University of St.Gallen
(Switzerland), and the University of Turku (Finland), and he is an active supporter of the AAU IS PhD
program at the University of Addis Ababa (Ethiopia).
Alexander Simons is Assistant Professor at the Institute of Information Systems at the University of
Liechtenstein. He received his BSc and MSc in Information Systems from the University of Muenster,
Germany, and his PhD in Economics from the University of Liechtenstein. Alexander’s main research
interests are document and content management as well as business process management.
Kai Riemer is Chair of Business Information Systems at the University of Sydney Business School. Kai
has held previous positions with Muenster University in Germany, The University of Melbourne and
University College Dublin. Kai’s expertise and research interests cover the areas of Social Networking,
Technology Appropriation, Enterprise 2.0, Virtual Work, Digital Disruption and the Philosophy of
Technology. His research follows practice theoretical and interpretivist approaches and has appeared in
journals such as the European Journal of Information Systems, Journal of Information Technology,
International Journal of Electronic Commerce, or Communications of the Association for Information
Systems. Kai is a board member of the Journal of Information Technology, Electronic Markets, and the
Business and Information Systems Engineering journal.
224 Standing on the Shoulders of Giants: Challenges and Recommendations of Literature Search in Information
Systems Research
Volume 37 Paper 9
Bjoern Niehaves is Full Professor of Information Systems at University of Siegen, Germany, Visiting
Distinguished Professor at Aalto University, Finland, and Fellow at the Hertie School of Governance,
Germany. He received a Ph.D. in Information Systems and a PhD in Political Science from Muenster
University. Bjoern held visiting positions at Harvard University (USA), the London School of Economics
and Political Science (UK), Waseda University (Japan), Royal Institute of Technology (Sweden), and
Copenhagen Business School (Denmark). He has published more than 200 research papers.
Ralf Plattfaut works as a management consultant. He received his PhD in Information Systems from the
University of Muenster, where he worked as a research assistant. He was also a research associate at the
Hertie School of Governance in Berlin, Germany, and studied Business Process Engineering at the
University of Liechtenstein. Ralf’s research areas are process-oriented dynamic capabilities and ICT and
Anne Cleven is working as a project manager. She received her B.S. and M.S. in Information Systems
from the University of Muenster, Germany, and her Ph.D. in Economics from the University of St.Gallen,
Switzerland. Her major research interests lie in the areas of Business Process Management as well as
Performance Measurement and Management.
Copyright © 2015 by the Association for Information Systems. Permission to make digital or hard copies of
all or part of this work for personal or classroom use is granted without fee provided that copies are not
made or distributed for profit or commercial advantage and that copies bear this notice and full citation on
the first page. Copyright for components of this work owned by others than the Association for Information
Systems must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on
servers, or to redistribute to lists requires prior specific permission and/or fee. Request permission to
publish from: AIS Administrative Office, P.O. Box 2712 Atlanta, GA, 30301-2712 Attn: Reprints or via e-
mail from
... To identify the value propositions and affordances of implemented e-Government projects from literature, we conducted a concept-centric systematic literature review (Cram et al., 2017;Okoli, 2015;Schryen et al., 2017;Vom Brocke et al., 2015;Webster & Watson, 2002). There are different approaches to conducting literature reviews, such as meta-analysis or scoping review (Paré et al., 2015;Rowe, 2014;Schryen et al., 2017). ...
... However, this study chose the concept-centric systematic literature advocated by IS researchers (Cram et al., 2017;Okoli, 2015;Schryen, 2015;Vom Brocke et al., 2015;Webster & Watson, 2002). We adopted the concept-centric approach because it provides the standard of a literature review and stimulates it toward a synthesis of what is known about the phenomenon of study. ...
Full-text available
This study examines the mechanisms fostering sustainable value propositions of implemented e‐Governments projects in sub‐Saharan Africa (SSA) as a research focus. Value propositions identify how public organizations use e‐Government to fulfill citizens' needs across different roles. Using a concept‐centric systematic review, we identified such value propositions as effective public organizations, quality service delivery, open government and democratic values, as well as social values and well‐being. We chose some selected e‐Government projects in Nigeria and Rwanda; adopted realist evaluation (RE) multiple case studies as a methodology and introduced affordance actualization, self‐determination theory, and sustainability framework for e‐Government success as theoretical lenses. To collect data, we adopted a mixed method RE approach and used a contingent valuation method (CVM) as a quantitative aspect to identify the actualized value propositions in operations in the implemented e‐Government projects in Nigeria and Rwanda as outcomes. Sequentially followed by a realist interview as a qualitative approach to collect and analyze data we uncovered the generative mechanisms causing such outcomes to exist and be sustainable. Our findings revealed technological cognizance and autonomy as well as competence as individual mechanisms identified for sustaining such outcomes. While organizational mechanisms revealed are continuous availability of funds, institutional, administrative, and coordinating capacity with leadership and political support. Other organizational mechanisms include institutionalizing the technology within local settings with ongoing monitoring and improvement evaluation enabled by both internal and external contextual conditions. We synthesized our findings into an actualized value sustainability framework for implemented e‐Government projects in SSA to contribute to academics and practice in ICT4D, particularly RE application in e‐Government study, which is nascent.
... Both authors maintain that synthesizing literature ought to identify what we know, as well as the gaps and research opportunities this implies, and provide a foundation for future theorizing within the research domain. There are different approaches to conducting literature reviews, such as meta-analysis or a scoping review (Paré et al., 2015;Rowe, 2014;Schryen et al., 2017), where this study elected to use the concept-centric systematic literature advocated by IS researchers (Cram et al., 2017;Okoli, 2015;Schryen, 2015;Vom Brocke et al., 2015;Webster & Watson, 2002). The study adopted the concept-centric approach, because this provides the standard of a literature review, by stimulating it towards a synthesis of what is known about the phenomenon of study, rather than a summary of what previous authors have reported while forming the foundations of theory building, testing, and developing future research direction (Watson & Webster, 2020;Webster & Watson, 2002). ...
... Such an approach involves a rigorous identification, synthesis, and assessment of qualitative and quantitative published articles to arrive at an empirical answer to the study's research questions (e.g., Schryen et al., 2017;Webster & Watson, 2002). With regards to how to conduct such a review, previous studies (e.g., Cram et al., 2017;Okoli, 2015;Schryen et al., 2017;Vom Brocke et al., 2015;Webster & Watson, 2002) have offered us a practical guide. The study summarized such guidance into: (1) producing a clear articulation of the review scope and boundaries; (2) detailing the steps taken to collect the relevant literature; and (3) showcasing how the literature was analyzed. ...
Full-text available
This study aims to develop a framework that identifies and measures the value propositions of implemented e-Government projects. Consequently, the study conducted a systematic review using a concept-centric and thematic approach by reviewing 84 articles as primary research data. In doing so, it identified: (1) effective public organizations; (2) quality service delivery; (3) open government and democratic value; and (4) social value and well-being. Furthermore, two gaps were identified, namely a dearth of studies on the measures well as actualization, and sustainability of the value propositions. The study synchronized the identified value propositions to develop a framework for value propositions of implemented e-Government projects. While multiple theoretical perspectives were advocated for this includes affordance actualization theory grounded in realist evaluation and self-determination theory. This was done to uncover the contextual conditions and mechanisms that will foster the actualization and sustainability of the identified value propositions of implemented e-Government projects as a future research direction.
... Der SLR-Prozess besteht aus drei Schritten "Planung, Durchführung" und "Ergebnisanalyse", die in den nächsten Abschnitten im Detail dokumentiert werden. Der Ansatz von Kitchenham wurde mit der "Literature Search Checklist" von vom Brocke et al. (2015) unterstützt, damit die Qualität des Suchprozesses gewährleistet werden kann. ...
... To identify the value propositions, we conducted a concept-centric systematic literature review (Cram et al., 2017;Okoli, 2015;Schryen et al., 2017;Vom Brocke et al., 2015;Webster & Watson, 2002) followed by thematic coding analysis as advocated by IS researchers (Aksulu & Wade, 2010;Bandara et al., 2015;Inuwa & Ononiwu, 2020;Roberts et al., 2012). We conducted a keyword search in titles, abstracts, and references of researches archived in PUBMED, Wiley online, Emerald, Springer, Elsevier. ...
... To synthesize existing research, we performed a descriptive literature review [20], following the guidelines proposed by vom Brocke et al. [57,58] and Webster and Watson [59]. ...
... To identify existing ITPPM process models, we considered the knowledge base and performed a literature review, and the environmental perspective by performing 24 expert interviews. For the literature review, we followed the guidelines proposed by vom Brocke et al. (2009Brocke et al. ( , 2015, Webster and Watson (2002), and Watson and Webster (2020). We searched in six databases (AISeL, Science Direct, ACM, IEEE, Web of Science, Springer Link), the International Journal of Project Management, and Project Management Journal using a keyword search string (("IT project selection" OR "IT project evaluation" OR "IT portfolio selection" OR "IT portfolio evaluation" OR "IT portfolio management") AND ("framework" OR "process model" OR "procedure model" OR "cycle")) and identified 989 relevant publications. ...
Conference Paper
Full-text available
Companies must optimize their information technology (IT) project portfolio to achieve goals. However, IT projects often exceed resources and do not create their promised value, for example, because of missing structured processes and evaluation methods. Continuous IT portfolio management is thus of importance and a critical business activity to reach value-driven goals. Guided by Design Science Research with literature reviews and expert interviews, we develop, evaluate, and adjust an IT project portfolio management process model, a holistic IT project evaluation framework, and implement a decision support system prototype. Our results and findings synthesize and extend previous research and expert opinions and guide decision-makers to make more informed and objective IT project portfolio management decisions aligned with optimal value creation. Furthermore, we deduce new research opportunities for IT project portfolio management process models, decision tools, and evaluation frameworks.
Full-text available
The black-box nature of Artificial Intelligence (AI) models and their associated explainability limitations create a major adoption barrier. Explainable Artificial Intelligence (XAI) aims to make AI models more transparent to address this challenge. Researchers and practitioners apply XAI services to explore relationships in data, improve AI methods, justify AI decisions, and control AI technologies with the goals to improve knowledge about AI and address user needs. The market volume of XAI services has grown significantly. As a result, trustworthiness, reliability, transferability, fairness, and accessibility are required capabilities of XAI for a range of relevant stakeholders, including managers, regulators, users of XAI models, developers, and consumers. We contribute to theory and practice by deducing XAI archetypes and developing a user-centric decision support framework to identify the XAI services most suitable for the requirements of relevant stakeholders. Our decision tree is founded on a literature-based morphological box and a classification of real-world XAI services. Finally, we discussed archetypical business models of XAI services and exemplary use cases.
Full-text available
Die fortschreitende Verbreitung digitaler Technologien verändert den Charakter von Dienstleistungsangeboten in nahezu allen wirtschaftlichen und gesellschaftlichen Bereichen grundlegend. Unternehmen können auf Basis von Daten, die mit Hilfe von Sensorik in vernetzten physischen Objekten gesammelt und auf digitalen Plattformen verarbeitet werden, digitale Dienste und persönliche Dienstleistungen bedarfsgerecht zu sogenannten „Smart Services“ kombinieren. Diese datenbasierte, individuelle Konfiguration der Leistungsangebote und ihre Anpassung an den situativen Kontext von Nutzenden eröffnet einerseits neue Potenziale für kundenseitige Mehrwerte. Andererseits ermöglichen Smart Services durch den Einsatz von Verfahren der künstlichen Intelligenz eine zunehmend automatisierte Individualisierung und beinhalten daher auch Potenziale für anbieterseitige Effizienzsteigerungen. Das Ausschöpfen der Potenziale von Smart Services stellt Unternehmen jedoch auch vor Herausforderungen. Denn der Einsatz von Sensorik in sensiblen Bereichen, der erhöhte Grad an Automatisierung und schwer nachvollziehbare Algorithmen bergen auch kundenseitige Risiken, welche sich negativ auf die Wahrnehmung und Akzeptanz von Smart Services auswirken und so ihren Markterfolg beeinträchtigen können. Entsprechend kommt dem Sicherstellen einer hohen wahrgenommenen Qualität bei der Entwicklung von Smart Services eine zentrale Bedeutung zu. Bisher existieren in der wissenschaftlichen Literatur jedoch weder Ansätze für ein Verständnis der Smart-Service-Qualität noch geeignete Methoden, um dieses Verständnis in Form von Gestaltungswissen in der Entwicklung systematisch zu nutzen. Um diese Forschungslücke zu schließen und Unternehmen bei der Entwicklung von qualitativ hochwertigen Smart Services zu unterstützen, wird in der vorliegenden Arbeit ein Verfahren für einen Test der wahrgenommenen Qualität entwickelt. Mit dem Verfahren sollen Unternehmen befähigt werden, bereits während der Entwicklung von Smart Services eine Bewertung der Qualitätswahrnehmung mit unterschiedlichen Anspruchsgruppen durchzuführen und so Fehlentwicklungen vorzubeugen. Das entwickelte Verfahren umfasst zwei zentrale Lösungskomponenten. Zunächst wird ein Rahmenkonzept zur Bewertung der Smart-Service-Qualität entwickelt. Das Rahmenkonzept integriert und strukturiert bekannte Qualitätsmerkmale aus bestehenden Ansätzen und passt diese auf Besonderheiten von Smart Services an. Darüber hinaus werden in der Arbeit neue, spezifische Qualitätsmerkmale identifiziert, detailliert beschrieben und für die Bewertung operationalisiert. Mit dem Rahmenkonzept wird somit ein ganzheitliches Verständnis für die Qualitätswahrnehmung von Smart Services geschaffen und die Grundlage für eine strukturierte Bewertung gelegt. Die zweite Lösungskomponente stellt ein Verfahren dar, welches die Durchführung eines systematischen Tests von Smart Services unter Einbindung potenzieller Kunden und Nutzer ermöglicht. Anhand von fünf Verfahrensphasen wird beschrieben, wie das Rahmenkonzept zur Absicherung einer hohen wahrgenommenen Qualität systematisch eingesetzt werden kann. Die Phasen werden anhand von Zielen, Aktivitäten und Techniken sowie der zu erreichenden Ergebnisse dargelegt. Zur Unterstützung der Durchführung werden zudem entsprechende Methoden und Werkzeuge vorgestellt. Abschließend wird die Praxistauglichkeit des entwickelten Testverfahrens durch die Anwendung der beiden Lösungskomponenten in drei Entwicklungsprojekten in Kooperation mit Unternehmen nachgewiesen. Damit stellt das Verfahren eine wertvolle Unterstützung für Unternehmen bei der Umsetzung eines systematischen und qualitätszentrierten Smart-Service-Engineerings dar.
Full-text available
Data mining approaches have been widely used to estimate student performance in online education. Various Machine Learning (ML) based data mining techniques have been developed to evaluate student performance accurately. However, they face specific issues in implementation. Hence, a novel hybrid Elman Neural with Apriori Mining (ENAM) approach was presented in this article to predict student performance in online education. The designed model was validated with the student's performance dataset. Incorporating the Elman neural system eliminates the noise data present in the dataset. Moreover, meaningful features are extracted in feature analysis and trained in the system. Then, the student's performances are sorted based on their average score and classified as good, bad, or average. In addition, a case study was developed to describe the working of the designed model. The presented approach was executed in python software, and performance metrics were estimated. Moreover, a comparative analysis was performed to prove that the proposed system earned better outcomes than existing approaches.
Full-text available
The digital transformation of work has long since arrived in our everyday lives. The crisis-driven digital transformation through COVID-19 undoubtedly accelerated many things and acted as a burning glass for processes that had not yet been digitally transformed. This dissertation is dedicated to the fundamental digital transformation of work and, with a total of fifteen research studies, to the framework of the crisis-driven digital transformation of work, among others. With reference to the framework concepts of "Transformation Governance", "Digital Leadership" and "Future Technology Management", approaches are identified on how to close the theoretical and practical gap regarding the digital transformation of work. Organizational, individual, and technical approaches are identified as prerequisites for the successful digital transformation of work. The findings will enable future research to investigate current and emerging phenomena, such as issues of collaboration in virtual realities and fostering cohesion in purely virtual teams. Methodologically, established approaches such as interviews, case studies, surveys, experimental and mixed methods approaches are combined. Therefore, the dissertation can provide a starting point for discussion, but also for further research in the field of business informatics and in other disciplines. Furthermore, this work provides valuable insights and recommendations for action for experts, especially in SMEs, who want to face the digital transformation.
Full-text available
The objective of this report is to propose comprehensive guidelines for systematic literature reviews appropriate for software engineering researchers, including PhD students. A systematic literature review is a means of evaluating and interpreting all available research relevant to a particular research question, topic area, or phenomenon of interest. Systematic reviews aim to present a fair evaluation of a research topic by using a trustworthy, rigorous, and auditable methodology. The guidelines presented in this report were derived from three existing guidelines used by medical researchers, two books produced by researchers with social science backgrounds and discussions with researchers from other disciplines who are involved in evidence-based practice. The guidelines have been adapted to reflect the specific problems of software engineering research. The guidelines cover three phases of a systematic literature review: planning the review, conducting the review and reporting the review. They provide a relatively high level description. They do not consider the impact of the research questions on the review procedures, nor do they specify in detail the mechanisms needed to perform meta-analysis.
Full-text available
General guidelines for conducting literature reviews often do not address the question of literature searches and dealing with a potentially large number of identified sources. These issues are specifically addressed by so-called systematic literature reviews (SLRs) that propose a strict protocol for the search and appraisal of literature. Moreover, SLRs are claimed to be a ‘standardized method’ for literature reviews that is replicable, transparent, objective, unbiased and rigorous, and thus superior to other approaches for conducting literature reviews. These are significant and consequential claims that – despite increasing adoption of SLRs – remained largely unnoticed in the information systems (IS) literature. The objective of this debate is to draw attention of the IS community to SLR’s claims, to question their justification and reveal potential risks of their adoption. This is achieved by first examining the origins of SLR and the prescribed SLR process and then by critically assessing their claims and implications. In this debate, we show that SLRs are applicable and useful for a very specific kind of literature review, a meta study that identifies and summarizes evidence from earlier research. We also demonstrate that the claims that SLRs provide superior quality are not justified. More importantly, we argue that SLR as a general approach to conducting literature reviews is highly questionable, concealing significant perils. The paper cautions that SLR could undermine critical engagement with literature and what it means to be scholarly in academic work.
Full-text available
The quality and success of scholarly work depends in large measure on the quality of the literature review process. This paper advances conceptual understanding of the literature review process and extends earlier guidelines on literature reviews. It proposes a hermeneutic framework that integrates the analysis and interpretation of literature and the search for literature. This hermeneutic framework describes the literature review process as fundamentally a process of developing understanding that is iterative in nature. Using the hermeneutic circle it describes the literature review process as being constituted by literature searching, classifying and mapping, critical assessment, and argument development. The hermeneutic approach emphasizes continuous engagement with and gradual development of a body of literature during which increased understanding and insights are developed. The paper contributes to better understanding of the literature review process and provides guidelines to assist researchers in conducting high quality reviews. Approaches for efficient searching are included in an Appendix.
Conference Paper
Full-text available
Science is a cumulative endeavour as new knowledge is often created in the process of interpreting and combining existing knowledge. This is why literature reviews have long played a decisive role in scholarship. The quality of literature reviews is particularly determined by the literature search process. As Sir Isaac Newton eminently put it: “If I can see further, it is because I am standing on the shoulders of giants.” Drawing on this metaphor, the goal of writing a literature review is to reconstruct the giant of accumulated knowledge in a specific domain. And in doing so, a literature search represents the fundamental first step that makes up the giant’s skeleton and largely determines its reconstruction in the subsequent literature analysis. In this paper, we argue that the process of searching the literature must be comprehensibly described. Only then can readers assess the exhaustiveness of the review and other scholars in the field can more confidently (re)use the results in their own research. We set out to explore the methodological rigour of literature review articles published in ten major information systems (IS) journals and show that many of these reviews do not thoroughly document the process of literature search. The results drawn from our analysis lead us to call for more rigour in documenting the literature search process and to present guidelines for crafting a literature review and search in the IS domain.
The aim of this publication is to provide basic information about, and links to, either paper or electronic journal web sites, and more particularly, to provide links to their instructions to authors information.
Such diverse thinkers as Lao-Tze, Confucius, and U.S. Defense Secretary Donald Rumsfeld have all pointed out that we need to be able to tell the difference between real and assumed knowledge. The systematic review is a scientific tool that can help with this difficult task. It can help, for example, with appraising, summarising, and communicating the results and implications of otherwise unmanageable quantities of data. This book, written by two highly-respected social scientists, provides an overview of systematic literature review methods: Outlining the rationale and methods of systematic reviews; Giving worked examples from social science and other fields; Applying the practice to all social science disciplines; It requires no previous knowledge, but takes the reader through the process stage by stage; Drawing on examples from such diverse fields as psychology, criminology, education, transport, social welfare, public health, and housing and urban policy, among others. Including detailed sections on assessing the quality of both quantitative, and qualitative research; searching for evidence in the social sciences; meta-analytic and other methods of evidence synthesis; publication bias; heterogeneity; and approaches to dissemination.
Although a large volume of literature is available on mobile commerce (m-commerce), the topic is still under development and offers potential opportunities for further research and applications. Since the subject is at the stage of development, a review of the literature on m-commerce with the objective of bringing to the fore the state-of-art in m-commerce research and applications will initiate further research on the growth of m-commerce technologies. This paper reviews the literature on m-commerce and applications using a suitable classification scheme to identify the gap between theory and practice and future research directions. The 149 m-commerce articles are classified and the results of these are presented based on a scheme that consists of five distinct categories: m-commerce theory and research, wireless network infrastructure, mobile middleware, wireless user infrastructure, and m-commerce applications and cases. A comprehensive list of references is presented. We hope that the findings of this research will provide useful insights into the anatomy of m-commerce literature and be a good source for anyone who is interested in m-commerce. The paper also provides some future directions for research.
Two paradigms characterize much of the research in the Information Systems discipline: behavioral science and design science. The behavioral-science paradigm seeks to develop and verify theories that explain or predict human or organizational behavior. The design-science paradigm seeks to extend the boundaries of human and organizational capabilities by creating new and innovative artifacts. Both paradigms are foundational to the IS discipline, positioned as it is at the confluence of people, organizations, and technology. Our objective is to describe the performance of design-science research in Information Systems via a concise conceptual framework and clear guidelines for understanding, executing, and evaluating the research. In the design-science paradigm, knowledge and understanding of a problem domain and its solution are achieved in the building and application of the designed artifact. Three recent exemplars in the research literature are used to demonstrate the application of these guidelines. We conclude with an analysis of the challenges of performing high-quality design-science research in the context of the broader IS community.