Conference PaperPDF Available, an IT Artifact Supporting Exploratory Literature Searches for Information Systems Research


Abstract and Figures

Information Systems (IS) researchers currently lack an obvious place to start their literature searches. Existing tools suffer from being either too narrow in their coverage of existing research, leading to an insufficiency effect (low recall); or they are too encompassing, leading to an impracticality effect (low precision). From 11 listings of IS-related journals, we identify a set of 1,042 journals receptive to IS research. We introduce a web interface that allows searching for literature across most of these journals. The search tool enables researchers to narrow or widen the focus of searches, thus allowing researchers to optimise the precision-recall trade-off of their literature searches. We provide an evaluation of our artifact and discuss the relevance of our artifact for exploratory literature searches. Our artifact seeks to facilitate knowledge claims in IS research based on a shared body of knowledge beyond the AIS basket of eight journals.
Content may be subject to copyright.
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches, an IT Artifact Supporting Exploratory
Literature Searches for Information Systems Research
Sebastian Boell
Discipline of Business Information Systems
The University of Sydney Business School
Sydney, Australia
Blair Wang
School of Information Systems and Technology Management
UNSW Business School
Sydney, Australia
Information Systems (IS) researchers currently lack an obvious place to start their literature searches.
Existing tools suffer from being either too narrow in their coverage of existing research, leading to an
insufficiency effect (low recall); or they are too encompassing, leading to an impracticality effect (low
precision). From 11 listings of IS-related journals, we identify a set of 1,042 journals receptive to IS
research. We introduce a web interface that allows searching for literature across most of these
journals. The search tool enables researchers to narrow or widen the focus of searches, thus allowing
researchers to optimise the precision-recall trade-off of their literature searches. We provide an
evaluation of our artifact and discuss the relevance of our artifact for exploratory literature searches.
Our artifact seeks to facilitate knowledge claims in IS research based on a shared body of knowledge
beyond the AIS basket of eight journals.
Keywords: Literature Search, IS Journals, Literature Reference Database, Bibliometrics, Seniors
Basket of Journals
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
1 Introduction
Identifying relevant literature to understand the current state of knowledge is an important endeavour
for any research project in order to successfully argue for the relevance of its contribution (Boell and
Hovorka 2019; Hovorka and Boell 2015). Moreover, stand-alone literature reviews have received
considerable attention in IS in recent years (Boell and Cecez-Kecmanovic 2014; Paré et al. 2015; Tate
et al. 2015; vom Brocke et al. 2015; Webster and Watson 2002). Searches in databases such as the
AIS e-Library (AISeL), Scopus, Web of Science (WoS), Google Scholar – are one important means for
identifying earlier research to be included in literature reviews (Greenhalgh and Peacock, 2005).
Good academic practice involves the documentation of completed literature searches. However, it is
not currently clear where IS scholars should start looking for literature. While there is the AISeL, it
only covers few IS journals, currently not even allowing a search through what is published in the AIS
Senior Scholars’ Basket of Eight journals (Bo8). In contrast, many disciplines have established
literature reference databases that serve as a starting point when looking for existing research. For
instance, Psychology researchers generally begin looking for literature in Psycinfo, indexing around
2,500 journals; Medical researchers would start in Medline, indexing around 5,200 journals; and in
Information Science, LISTA selectively indexes around 600 journals. By comparison, the AISeL covers
15 journals, only 2 of which are included in the Bo8. Our research is thus motivated by the lack of a
simple search tool that provides a starting point for IS researchers when they embark on their journey
of identifying existing research of relevance to IS.
We develop an artifact that supports IS researchers in undertaking exploratory literature searches
(Bates 1989; Marchionini 2006; Savolainen 2017). While the Bo8 serves well as an aspiration for
publication, limiting the literature identification process to only what is published in the Bo8 ignores
many relevant articles published outside of it. This is what we, in this paper, call the insufficiency
effect. Besides a narrow search in the Bo8, IS researchers frequently use large-scale multidisciplinary
literature reference databases for their searches such as Google Scholar, Scopus or WoS. These
databases, however, generally cover tens of thousands of journals, many of which publish research of
very little relevance to IS researchers. Researchers are therefore often overwhelmed by a huge number
of so called ‘false positives’ (Larsen et al. 2019). This makes it difficult to make sense of the huge
volume of retrieved documents in a feasible amount of time, something we call in this paper the
impracticality effect. This paper engages with the need to balance between the insufficiency effect and
the impracticality effect.
Examining journals appearing across eleven listings of IS journals, we develop an extensive set of
1,042 journals whose at least partial relevance to IS research has been previously established. Ranking
these journals by how often they are included in multiple journal listings, we identify a means through
which literature searches can be conducted by narrowing and widening the focus of searches. Setting
up an open source project, we build an IT artifact for the IS community that enables searching though
these journals using the Scopus database. In the spirit of open source we propose that the literature
search artifact available at be refined by the collective effort of the IS community.
2 Literature Reviews and Literature Searches
As the IS discipline has matured, recognition of the need to stock of existing IS research has also
increased (Boell and Cecez-Kecmanovic 2014; Tate et al. 2015; vom Brocke et al. 2015; Webster and
Watson 2002). There are different types of literature reviews that serve different purposes including
theory development, providing an overview over a research area, and detailed analysis of a particular
effect in a meta-analysis (Paré et al. 2015). Such literature reviews use various approaches for
identifying relevant publications to be included in the review, ranging from keyword searches in
databases, citation tracking of relevant papers, analysis of bibliographies, to serendipitous encounters
and suggestions and advice by colleagues (Greenhalgh and Peacock, 2005).
In recent years in particular so called systematic or structured approaches have gained popularity in IS
(Boell and Cecez-Kecmanovic 2015). As part of this trend, we see an increase in literature reviews that
concentrate their efforts in identifying earlier IS research on the Bo8 and leading conferences in the
field: e.g. “we restrict our scope to the eight journals included in the AIS Senior Scholars' Basket of
Journals” (Schuster et al. 2018, p. 6); “we focused on the widely acknowledged AIS Senior Scholars’
Basket of Journals” (Winkler and Wessel 2018, p. 4); “we focused on articles published in the basket of
eight” (Diederich et al. 2019, p. 1553). However, focusing on only few leading outlets for literature
identification is problematic as Larsen et al. (2019) highlight:
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
“Like the blind men, we may focus on the voices of manuscripts at an arbitrary set of ‘top
journals’, which often eliminate manuscripts that challenge dominant theory, have unsupported
hypotheses or nonsignificant findings and thus did not clear the hurdles of the small set of
editors and reviewers at top journals.” (Larsen et al. 2019, p. 888)
Accordingly, much relevant existing knowledge may not be published in top tier journals such as the
Bo8. Nonetheless, findings of such research are relevant and should inform the generation of new
knowledge. While much is to be learned from an exceptional piece of research published in a top tier
journal, other journals also make important contributions to the knowledge of the field. One may even
go as far as arguing that an inward gaze and a sole focus on a few leading outlets can, in the long run,
be detrimental to the intellectual development of the discipline as controversial and unusual ideas are
sidelined in the academic discourse (Boell and Hovorka 2019; Larsen et al. 2019). However, such ideas
can drive innovative approaches and new thinking about phenomena (Alvesson and Sandberg 2014).
Besides the problem of literature reviews taking a too narrow focus, researchers also learn more about
the existing literature as they engage in the literature review process (Boell and Cecez-Kecmanovic
2014). Bates (1989), Marchionini (2006), Savolainen (2017) and others have thus emphasised the
exploratory nature of literature searches. As He et al. (2019) put it
“Searchers pick up pieces of information as they navigate through an information space. In this
evolving process, searchers acquire new knowledge and change their perception of search tasks
through their interaction with the information space” (p. 844).
For instance, as researchers engage with the existing literature they will discover different theories,
concepts, methodologies, frameworks, models, or constructs, associated with specialised vocabulary or
phrases that then can be used to further refine the literature identification process (Boell and Cecez-
Kecmanovic 2014). However initially, when researchers delve into the literature discovery process,
they are often not familiar with such specific vocabulary useful for locating highly specialised
literature. At these points, researchers often need to use general vocabulary as search terms in order to
describe the phenomena of interest to them. However, when searching with general vocabulary in a
large database often many irrelevant documents will be retrieved. Our research is thus motivated by
two aspects:
(1) The first motivation of our research is to addressing the problem of literature reviews that
are taking a too narrow focus when considering earlier published research, which we call the
insufficiency effect in the coverage of existing literature.
(2) The second motivation for our research is to address the need of researchers to interactively
manage the amount of search results retrieved in response to a query in order to deal with what
we call the impracticality effect.
3 What is an IS Journal? — Identifying a Set of IS Journals
IS as a discipline has many interests thus making it often unclear where IS research begins and where
it ends. This, however, can make it difficult to identify what journals outside of the Bo8 are publishing
research of relevance to IS. Before building a search artifact we thus first needed to identify a suitable
set of sources through which a literature search should be conducted. We decided to focus on journal
articles as a starting point since they are generally considered the primary research output of
completed IS research projects. We further believe that existing listings of IS journals offer a suitable
starting point for our purpose.
In order to identify journals that are of current interest to IS we decided to use any listing or ranking of
IS journals published in 2008 or later. For those interested in earlier journal rankings see, for
instance, Gillenson and Stefford (2008) or Walstrom and Hardgrave (2001). Overall, we found eleven
such journal lists from which we identified journals (Table 1). Importantly, our focus here is not on the
alleged quality of journals (Lewis et al. 2007; Lowry et al. 2013) or the diversity of editorial boards
(Burgess es al. 2017; Cabanac 2012), but simply on the journals considered.
One well-known journal list is the one published by the Association for Information Systems (AIS
2011). In addition to the Bo8, the AIS listing also includes journals nominated by various special
interest groups (SIGs) of the AIS. For instance, Information and Organization and Decision Support
Systems are nominated by multiple SIGs as one of their top 5 journals. All together the AIS thus
introduces a list of 55 unique journals receptive to IS research (AIS 2011). Beyond the AIS list there are
other reputable lists of IS journals. Some of these listings are used for research assessment purposes,
such as the Academic Journal Guide by the UK-based Chartered Association of Business Schools
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
(CABS 2018) and the listing of 177 IS journals by the Australian Business Deans Council (ABDC 2016).
In addition, the London School of Economics (LSE) has published an IS journal list (Willcocks et al.
2008); Holsapple (2009) provides a list of 43 journals based on publication behaviour at US public
universities; Chan et al. (2015) used a citation based analysis for identifying IS journals; and Stewart et
al. (2017) build a list of IS journals based on the WoS journal category for Computer
Science/Information Systems. Furthermore, Scopus also provides three different listings of IS
journals. Finally, John Lamp (2004) maintains a listing of IS journals on his website that is regularly
updated since 1995 and well recognised within the IS community. For the purpose of our research we
only focused on journals listed by Lamp (2004) as active.
Table 1. Overview of IS journal lists.
Listing Description # of journals
(% unique)
(1) ABS –
Ranking journals from 4* (best) to 1. Aims to show “where the best work in
their field tends to be clustered”. Based upon peer review, editorial and
expert judgements and citation statistics.
(2) ABDC /
Ranking journals from A* (best) to C. The listing for IS is identical to the
listing provided by the Australian Council of Professors and Heads of
Information Systems (ACPHIS).
(3) AIS Journals nominated by the AIS “college of senior scholars” including
journals nominated by special interest groups (SIGs) as “worthy of broader
AIS recognition extending beyond the basket of 8” (AIS 2011).
(4) WoS
Based on the WoS category ‘Computer Science and Information Systems’,
extended with additional IS journals in order to identify “a large enough
sample to reflect the diversity of IS journals” (Stewart et al 2017, p. 549).
(5) Citation
Citation-based analysis with the aim to develop “an objective method to
identify a set of core journals” (Chan et al. 2015, p. 239).
(6) LSE LSE’s ranking to inform promotions decisions and to “advise junior staff and
students on where they should publish.” (Willcocks et al. 2008, p. 163)
(7) US Public
Based on the publications of “106 full-time, tenured IS faculty members at
[...] top public research universities” in the US (Holsapple 2009, p. 122).
(8) Scopus IS
In addition, we added three Scopus “subject areas” designated IS: First,
‘Computer Science – Information Systems’, oriented towards technical
perspectives on Information Systems.
(9) Scopus
MIS List
As above, for: ‘Business – Management Information Systems’, oriented
towards the use of Information Systems for business management.
(10) Scopus
ISM List
As above, for: ‘Decision Sciences – Information Systems and Management’,
oriented towards the use of IS to assist decision-making.
(11) John
Lamp’s List
Since 1995 John Lamp actively maintains an Index of Information Systems
Journal. The aim of the index is to be exhaustive in its coverage. For our
purpose we chose to include all journals listed as active (‘current’).
Taken together, the eleven lists summarised in Table 1 identify 1,042 different journals. Out of the
eleven lists, ten contain journals that are unique to only one list leading to a list of 714 journals unique
to a single journal listing (Table 1). Thus, when assessing journals as to how often they appear across
different journal lists, we see a strong concentration pattern as per Table 2 below. This skewed
distribution can be expected as it is in line with what can be observed for most bibliometric
phenomena (Wilson 1999): skewed distribution patterns are commonly observed when looking at
authors publishing on a topic, publications on a given topic across journals or citations across
publications. A possible explanation for the skewed distribution of journals across lists is that they are
published by bodies with differing expertise and research orientation, using various methodologies for
1 John Lamp’s list has 684 entries marked ‘current’; however, of these, 3 are internally duplicated:
Information, Communication and Society (appears twice); International Journal of Information
Communication Technologies and Human Development (appears twice); International Journal
of Technology Diffusion (appears twice).
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
identifying IS journals. Thus, while there is agreement on core IS journals, IS as a reference discipline
(Baskerville and Myers 2002) is connected to the interests of diverse related communities. These
communities will be reflected differently across journal lists and hence the composition of journals in
different listings will vary. For instance, Burgess et al. (2017) argue for the existence of sub-
communities within IS such as E-commerce, Human-Computer Interaction, Management IS, among
others. Likewise, the range of SIGs within the AIS and the range of journals they nominate suggest that
there is a range of journals receptive to different types of IS research.
Table 2. Appearance of journals across 11 IS journal listings and their coverage in Scopus.
# of lists
journal is
listed in
11 10 9 8 7 6 5 4 3 2 1 Total
# journals
in this set 5 1 1 5 4 13 22 37 72 168 714 1,042
% of total 0.5% 0.1% 0.1% 0.5% 0.4% 1.2% 2.1% 3.6% 6.9% 16% 69% 100%
# indexed
by Scopus 5 1 1 5 4 13 22 36 67 148 545 847
% of set
indexed ALL ALL ALL ALL ALL ALL ALL 97% 93% 88% 76% 81%
Based on the skewed appearance of journals across listings, we see the emergence of a ‘core’ of IS
journals appearing across the majority of lists. For instance, besides the Bo8 only four journals are
listed in 8 or more rankings (see figure 1 below). We therefore argue that a list of 51 journals appearing
in five or more lists would constitute a suitable set for embarking on initial exploratory literature
searches. As the review process progresses or if insufficient number of results are retrieved IS
researchers may want to ‘cast a wider net’, hence literature searches can be further expanded into
journals appearing across fewer journal lists.
Figure 1. IS journals appearing in 5 or more lists including years covered in Scopus.
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
4 Building a Search Artifact
To realise the vision of an IT artifact for literature searches, we needed to connect to an existing
literature database covering the majority of the journals identified as publishing research of relevance
to IS. Our first attempt, based on Google Scholar, was of limited success. While Google Scholar allows
users to manually specify the journal to target for a particular search it does not allow this journal filter
to compound. We would therefore need to generate a different results page for each journal searched.
As many research universities provide their staff with access to Scopus we thus saw an opportunity to
leverage the existing Scopus advanced search functionality for building a tool that enables targeted
searches for IS. Advanced searches in Scopus allow the use of a retrieval language which adds
compounding conditions to searches beyond keywords. For instance, all resources indexed in Scopus
have a unique SOURCE-ID. Thus, an advanced search for ‘SOURCE-ID(12402)’ will identify all
articles published in MIS Quarterly that are indexed by Scopus. When the SOURCE-ID is combined
with keywords one is thus able to search through the title, keywords and abstracts of articles published
in MIS Quarterly. Furthermore, additional conditions can be included in searches such as document
types in order to search only for literature review articles or editorials or year of publication. The
ability to search for review articles and editorials are particularly useful for exploratory searches.
Literature reviews enable the identification of publications that offer an overview of earlier research. A
search through editorials is useful for identifying special issues on particular topics.
To proceed, we first matched the complete set of all journals identified by us (see Tables 1, 2) to their
respective SOURCE-ID registered in Scopus. In the case of John Lamp’s list a SCOPUS-ID was already
provided for most journals for other lists used identifying features such as ISSN or journal titles. This
process was facilitated by a relational database system (mySQL) in which we captured other useful
attributes such as Scopus coverage years and journal URL, where applicable. Of the 1,042 unique
journals identified during the discovery phase, 847 are indexed by Scopus (81%, see Table 2). As
shown in Figure 1, all 51 journals included in 5 or more lists are currently indexed by Scopus with
coverage in many cases is also going back several decades. Using the relational database of SOURCE-
ID for IS journals we developed a web-based graphical user interface accessible at in
which users can generate search strings for Scopus’ advanced search and customise the selection of
journals. IS researchers can thus use our tool to generate elaborated search strings for Scopus
advanced search that enables simultaneous search in up to 847 journals publishing research or interest
to IS. All users have to do is copy the search string into Scopus’ advances search window (Figure 2).
Figure 2. Transferring Scopus Query String into Scopus Advanced Search.
Furthermore, if users are accessing Scopus from within an organisational network that has access to
Scopus (e.g. on campus or via VPN) search queries can be directly performed in Scopus. Since we
established our own relational database containing Scopus SOURCE-IDs, we can directly generate
predictive URLs for specific Scopus searches. As URL length are capped, direct searches work only for
sets of ~52 journals. However, as our artifact generates complete search strings using the correct
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
syntax for Scopus’ advanced search, simultaneous searching across all 847 journals indexed by Scopus
is possible by copying the generated search string directly into Scopus (Figure 2).
Our codebase is open-source and available on GitHub2 to facilitate the development of our search
artifact by the IS community. Our search artifact is tentatively named Litbaskets (short for Literature
Baskets) and deployed at It has the following features:
‘Search’: Landing page with a search box and instructions. The instructions encourage the user
to think about the nature of their search, and on that basis, make adjustments using the
configuration pages (see below). If the user chooses to proceed with the search without
customising the settings, the search is performed on our default core set of 51 journals with the
user-entered search terms matched against the title, keywords and abstract fields in Scopus.
Upon submitting their search and if fewer than 52 journals are searched, the user (assuming
they have access to Scopus) is redirected to a Scopus search results page in a new browser tab.
In addition, a message appears on with the relevant query string that can be
pasted directly into Scopus’ advanced search if needed (see Figure 2), and options to execute
the same search directly on AISeL or the bibliographic library. (Although
relates to Computer Science and not IS, we provide the option for the cases where the research
topic is sufficiently technical to make use of Computer Science literature.)
‘1. Choose Litbasket’: Configuration page that allows the user to select a ‘litbasket’, which is
simply a predefined set of journals that the search can be performed on. The default litbasket
is our ‘core’ set of 51 journals, which we call ‘L5’ (mentioned in 5 or more of our reference lists,
as per Figure 1). Other litbaskets based on list counts range from the largest, ‘L1’ (all 847
journals indexed by Scopus) to the smallest, ‘L7’ (only the 16 journals that appear in 7 or more
of our reference lists, as per Figure 1).We also include an even smaller litbasket than L7 which
is the ‘Bo8’ option so that users can, if they deem appropriate, still simply just search through
the AIS Basket of Eight.
‘2. Customise Sources’: Configuration page that allows the user to select or deselect individual
journals to make adjustments to the ‘litbasket’ that they have selected. To assist the user to
find a specific journal, this configuration page includes all reference lists from Table 1, as well
as the aggregate of the journals from all lists combined. Thus, users can customise a particular
journal set from all 847 journals based on the specific needs imposed by their research topic.
‘3. Apply Filters’: Configuration page that allows the user to use additional Scopus search
features. For example, in order to return only current research, there is a default filter that
limits search results to only retrieve articles published from 2013 onwards. If older results are
desired, researchers can adjust this filter. In addition, Scopus (unlike Google Scholar) indexes
articles into document types. This enables filtered searches: either only for literature review
articles (for an overview of earlier research and how studies are related to each other); or only
for editorials (for identifying special issues on particular topics).
5 Assessing the Artifact
Two crucial measures for assessing any search system are recall and precision (Salton and McGill
1983). Underpinning both measures is the criterion of relevance, since items retrieved in a search can
either be described as addressing (‘relevant’), or not addressing (‘irrelevant’), what is sought. Recall is
a measure describing how many of all potentially relevant items available in a search system are
retrieved by a search. Ideally, a retrieval system would aim for a recall of 100% to identify all relevant
items available. A search system that always reproduces all items available in response to a query
would, by definition, have a perfect recall. Nonetheless, such a system would be largely unusable due
to the aforementioned impracticality effect.
Therefore, precision is established as a second measure of the exactness of retrieval systems. This
measure assesses how many of the results retrieved by the system are actually relevant. Ideally, a
perfect retrieval system would aim for a precision of 100% if, in response to a query, the system only
showed users documents that are relevant to them. However, in practice, attempts to achieve high
precision result in too few results, as per the aforementioned insufficiency effect. This is the recall-
precision trade-off relationship (Buckland and Gey 1994), wherein an increase in recall generally
comes at the expense of precision, and searches with high precision will generally yield a lower recall.
Figure 3 visualises the recall-precision trade-off, and how it leads to the impracticality effect and the
insufficiency effect.
2 GitHub repository:
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
Figure 3. How manages the recall-precision trade-off.
To exemplify the usefulness of for dealing with both the insufficiency effect (when
too little literature is found) and the impracticality effect (when researchers are overwhelmed with too
many results), we conduct some exemplary searches. Since the actual judgement of whether a paper is
relevant will depend on what a researcher seeks to achieve, we chose to use the number of documents
retrieved as a proxy for judging how encompassing the results retrieved by our artifact are in
comparison to existing alternatives. To do this, we selected four broad search terms that we believe are
representative for what may be initially used by researchers conducting exploratory searches (Table 3).
Table 3. Number of results for different search tools using example searches based on title,
abstract and keywords and publication year > 2013.
Core Set (‘L5’)
Full Set (‘L1’) WoS Google
blockchain 4 154 51 692 2,974 ~24,100
crowdfunding 25 144 77 204 1,302 ~20,400
e-health 2 135 104 830 6,187 ~78,900
outsourcing 63 243 338 1,452 18,338 ~84,300
*Searches in Google Scholar could not be limited to abstracts, titles, keywords only.
As aforementioned, relying on a set of ‘top journals’ is problematic when conducting literature reviews.
Further looking at the number of documents retrieved using titles, abstracts and keywords for the AIS
Bo8, it is seems that for many searches too few documents are retrieved for a comprehensive literature
review. When limiting the search to titles, abstracts and subjects in the AISeL the number of
documents retrieved are more promising. However, looking more closely at the results almost all of
them are conference papers. We thus believe that the AISeL can offer a good additional resource to for identifying conference papers. In contrast, the number of search results retrieved from
Web of Science (WoS) and Google Scholar indicate an impracticality effect as researchers would have
to traverse through thousands of documents of which many will be irrelevant to the research topic.
While the usefulness of the ability for IS researchers to search through a specific set of IS journals
should be evident, the number of results for further exemplifies this. As can be seen
in Table 3, users are able to interactively manage the focus of their literature searches and thereby
control the number of documents retrieved. Our core set (‘L5’) in Table 3 is for searches across
journals listed in Figure 2, i.e., those appearing in the majority of all eleven of our reference lists. As
the range of journals through which a search is conducted increases, so does the number of documents
retrieved. This is illustrated by the increase in the number of documents retrieved for the full set of all
847 journals indexed by Scopus. Thus our artifact allows researchers to manage the recall-precision
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
trade-off as searchers can increase the number of journals if the recall in response to their query is too
low. However, too many documents are retrieved for an initial engagement with the literature
researchers can easily apply filters such as review articles only’ or reduce the number of journals
through which a search is conducted.
6 Discussion
6.1 Contributions
The contributions of our research are related to the bibliometric work underpinning the creation of our
list of IS journals as well as the IT artifact we created. Firstly, our artifact contributes an important
new means for discovering IS knowledge. Many established disciplines have a primary resource for
discovering publications, yet no comparable resource exists for IS. Our tool thus addresses an
important need as discipline-specific literature reference databases fulfil notable functions: (1) They
make literature accessible through searches; (2) They support researchers in making knowledge claims
based on a disciplines discourse by ensure good coverage of established knowledge; (3) They select
specific sources to filter out predatory journals and any sources not central to disciplinary knowledge.
Moreover, by enabling a search through an extensive set of journals in which research of relevance to
IS may be published, we enable IS researchers to improve the effectiveness of their literature searches.
Recall and precision and therefore the effects of insufficiency as well as the impracticality are related to
both the range of sources considered for a literature search as well as the search terms chosen for the
search. We manage and optimise precision by reducing the sources considered for searching, from the
broad coverage offered by large multidisciplinary sources such as Google Scholar or WoS to a more
targeted and specific set of journals. Furthermore, Scopus further supports the literature identification
process as it allows researchers to employ elaborate search strategies such as successive fractions or
snowballing through its ‘refine search’ function (Boell and Cecez-Kecmanovic 2014). Hence,
researchers are less likely to be overwhelmed by huge numbers of irrelevant documents. We also
manage and optimise recall, as searches through a more targeted set of journals enable a more
thorough search within this set, for instance, by using single word searches or common language in
addition to specialised terminology. Hence, researchers are less likely to miss relevant publications as
they can use more inclusive search terms in their searches without risking that too many irrelevant
documents (‘false positives’) are retrieved.
Our tool therefore is useful to IS researcher in three ways: (1) By enabling targeted searches and filters,
we empower researchers to quickly identify a highly relevant article to be added on top of their reading
list. Based on the reading of such an article, researchers discover further relevant terminology,
constructs, concepts, theories, models, etc. that then inform further literature searches (Boell and
Cecez-Kecmanovic 2014); (2) By using our tool, researchers can also compile lists of relevant articles
related to a topic that provide them with an overview of what is well known and researched within the
wider IS community, thus better frame the contribution of their work to the disciplinary discourse
(Boell and Hovorka 2019); (3) By enabling a targeted search, our tool supports editors by facilitating
identification of potential reviewers.
Finally, the methodology we used for creating the journal list demonstrates that skewed distributions
underpinning bibliometric phenomena also apply to how frequently journals are mentioned across
journal listings. In other words, inclusion of journals in journal lists follows a similar distribution
pattern as, for instance, citations to articles. This finding has implications for journal analysis in
bibliometric studies more widely as it questions the validity of studies underpinned by a single journal
list. For instance, Stewart et al. (2017) make important observation about the cohesion of the IS
discipline. Yet when looking at the set of journals used for the analysis, 30% of the 98 journals did not
appear in any of the other ten journal listings identified by us. Likewise, Burgess et al. (2017)
exploration of the diversity of editorial boards in IS journals was based on the CABS journal listing.
However, the CABS ranking not only includes 6 journals not mentioned in any of the other journal
lists, it also excludes most journals of our journal set of 51 journals appearing in the majority of all
journal lists. Hence our research encourages caution when looking at findings from earlier bibliometric
studies not only for IS, but for bibliometric studies related to journals in general.
6.2 Limitations and Next Steps
Although our search tool enables most IS researchers to begin comprehensive literature reviews on
most IS topics, it is not perfect and has some limitations. Firstly, users must have access to Scopus.
Secondly, while we identified an extensive set of IS journals, depending on the research undertaken
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
additional journals should be searched. Thirdly, our search tool enables searching though journal
article and excludes other publications such as conference proceedings or book chapters. These
limitations do, however, have mitigating factors. Firstly, Scopus access is very common. We inspected
the institutional login page for Scopus and found that, for example, every university that has ever
hosted the Australasian Conference on Information Systems (ACIS) currently has institutional access
to Scopus (Scopus 2019). Secondly, we argue that by starting with journal papers – i.e., documentation
of completed research projects in their first literature review iteration, IS researchers can have a
better appreciation for conference papers that they discover in later iterations. Furthermore our
artifact can be paired with AISeL that offers much better coverage of IS conference, but lacks coverage
of IS journals. We thus recognise the value of potentially including additional databases in future to
reduce the high level of reliance on Scopus, and we are currently investigating the technical feasibility
of connecting to APIs available for and Google Scholar.
One of the reasons for making our codebase entirely open-source is to encourage other IS academics to
contribute, such that the project as it is ‘by the community, for the community’. We take inspiration
from other significant open-source projects started in academia and a view that the IS community has
a critical mass of members who possess the programming skills required to participate. For instance,
we would welcome an initiative to extend the coverage of to also include the
proceedings of leading IS conferences in its searches.
7 Conclusion
To make a meaningful contribution, research needs to be framed in relation to the prevailing discourse
of an academic community (Boell and Hovorka 2019; Hovorka and Boell 2015). The literature
identification process is central for understanding share knowledge and hence for expressing new
knowledge claims. However, unlike Medline for Medicine or Psycinfo for Psychology, there is currently
no database that IS researchers can go to as a starting point for conducting exploratory literature
searches, leading to either too little literature to review (the insufficiency effect) or too much literature
to review (the impracticality effect). Our study addresses this deficit by identifying a set of journals
that can be considered as receptive to IS research. By building as an interface for
searching through these journals we hope to not only make literature searches more effective for IS
researchers, but also to facilitate the IS discipline uniting around a core of shared knowledge.
8 References
Alvesson, M., and Sandberg, J. 2014. “Habitat and Habitus: Boxed-in versus Box-Breaking Research”,
Organization Studies (35:7), pp. 967–987.
Association for Information Systems (AIS) 2011. “Senior Scholars’ Basket of Journals” [online].
Australian Business Deans Council (ABDC) 2016. “ABDC Journal Quality List” [online].
Baskerville, R.L., and Myers, M.D. 2002. “Information Systems as a Reference Discipline”, MIS
Quarterly (26:1), pp. 1–14.
Bates, M.J. 1989. “The design of browsing and berrypicking techniques for the online search interface”,
Online Review (13:5), pp. 407–424.
Boell, S.K., and Cecez-Kecmanovic, D. 2014. “A Hermeneutic Approach for Conducting Literature
Reviews and Literature Searches”, Communications of the AIS (34:1), pp. 257–286.
Boell, S.K., and Cecez-Kecmanovic, D. 2015. “On Being ‘systematic’ in Literature Reviews in IS”,
Journal of Information Technology (30:2), pp. 161–173.
Boell, S.K. and Hovorka, D.S. 2019. “Writing, Arguing, Contributing - A Cogent Argumentation
Framework for Identifying, Specifying, and Evaluating Research Contribution”, Australasian
Journal of Information Systems (23). 10.3127/ajis.v23i0.1857.
Buckland, M., and Gey, F. 1994. “The relationship between Recall and Precision”, Journal of the
American Society for Information Science (45:1), pp. 12–19.
Burgess, T.F., Grimshaw, P., and Shaw, N.E. 2017. “Research Commentary—Diversity of the
Information Systems Research Field: A Journal Governance Perspective”, Information Systems
Research (28:1), pp. 5–21.
Cabanac, G. 2012. “Shaping the Landscape of Research in Information Systems From the Perspective
of Editorial Boards: A Scientometric Study of 77 Leading Journals”, Journal of the American
Society for Information Science and Technology (63:5), pp. 977–996.
Chan, H.C., Guness, V., and Kim, H.-W. 2015. “A Method for Identifying Journals in a Discipline: An
Application to Information Systems”, Information and Management (52:2), pp. 239–246.
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
Chartered Association of Business Schools (CABS) 2018. “Academic Journal Guide 2018” [online].
Diederich, S., Brendel, A.B., and Kolbe L.M. 2019. “On Conversational Agents in Information Systems
Research: Analyzing the Past to Guide Future Work”, Proceedings of the 14th International
Conference on Wirtschaftsinformatik, p. 1550-1564.
Gillenson, M., and Stafford, T. 2008. “Journal Rankings 2008: A Synthesis of Studies”, Americas
Conference on Information Systems (AMCIS), Paper 58.
Greenhalgh, T., and Peacock, R. 2005. “Effectiveness and efficiency of search methods in systematic
reviews of complex evidence: audit of primary sources”, British Medical Journal (331:7524), pp.
He, J., Ping, Q., Lou, W., and Chen, C. 2019. “PaperPoles: Facilitating Adaptive Visual Exploration of
Scientific Publications by Citation Links,” Journal of the Association for Information Science
and Technology (70:8), pp. 843-857.
Holsapple, C.W. 2009. “A new map for knowledge dissemination channels”, Communications of the
ACM (52:3), pp. 117–125.
Hovorka, D.S., and Boell, S.K. 2015. “Cogency and Contribution in IS Research”, International
Conference on Information Systems, Fort Worth (USA), Paper 1574.
Lamp, J. W. 2004. Index of Information Systems Journals. Geelong, Deakin University [online].
Larsen, K.R., Hovorka, D.D., Dennis, A.R., and West, J.D. 2019. “Understanding the Elephant: The
Discourse Approach to Boundary Identification and Corpus Construction for Theory Review
Articles”, Journal of the AIS (20:7), pp. 887-916.
Lewis, B., Templeton, G., and Luo, X. 2007. “A Scientometric Investigation into the Validity of IS
Journal Quality Measures”, Journal of the AIS (8:12), pp. 619–633.
Lowry, P.B., Moody, G.D., Gaskin, J., Galletta, D.F., Humpherys, S.L., Barlow, J.B., and Wilson, D.
2013. “Evaluating Journal Quality and the Association for Information Systems Senior Scholars’
Journal Basket Via Bibliometric Measures: Do Expert Journal Assessments Add Value?”, MIS
Quarterly (37:4), pp. 993–1012.
Marchionini, G. 2006. “Exploratory search: From finding to understanding”, Communications of the
ACM, (49:4), pp. 41–46.
Paré, G., Trudel, M.-C.C., Jaana, M., and Kitsiou, S. 2015. “Synthesizing Information Systems
Knowledge: A Typology of Literature Reviews,” Information and Management (52:2), pp. 183–
Salton, G., and McGill, M.J. 1983. Introduction to modern information retrieval, New York: McGraw-
Savolainen, R. 2017. “Berrypicking and information foraging: Comparison of two theoretical
frameworks for studying exploratory search”, Journal of Information Science (43:3), pp. 1–14.
Scopus 2019. Login via your Institution [online],
Schuster, R., Wagner, G., and Schryen, G. 2018. “Information Systems Design Science Research and
Cumulative Knowledge Development: An Exploratory Study”, International Conference on
Information Systems, San Francisco (USA).
Stewart, A., Cotton, J. L., and Adya, M. 2017. “Information Systems: A House Divided?”,
Communications of the AIS (41:24), pp. 544–586.
Tate, M., Furtmueller, E., Evermann, J., and Bandara, W. 2015. “Introduction to the special issue: The
literature review in information systems”, Communications of the AIS (37), Paper 5.
vom Brocke, J., Simons, A., Riemer, K., Niehaves, B., Plattfaut, R., and Cleven, A. 2015. Standing on
the shoulders of giants: Challenges and recommendations of literature search in information
systems research, Communications of the AIS (37), Paper 9.
Walstrom, K.A., and Hardgrave, B.C. 2001. “Forums for information systems scholars: III,”
Information and Management (39:2), pp. 117–124.
Webster, J., and Watson, R.T. 2002. Analyzing the past to prepare for the future: Writing a literature
review. MIS Quarterly (26:2), pp. 13-23.
Willcocks, L., Whitley, E. A., and Avgerou, C. 2008. “The ranking of top IS journals: A perspective
from the London School of Economics”, European Journal of Information Systems (17:2), pp.
Wilson, C.S. 1999. “Informetrics”, Annual Review of Information Science and Technology (34), pp.
Winkler, T. J., and Wessel, M. 2018. “A Primer on Decision Rights in Information Systems: Review
and Recommendations”, International Conference on Information Systems, San Francisco
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
Copyright: © 2019 Boell & Wang. This is an open-access article distributed under the terms of the
Creative Commons Attribution-NonCommercial 3.0 Australia License, which permits non-commercial
use, distribution, and reproduction in any medium, provided the original author and ACIS are
... To retrieve literature, we employ the AIS eLibrary and (Boell and Wang 2019). The former offers IS conference articles. ...
... Among others, this includes the AMCIS 2023 proceedings that are upcoming at the point of writing this article. The latter provides access to 847 journals that are relevant to IS research (largest basket according to Boell and Wang 2019). Thus, we source articles from a wide range of journals and conference proceedings that are pertinent to the field of Information Systems. ...
Conference Paper
Full-text available
Climate change causes ever more natural hazards. In such crisis situations, individuals seek for information about the situations constantly. Due to its ever-growing relevancy in the everyday life, social media are increasingly used to seek and discuss crisis-related information. Social media platforms offer the possibility to deploy social bots (i.e., automated user accounts) that are partly credited to be used for malicious purposes but also considered useful by emergency management agencies (EMAs) to disseminate situational updates and information. Many tasks for which EMAs employ social bots rely on the publication of information in textual form. Recent advancements of generative artificial intelligence (GenAI) offer means to generate texts, images, and videos automatically. Therefore, we explore how social bots can benefit from GenAI in crisis communication. To this end, we conduct a systematic literature review and offer a research agenda to guide future endeavours.
... We decided to include bibliographic databases specified to journals and conference proceedings because in fastmoving fields such as IS, conferences are crucial in the research dissemination process. Consequently, we selected basket "M" on, which covers 51 journals relevant for IS (Boell and Wang 2019). In addition, we included the Association for Information Systems eLibrary (AISeL) database and limited the search to conferences. ...
Conference Paper
Full-text available
Social media platforms such as Facebook, Twitter/X, or TikTok are controlled by and rely on a single service provider. However, repeated controversies around data breaches, algorithmic bias, or misinformation have eroded trust in these centralised platforms. Concurrently, advancements in blockchain technology have opened avenues for decentralised social media that distribute control and data across a network of participants, thereby challenging their centralised counterparts. This study reviews existing literature on the topic and finds that what we know largely focuses on the technological implementation of decentralised social media. Therefore, we surveyed the literature for socio-technical dimensions of the phenomenon and found that control, privacy, operation, security, rewards, adoption, and research are central themes in this regard. We use those themes to derive recommendations for future research in Information Systems (IS) and beyond that aims to unpack individual, organisational, and societal implications of decentralised social media.
... Our journal list comprised the AIS basket of top IS journals, all high-quality IS journals identified by (Boell & Wang, 2019), a range of top-and high-quality business journals in management and organization studies, as well as an inclusive range of interdisciplinary journals from computer science, psychology, and marketing. ...
Conference Paper
Full-text available
This paper problematizes the literature on the non-use of algorithmic decision-making systems (ADMS), commonly examined as algorithm aversion. Whilst prior literature attributes algorithm aversion primarily to human bias and irrationality, assuming utility-based evaluations, we argue that it may also stem from values-based evaluations of technology, which are overlooked. Through an integrated ethical analysis, drawing upon the "big three" ethical theories of consequentialism, deontology, and virtue ethics, we examine implicit normative judgments within the algorithm aversion literature. Consequently, we positively reframe algorithm aversion as a potentially principled resistance to ADMS, expanding prior views of the phenomenon. We argue that such resistance may be constructive and lead to a better alignment of ADMS with societal needs and values. Thus, we call on IS scholars to explore this phenomenon as an ethical and sociotechnical issue, rather than as a costly problem to be mitigated, as prior literature might suggest.
... But it has been used scarcely and selectively in IS research, as evidenced by just 12 IS publications in reputable journals citing the framework at all. We used Scopus to search for articles citing Nicolini et al. (2012) in the 51 renowned journals suggested by Boell and Wang (2019). These journals do not only include the AIS Basket of widely regarded IS journals, but also top journals in management and organization studies that publish IS research, such as Organization Science. ...
Full-text available
Information systems development (ISD) requires dynamic and flexible ways of working, particularly when developing requirements in collaboration with customers. Although prior research has acknowledged the importance of objects to support ISD practices, there has been a lack of frameworks to help discern the multiple overlapping roles objects play to support requirements development in a variety of ways throughout an ISD project. This paper explores and theorizes this phenomenon by leveraging multiplexity as a theoretical lens to analyze an extensive qualitative data set from a case study at a Swiss multinational banking software provider. Results show how diagrams and prototypes both play the roles of epistemic, activity, boundary, and infrastructure objects as a reflection of how they are used in requirements development. Our analysis articulates how two classical requirements specifications play multiple overlapping roles to support dynamic and flexible ISD practices. Based on these findings, we advance a framework for discerning the multiplex role of objects in practice.
... Furthermore, as the literature on both blockchain and DRM systems is in a nascent stage, we used snowballing (Boell & Cecez-Kecmanovic, 2014) to find further relevant literature in related contexts (e.g., DRM systems in other media industries). In line with our deepening understanding of the literature and empirical data, we periodically screened the literature throughout the 3-year duration of the research project using the query (("digital rights management" OR "digital music rights management") OR (blockchain AND music)) to search in 847 relevant journals with the help of litbaskets (Boell & Wang, 2019). In all, we identified 19 relevant papers on digital music rights management and 14 relevant papers on blockchain (as listed in the right-most column of Table 1), which we used to strengthen our design requirements and design principles, respectively. ...
Full-text available
Initially designed to protect intellectual property (IP) of digitalized information goods such as music, games, or books, existing centralized digital rights management (DRM) systems mostly serve the interests of major publishers, with scant inclusion of rights owners, creators, and consumers. Although various blockchain-based DRM systems have been proposed, most of them mirror existing counterproductive IP restrictions. Analyzing the music industry as a case in point, this paper proposes design principles for blockchain-based DRM systems that provide an integrated and flexible solution by enabling transparent music licensing structures, consistent and complete rights metadata, and efficient and transparent royalty payout. The solution can be achieved by storing rights metadata on a public distributed ledger, by validating metadata through a consensus mechanism on a permissioned blockchain, and by algorithmically enforcing royalty payout via stablecoin through a smart contract. The design principles were evaluated by industry experts, validating their benefit for the music industry by increasing surplus value that is currently destroyed through previous suboptimal designs.
... In particular, we used the website (Boell and Wang 2019) to perform a keyword-based search covering the 51 most prestigious IS journals (referred to as "Basket M" on the website). Moreover, we also searched the proceedings of leading IS conferences (ICIS, ECIS, AMCIS, PACIS, HICSS, and ACIS). ...
Conference Paper
Full-text available
With companies and other organizations increasingly striving to become (more) data-driven, there has been growing research interest in the notion of a data-driven organization (DDO). In existing literature, however, it seems that different understandings of such an organization emerged. The study at hand sets forth to synthesize the fragmented body of research through a review of DDO understandings in the IS and related literatures. Based on the review results and drawing on the concept of the 'knowing organization,' our study identifies five core elements of a DDO (data sourcing & sensemaking, data capabilities, data-driven culture, data-driven decision-making, and data-driven value creation), which we integrate into a conceptual DDO framework. Most notably, the proposed framework suggests that an ideal-typical DDO combines an outside-in view with an inside-out view, thereby setting itself apart from its predecessor, the knowing organization. Theoretical and practical contributions as well as directions for future research are discussed.
... The IS relevance of the identified outlets was defined by using the platform which provides an interface to filter an advanced Scopus search for IS relevant journals (Boell and Wang 2019). To avoid that a too narrow focus on the selected outlets weakens the generalizability of our findings, we selected the largest basket available on the website that includes 847 essential IS journals (to see a list of the journals selected, please visit the ...
Conference Paper
Full-text available
Applications of digital twin (DT) technology have gained momentum in IS research and cognate disciplines. Several studies have documented how DTs create value in contexts such as manufacturing or smart cities through virtual monitoring and decision-making. While these contexts benefit from DTs of products or production steps, this research is the first to investigate the potentials of human DTs to improve customer experience (CX) (i.e., customer twins). Drawing on a structured literature review, we derive new conceptualizations of DTs as (i) virtual mirrors that depict a physical entity and its interactions in virtual space, and (ii) virtual orchestrators which extend the virtual mirror by also simulating potential virtual interactions. These new conceptualizations, by applying them to human DTs, enable us to discuss DT's implications to approach current CX potentials. The results of the discussion indicate that human DTs can support CX management to improve CX throughout the whole customer journey.
Full-text available
Las revisiones de la literatura para tareas de investigación y docencia se enfrentan cada vez más a nuevos retos, debido principalmente al crecimiento considerable del material bibliográfico en todas las disciplinas. Esencialmente, la búsqueda y selección de información se tornan exhaustivas y agotadoras debido a la falta de organización en la recopilación sistemática de documentos, así como el manejo incorrecto de los metadatos y la gran variabilidad en los resultados que ofrecen los mecanismos automáticos de consulta. En este sentido, la inteligencia artificial (IA) está revolucionando la revisión de la literatura al emplear sistemas capaces de analizar e interpretar el contenido de muchos documentos en una forma rápida, así como ofrecer resultados con un nivel de precisión mayor que los motores de búsqueda tradicionales. Este trabajo tiene como objetivo describir los avances recientes en la etapa de acoplamiento inteligente de los métodos para la revisión de la literatura académica, refiriendo al uso de herramientas de aprendizaje automático y aprendizaje profundo asociadas a la optimización en el procesamiento del lenguaje natural. Finalmente se abordan las implicaciones que conlleva emplear sistemas capaces de omitir juicios y tomar decisiones con respecto a la utilidad y relevancia de los datos analizados.
Conference Paper
Full-text available
As Cybersecurity continues to have a significant impact on modern society, there is a pressing need for a more comprehensive research agenda in Information Systems (IS). In this study, we conducted a thorough literature review of prominent IS journals to identify gaps in Cybersecurity research practices. Our findings indicate that there is a significant gap between research and practice, particularly in terms of focus on Cybersecurity behavioural factors in the past decade. To address this gap, we recommend that future Cybersecurity research in IS should adopt a broader perspective that incorporates relevant sociotechnical knowledge areas and theories. We provide an example of Cybersecurity research topics that go beyond behavioural aspects and suggest mapping of Cybersecurity sociotechnical research knowledge areas in Information Systems to guide future research efforts. This study highlights the importance of broadening the scope of Cybersecurity research in IS to address the complex Cybersecurity challenges in contemporary practice.
Full-text available
This study determined factors affecting the security of Information Systems in Africa. Also, it established the quality of publications in the area of Information Security. The study is based on peer-reviewed publications conducted in Africa. The study adopted the mixed research approach. The study used a systematic literature review, with part of the analysis using descriptive analysis. In total, 70 papers formed the population of publications extracted in the area of Information Security. In addition, 37 papers had the quality to be included in the analysis of factors affecting Information Security. The study found that information Security factors are in four key categories: human factors, policy-related issues, work environment, and demographic factors. Overall, the work environment is the most reported category affecting the security of Information Systems in Africa. In addition, gender is the highest reported individual factor associated with the insecurity of Information Systems; female is the highly affected gender. Other factors include the lack of Information Security training, the unchecked level of trust, carelessness and poor security policies. The study recommends training programs, policy improvement and promoting behaviours that minimise exposure to attacks.
Full-text available
The predominant means by which research becomes visible and accessible to the research community is through publication. Generally, publication requires careful framing of the research in relation to existing knowledge. As a contribution to knowledge cannot be self-evident, authors must persuade, through argumentation, the editors, reviewers, and the research community that their work offers a contribution. In Information Systems, the discussion of argumentation is often limited to the logic dimensions of argumentation, namely deductive, inductive, and abductive reasoning. In this paper, we demonstrate that argumentation requires the consideration of three additional dimensions of argumentation: rhetoric, dialectic, and social-institutional. Kuhn’s concept of the disciplinary matrix is introduced as the background toward which a cogent argument is directed and against which contribution is evaluated. We then illustrate the role of argumentation through the example of the seminal paper by Orlikowski and Iacono on the role of IT in Information Systems research. Understanding the importance of argumentation in framing one’s research contribution is critical to authors, editors, and reviewers alike within and beyond Information Systems and its reference disciplines.
Conference Paper
Full-text available
Conversational agents (CA), i.e. software that interacts with its users through natural language, are becoming increasingly prevalent in everyday life as technological advances continue to significantly drive their capabilities. CA exhibit the potential to support and collaborate with humans in a multitude of tasks and can be used for innovation and automation across a variety of business functions, such as customer service or marketing and sales. Parallel to the increasing popularity in practice, IS researchers have engaged in studying a variety of aspects related to CA in the last few years, applying different research methods and producing different types of theories. In this paper, we review 36 studies to assess the status quo of CA research in IS, identify gaps regarding both the studied aspects as well as applied methods and theoretical approaches, and propose directions for future work in this research area.
Conference Paper
Full-text available
The decision rights construct deserves increasing attention in the Information Systems (IS) field, yet integrative knowledge is sparse. This review first establishes two key distinctions from managerial theory regarding decision rights classes (decision input, control, and management) and manifestations (formal versus practiced) and then analyzes the current state of the IS literature based on 53 articles published in nine leading IS outlets between 1992 and 2018. Our review identifies six themes of IS research that have used the decision rights construct on various levels, with differing conceptualizations, and in different nomological roles. We find that the majority of studies mix decision control with decision management rights (38%) and focus on formal manifestations only (68%). We close by putting forward five recommendations, arguing that a more nuanced conceptualization of decision rights will aid future IS research in addressing contemporary IS phenomena.
Full-text available
The goal of a review article is to present the current state of knowledge in a research area. Two important initial steps in writing a review article are boundary identification (identifying a body of potentially relevant past research) and corpus construction (selecting research manuscripts to include in the review). We present a theory-as-discourse approach which a) creates a theory ecosystem of potentially relevant prior research using a citation-network approach to boundary identification; and b) identifies manuscripts for consideration using machine learning or random selection. We demonstrate an instantiation of the theory as discourse approach through a proof-of-concept, which we call the Automated Detection of Implicit Theory (ADIT) technique. ADIT improves performance over the conventional approach as practiced in past Technology Acceptance Model reviews (i.e., keyword search, sometimes manual citation chaining); it identifies a set of research manuscripts that is more comprehensive and at least as precise. Our analysis shows that the conventional approach failed to identify a majority of past research. Like the three blind men examining the elephant, the conventional approach distorts the totality of the phenomenon. ADIT also enables researchers to statistically estimate the number of relevant manuscripts which were excluded from the resulting review article, thus enabling an assessment of the review article's representativeness.
Conference Paper
Full-text available
Contributing to cumulative knowledge development is a central goal in Information Systems design science research (IS-DSR). IS design theories (ISDTs) have been recognized as a particularly important building block for the accumulation of design-oriented knowledge. Yet, there are limited insights into how follow-up research builds on IS design theories in terms of testing and extending. To address these issues, we present results from the first empirical analysis of how ISDTs have been tested and extended within and beyond IS journals. Our qualitative analysis of papers citing ISDTs uncovers an alarming paucity of follow-up research that builds on these ISDTs. Specifically, the overall number of papers testing and extending any of the selected ISDTs ranges in the single digits. To propose an actionable path forward, we formulate four specific guidelines on how the IS(-DSR) community can facilitate the cumulative extension of the IS knowledge base.
Conference Paper
Full-text available
Although it is accepted that to make a contribution, Information Systems (IS) research must be novel and provide utility, these characteristics are insufficient to account for the cogency or persuasion that research makes a contribution. Novelty and utility are not self-evident and authors need to articulate an argument that their paper makes a contribution in relation to the discourse to which it seeks to contribute. We use the Habermasian ideal of argumentation to explicate the rhetorical, dialectic, logical and socio-institutional dimensions of cogent argument. We use two examples of published research to illustrate how these elements can be extended to frame research as a contribution in relation to the wider disciplinary matrix. We outline next steps of the research and the utility of the framework for researchers, reviewers and editors.
Exploratory search is a specialised form of information searching behaviour that is open-ended, dynamic and multi-faceted in nature. The study elaborates the picture of exploratory search by concentrating on two theoretical frameworks for studying searching of this type, that is, the berrypicking model and the information foraging theory. The study focuses on two major issues: the motivators for exploratory search and the nature of search processes. The findings draw on the conceptual analysis of 30 key studies on this topic. The berrypicking model posits changing information needs as the main motivator for information searching, while the information foraging theory emphasises the role need for performing a task as a main trigger of information seeking. The berrypicking model devotes more attention to focused searching as a key constituent of exploratory search, while the information foraging theory concentrates on the characterisation of exploratory browsing.
Diversity has attracted much attention within the information systems (IS) field, with literature concentrating on diversity in topics and methods. These constitute two of three identified areas of research field diversity; the little-investigated third area includes demographic and social diversity of researchers. This study explores this gap for researchers comprising the editorial advisory boards (EABs) of 52 IS journals and links the underexplored types of diversity to topic diversity. The journals are categorized into seven intellectual communities, using topic affinity of journal content, and a social network of EAB members constructed from board interlocks. The network structure appears to reflect the topic-based community links. Journal communities are aggregated into two components of the social network: a business-school-related core set of journals and a more diverse computing- and engineering-related periphery. The strong ties at the network center do not necessarily reflect journal status. The observed combination of focus and diversity is consistent with a polycentric view of the IS field. Findings suggest low demographic diversity in the field and that demographic diversity correlates with other types of diversity. The field’s separation into business core and computing periphery is highlighted as potentially challenging to the IS field’s identity.
There has been a flowering of scholarly interest in the literature review as a research method in the information systems discipline. We feel privileged to contribute to this conversation and introduce the work of the authors represented in this special issue. Some of the highlights include three new methods for conducting literature analysis and guidelines, tutorials, and approaches for coping with some of the challenges involved in carrying out a literature review. Of the three “new method” papers, one (ontological meta-analysis and synthesis) is entirely new, and two (stylized facts and critical discourse analysis) are novel in the information systems context. The other four paper address more general issues: the challenges of effective search strategies when confronted with the burgeoning volume of research available, a detailed tool-supported approach for conducting a rigorous review, a detailed tutorial for conducting a qualitative literature review, and a discussion of quality issues. Collectively, the papers place emphasis beyond the traditional “narrative synthesis” on the importance of selecting the appropriate approach for the research context and the importance of attention to quality and transparency at all stages of the process, regardless of which approach is adopted.