Content uploaded by Blair Wang
Author content
All content in this area was uploaded by Blair Wang on Oct 29, 2019
Content may be subject to copyright.
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
www.litbaskets.io, an IT Artifact Supporting Exploratory
Literature Searches for Information Systems Research
Sebastian Boell
Discipline of Business Information Systems
The University of Sydney Business School
Sydney, Australia
Email: sebastian.boell@sydney.edu.au
Blair Wang
School of Information Systems and Technology Management
UNSW Business School
Sydney, Australia
Email: blair.wang@unsw.edu.au
Abstract
Information Systems (IS) researchers currently lack an obvious place to start their literature searches.
Existing tools suffer from being either too narrow in their coverage of existing research, leading to an
insufficiency effect (low recall); or they are too encompassing, leading to an impracticality effect (low
precision). From 11 listings of IS-related journals, we identify a set of 1,042 journals receptive to IS
research. We introduce a web interface that allows searching for literature across most of these
journals. The search tool enables researchers to narrow or widen the focus of searches, thus allowing
researchers to optimise the precision-recall trade-off of their literature searches. We provide an
evaluation of our artifact and discuss the relevance of our artifact for exploratory literature searches.
Our artifact seeks to facilitate knowledge claims in IS research based on a shared body of knowledge
beyond the AIS basket of eight journals.
Keywords: Literature Search, IS Journals, Literature Reference Database, Bibliometrics, Seniors
Basket of Journals
1
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
1 Introduction
Identifying relevant literature to understand the current state of knowledge is an important endeavour
for any research project in order to successfully argue for the relevance of its contribution (Boell and
Hovorka 2019; Hovorka and Boell 2015). Moreover, stand-alone literature reviews have received
considerable attention in IS in recent years (Boell and Cecez-Kecmanovic 2014; Paré et al. 2015; Tate
et al. 2015; vom Brocke et al. 2015; Webster and Watson 2002). Searches in databases – such as the
AIS e-Library (AISeL), Scopus, Web of Science (WoS), Google Scholar – are one important means for
identifying earlier research to be included in literature reviews (Greenhalgh and Peacock, 2005).
Good academic practice involves the documentation of completed literature searches. However, it is
not currently clear where IS scholars should start looking for literature. While there is the AISeL, it
only covers few IS journals, currently not even allowing a search through what is published in the AIS
Senior Scholars’ Basket of Eight journals (Bo8). In contrast, many disciplines have established
literature reference databases that serve as a starting point when looking for existing research. For
instance, Psychology researchers generally begin looking for literature in Psycinfo, indexing around
2,500 journals; Medical researchers would start in Medline, indexing around 5,200 journals; and in
Information Science, LISTA selectively indexes around 600 journals. By comparison, the AISeL covers
15 journals, only 2 of which are included in the Bo8. Our research is thus motivated by the lack of a
simple search tool that provides a starting point for IS researchers when they embark on their journey
of identifying existing research of relevance to IS.
We develop an artifact that supports IS researchers in undertaking exploratory literature searches
(Bates 1989; Marchionini 2006; Savolainen 2017). While the Bo8 serves well as an aspiration for
publication, limiting the literature identification process to only what is published in the Bo8 ignores
many relevant articles published outside of it. This is what we, in this paper, call the insufficiency
effect. Besides a narrow search in the Bo8, IS researchers frequently use large-scale multidisciplinary
literature reference databases for their searches such as Google Scholar, Scopus or WoS. These
databases, however, generally cover tens of thousands of journals, many of which publish research of
very little relevance to IS researchers. Researchers are therefore often overwhelmed by a huge number
of so called ‘false positives’ (Larsen et al. 2019). This makes it difficult to make sense of the huge
volume of retrieved documents in a feasible amount of time, something we call in this paper the
impracticality effect. This paper engages with the need to balance between the insufficiency effect and
the impracticality effect.
Examining journals appearing across eleven listings of IS journals, we develop an extensive set of
1,042 journals whose at least partial relevance to IS research has been previously established. Ranking
these journals by how often they are included in multiple journal listings, we identify a means through
which literature searches can be conducted by narrowing and widening the focus of searches. Setting
up an open source project, we build an IT artifact for the IS community that enables searching though
these journals using the Scopus database. In the spirit of open source we propose that the literature
search artifact available at www.litbaskets.io be refined by the collective effort of the IS community.
2 Literature Reviews and Literature Searches
As the IS discipline has matured, recognition of the need to stock of existing IS research has also
increased (Boell and Cecez-Kecmanovic 2014; Tate et al. 2015; vom Brocke et al. 2015; Webster and
Watson 2002). There are different types of literature reviews that serve different purposes including
theory development, providing an overview over a research area, and detailed analysis of a particular
effect in a meta-analysis (Paré et al. 2015). Such literature reviews use various approaches for
identifying relevant publications to be included in the review, ranging from keyword searches in
databases, citation tracking of relevant papers, analysis of bibliographies, to serendipitous encounters
and suggestions and advice by colleagues (Greenhalgh and Peacock, 2005).
In recent years in particular so called systematic or structured approaches have gained popularity in IS
(Boell and Cecez-Kecmanovic 2015). As part of this trend, we see an increase in literature reviews that
concentrate their efforts in identifying earlier IS research on the Bo8 and leading conferences in the
field: e.g. “we restrict our scope to the eight journals included in the AIS Senior Scholars' Basket of
Journals” (Schuster et al. 2018, p. 6); “we focused on the widely acknowledged AIS Senior Scholars’
Basket of Journals” (Winkler and Wessel 2018, p. 4); “we focused on articles published in the basket of
eight” (Diederich et al. 2019, p. 1553). However, focusing on only few leading outlets for literature
identification is problematic as Larsen et al. (2019) highlight:
2
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
“Like the blind men, we may focus on the voices of manuscripts at an arbitrary set of ‘top
journals’, which often eliminate manuscripts that challenge dominant theory, have unsupported
hypotheses or nonsignificant findings and thus did not clear the hurdles of the small set of
editors and reviewers at top journals.” (Larsen et al. 2019, p. 888)
Accordingly, much relevant existing knowledge may not be published in top tier journals such as the
Bo8. Nonetheless, findings of such research are relevant and should inform the generation of new
knowledge. While much is to be learned from an exceptional piece of research published in a top tier
journal, other journals also make important contributions to the knowledge of the field. One may even
go as far as arguing that an inward gaze and a sole focus on a few leading outlets can, in the long run,
be detrimental to the intellectual development of the discipline as controversial and unusual ideas are
sidelined in the academic discourse (Boell and Hovorka 2019; Larsen et al. 2019). However, such ideas
can drive innovative approaches and new thinking about phenomena (Alvesson and Sandberg 2014).
Besides the problem of literature reviews taking a too narrow focus, researchers also learn more about
the existing literature as they engage in the literature review process (Boell and Cecez-Kecmanovic
2014). Bates (1989), Marchionini (2006), Savolainen (2017) and others have thus emphasised the
exploratory nature of literature searches. As He et al. (2019) put it
“Searchers pick up pieces of information as they navigate through an information space. In this
evolving process, searchers acquire new knowledge and change their perception of search tasks
through their interaction with the information space” (p. 844).
For instance, as researchers engage with the existing literature they will discover different theories,
concepts, methodologies, frameworks, models, or constructs, associated with specialised vocabulary or
phrases that then can be used to further refine the literature identification process (Boell and Cecez-
Kecmanovic 2014). However initially, when researchers delve into the literature discovery process,
they are often not familiar with such specific vocabulary useful for locating highly specialised
literature. At these points, researchers often need to use general vocabulary as search terms in order to
describe the phenomena of interest to them. However, when searching with general vocabulary in a
large database often many irrelevant documents will be retrieved. Our research is thus motivated by
two aspects:
(1) The first motivation of our research is to addressing the problem of literature reviews that
are taking a too narrow focus when considering earlier published research, which we call the
insufficiency effect in the coverage of existing literature.
(2) The second motivation for our research is to address the need of researchers to interactively
manage the amount of search results retrieved in response to a query in order to deal with what
we call the impracticality effect.
3 What is an IS Journal? — Identifying a Set of IS Journals
IS as a discipline has many interests thus making it often unclear where IS research begins and where
it ends. This, however, can make it difficult to identify what journals outside of the Bo8 are publishing
research of relevance to IS. Before building a search artifact we thus first needed to identify a suitable
set of sources through which a literature search should be conducted. We decided to focus on journal
articles as a starting point since they are generally considered the primary research output of
completed IS research projects. We further believe that existing listings of IS journals offer a suitable
starting point for our purpose.
In order to identify journals that are of current interest to IS we decided to use any listing or ranking of
IS journals published in 2008 or later. For those interested in earlier journal rankings see, for
instance, Gillenson and Stefford (2008) or Walstrom and Hardgrave (2001). Overall, we found eleven
such journal lists from which we identified journals (Table 1). Importantly, our focus here is not on the
alleged quality of journals (Lewis et al. 2007; Lowry et al. 2013) or the diversity of editorial boards
(Burgess es al. 2017; Cabanac 2012), but simply on the journals considered.
One well-known journal list is the one published by the Association for Information Systems (AIS
2011). In addition to the Bo8, the AIS listing also includes journals nominated by various special
interest groups (SIGs) of the AIS. For instance, Information and Organization and Decision Support
Systems are nominated by multiple SIGs as one of their top 5 journals. All together the AIS thus
introduces a list of 55 unique journals receptive to IS research (AIS 2011). Beyond the AIS list there are
other reputable lists of IS journals. Some of these listings are used for research assessment purposes,
such as the Academic Journal Guide by the UK-based Chartered Association of Business Schools
3
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
(CABS 2018) and the listing of 177 IS journals by the Australian Business Deans Council (ABDC 2016).
In addition, the London School of Economics (LSE) has published an IS journal list (Willcocks et al.
2008); Holsapple (2009) provides a list of 43 journals based on publication behaviour at US public
universities; Chan et al. (2015) used a citation based analysis for identifying IS journals; and Stewart et
al. (2017) build a list of IS journals based on the WoS journal category for Computer
Science/Information Systems. Furthermore, Scopus also provides three different listings of IS
journals. Finally, John Lamp (2004) maintains a listing of IS journals on his website that is regularly
updated since 1995 and well recognised within the IS community. For the purpose of our research we
only focused on journals listed by Lamp (2004) as active.
Table 1. Overview of IS journal lists.
Listing Description # of journals
(% unique)
(1) ABS –
Journal
Ranking
Ranking journals from 4* (best) to 1. Aims to show “where the best work in
their field tends to be clustered”. Based upon peer review, editorial and
expert judgements and citation statistics.
93
(7%)
(2) ABDC /
ACPHIS
Ranking journals from A* (best) to C. The listing for IS is identical to the
listing provided by the Australian Council of Professors and Heads of
Information Systems (ACPHIS).
177
(10%)
(3) AIS Journals nominated by the AIS “college of senior scholars” including
journals nominated by special interest groups (SIGs) as “worthy of broader
AIS recognition extending beyond the basket of 8” (AIS 2011).
55
(11%)
(4) WoS
based
Based on the WoS category ‘Computer Science and Information Systems’,
extended with additional IS journals in order to identify “a large enough
sample to reflect the diversity of IS journals” (Stewart et al 2017, p. 549).
98
(30%)
(5) Citation
based
Citation-based analysis with the aim to develop “an objective method to
identify a set of core journals” (Chan et al. 2015, p. 239).
35
(0%)
(6) LSE LSE’s ranking to inform promotions decisions and to “advise junior staff and
students on where they should publish.” (Willcocks et al. 2008, p. 163)
46
(26%)
(7) US Public
Universities
Based on the publications of “106 full-time, tenured IS faculty members at
[...] top public research universities” in the US (Holsapple 2009, p. 122).
43
(7%)
(8) Scopus IS
List
In addition, we added three Scopus “subject areas” designated IS: First,
‘Computer Science – Information Systems’, oriented towards technical
perspectives on Information Systems.
309
(52%)
(9) Scopus
MIS List
As above, for: ‘Business – Management Information Systems’, oriented
towards the use of Information Systems for business management.
99
(33%)
(10) Scopus
ISM List
As above, for: ‘Decision Sciences – Information Systems and Management’,
oriented towards the use of IS to assist decision-making.
108
(32%)
(11) John
Lamp’s List
Since 1995 John Lamp actively maintains an Index of Information Systems
Journal. The aim of the index is to be exhaustive in its coverage. For our
purpose we chose to include all journals listed as active (‘current’).
6811
(61%)
Taken together, the eleven lists summarised in Table 1 identify 1,042 different journals. Out of the
eleven lists, ten contain journals that are unique to only one list leading to a list of 714 journals unique
to a single journal listing (Table 1). Thus, when assessing journals as to how often they appear across
different journal lists, we see a strong concentration pattern as per Table 2 below. This skewed
distribution can be expected as it is in line with what can be observed for most bibliometric
phenomena (Wilson 1999): skewed distribution patterns are commonly observed when looking at
authors publishing on a topic, publications on a given topic across journals or citations across
publications. A possible explanation for the skewed distribution of journals across lists is that they are
published by bodies with differing expertise and research orientation, using various methodologies for
1 John Lamp’s list has 684 entries marked ‘current’; however, of these, 3 are internally duplicated:
Information, Communication and Society (appears twice); International Journal of Information
Communication Technologies and Human Development (appears twice); International Journal
of Technology Diffusion (appears twice).
4
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
identifying IS journals. Thus, while there is agreement on core IS journals, IS as a reference discipline
(Baskerville and Myers 2002) is connected to the interests of diverse related communities. These
communities will be reflected differently across journal lists and hence the composition of journals in
different listings will vary. For instance, Burgess et al. (2017) argue for the existence of sub-
communities within IS such as E-commerce, Human-Computer Interaction, Management IS, among
others. Likewise, the range of SIGs within the AIS and the range of journals they nominate suggest that
there is a range of journals receptive to different types of IS research.
Table 2. Appearance of journals across 11 IS journal listings and their coverage in Scopus.
# of lists
journal is
listed in
11 10 9 8 7 6 5 4 3 2 1 Total
# journals
in this set 5 1 1 5 4 13 22 37 72 168 714 1,042
% of total 0.5% 0.1% 0.1% 0.5% 0.4% 1.2% 2.1% 3.6% 6.9% 16% 69% 100%
# indexed
by Scopus 5 1 1 5 4 13 22 36 67 148 545 847
% of set
indexed ALL ALL ALL ALL ALL ALL ALL 97% 93% 88% 76% 81%
Based on the skewed appearance of journals across listings, we see the emergence of a ‘core’ of IS
journals appearing across the majority of lists. For instance, besides the Bo8 only four journals are
listed in 8 or more rankings (see figure 1 below). We therefore argue that a list of 51 journals appearing
in five or more lists would constitute a suitable set for embarking on initial exploratory literature
searches. As the review process progresses or if insufficient number of results are retrieved IS
researchers may want to ‘cast a wider net’, hence literature searches can be further expanded into
journals appearing across fewer journal lists.
Figure 1. IS journals appearing in 5 or more lists including years covered in Scopus.
5
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
4 Building a Search Artifact
To realise the vision of an IT artifact for literature searches, we needed to connect to an existing
literature database covering the majority of the journals identified as publishing research of relevance
to IS. Our first attempt, based on Google Scholar, was of limited success. While Google Scholar allows
users to manually specify the journal to target for a particular search it does not allow this journal filter
to compound. We would therefore need to generate a different results page for each journal searched.
As many research universities provide their staff with access to Scopus we thus saw an opportunity to
leverage the existing Scopus advanced search functionality for building a tool that enables targeted
searches for IS. Advanced searches in Scopus allow the use of a retrieval language which adds
compounding conditions to searches beyond keywords. For instance, all resources indexed in Scopus
have a unique SOURCE-ID. Thus, an advanced search for ‘SOURCE-ID(12402)’ will identify all
articles published in MIS Quarterly that are indexed by Scopus. When the SOURCE-ID is combined
with keywords one is thus able to search through the title, keywords and abstracts of articles published
in MIS Quarterly. Furthermore, additional conditions can be included in searches such as document
types in order to search only for literature review articles or editorials or year of publication. The
ability to search for review articles and editorials are particularly useful for exploratory searches.
Literature reviews enable the identification of publications that offer an overview of earlier research. A
search through editorials is useful for identifying special issues on particular topics.
To proceed, we first matched the complete set of all journals identified by us (see Tables 1, 2) to their
respective SOURCE-ID registered in Scopus. In the case of John Lamp’s list a SCOPUS-ID was already
provided for most journals for other lists used identifying features such as ISSN or journal titles. This
process was facilitated by a relational database system (mySQL) in which we captured other useful
attributes such as Scopus coverage years and journal URL, where applicable. Of the 1,042 unique
journals identified during the discovery phase, 847 are indexed by Scopus (81%, see Table 2). As
shown in Figure 1, all 51 journals included in 5 or more lists are currently indexed by Scopus with
coverage in many cases is also going back several decades. Using the relational database of SOURCE-
ID for IS journals we developed a web-based graphical user interface accessible at www.litbaskets.io in
which users can generate search strings for Scopus’ advanced search and customise the selection of
journals. IS researchers can thus use our tool to generate elaborated search strings for Scopus
advanced search that enables simultaneous search in up to 847 journals publishing research or interest
to IS. All users have to do is copy the search string into Scopus’ advances search window (Figure 2).
Figure 2. Transferring Scopus Query String into Scopus Advanced Search.
Furthermore, if users are accessing Scopus from within an organisational network that has access to
Scopus (e.g. on campus or via VPN) search queries can be directly performed in Scopus. Since we
established our own relational database containing Scopus SOURCE-IDs, we can directly generate
predictive URLs for specific Scopus searches. As URL length are capped, direct searches work only for
sets of ~52 journals. However, as our artifact generates complete search strings using the correct
6
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
syntax for Scopus’ advanced search, simultaneous searching across all 847 journals indexed by Scopus
is possible by copying the generated search string directly into Scopus (Figure 2).
Our codebase is open-source and available on GitHub2 to facilitate the development of our search
artifact by the IS community. Our search artifact is tentatively named Litbaskets (short for Literature
Baskets) and deployed at www.litbaskets.io. It has the following features:
•‘Search’: Landing page with a search box and instructions. The instructions encourage the user
to think about the nature of their search, and on that basis, make adjustments using the
configuration pages (see below). If the user chooses to proceed with the search without
customising the settings, the search is performed on our default core set of 51 journals with the
user-entered search terms matched against the title, keywords and abstract fields in Scopus.
Upon submitting their search and if fewer than 52 journals are searched, the user (assuming
they have access to Scopus) is redirected to a Scopus search results page in a new browser tab.
In addition, a message appears on www.litbaskets.io with the relevant query string that can be
pasted directly into Scopus’ advanced search if needed (see Figure 2), and options to execute
the same search directly on AISeL or the dblp.org bibliographic library. (Although dblp.org
relates to Computer Science and not IS, we provide the option for the cases where the research
topic is sufficiently technical to make use of Computer Science literature.)
•‘1. Choose Litbasket’: Configuration page that allows the user to select a ‘litbasket’, which is
simply a predefined set of journals that the search can be performed on. The default litbasket
is our ‘core’ set of 51 journals, which we call ‘L5’ (mentioned in 5 or more of our reference lists,
as per Figure 1). Other litbaskets based on list counts range from the largest, ‘L1’ (all 847
journals indexed by Scopus) to the smallest, ‘L7’ (only the 16 journals that appear in 7 or more
of our reference lists, as per Figure 1).We also include an even smaller litbasket than L7 which
is the ‘Bo8’ option so that users can, if they deem appropriate, still simply just search through
the AIS Basket of Eight.
•‘2. Customise Sources’: Configuration page that allows the user to select or deselect individual
journals to make adjustments to the ‘litbasket’ that they have selected. To assist the user to
find a specific journal, this configuration page includes all reference lists from Table 1, as well
as the aggregate of the journals from all lists combined. Thus, users can customise a particular
journal set from all 847 journals based on the specific needs imposed by their research topic.
•‘3. Apply Filters’: Configuration page that allows the user to use additional Scopus search
features. For example, in order to return only current research, there is a default filter that
limits search results to only retrieve articles published from 2013 onwards. If older results are
desired, researchers can adjust this filter. In addition, Scopus (unlike Google Scholar) indexes
articles into document types. This enables filtered searches: either only for literature review
articles (for an overview of earlier research and how studies are related to each other); or only
for editorials (for identifying special issues on particular topics).
5 Assessing the Artifact
Two crucial measures for assessing any search system are recall and precision (Salton and McGill
1983). Underpinning both measures is the criterion of relevance, since items retrieved in a search can
either be described as addressing (‘relevant’), or not addressing (‘irrelevant’), what is sought. Recall is
a measure describing how many of all potentially relevant items available in a search system are
retrieved by a search. Ideally, a retrieval system would aim for a recall of 100% to identify all relevant
items available. A search system that always reproduces all items available in response to a query
would, by definition, have a perfect recall. Nonetheless, such a system would be largely unusable due
to the aforementioned impracticality effect.
Therefore, precision is established as a second measure of the exactness of retrieval systems. This
measure assesses how many of the results retrieved by the system are actually relevant. Ideally, a
perfect retrieval system would aim for a precision of 100% if, in response to a query, the system only
showed users documents that are relevant to them. However, in practice, attempts to achieve high
precision result in too few results, as per the aforementioned insufficiency effect. This is the recall-
precision trade-off relationship (Buckland and Gey 1994), wherein an increase in recall generally
comes at the expense of precision, and searches with high precision will generally yield a lower recall.
Figure 3 visualises the recall-precision trade-off, and how it leads to the impracticality effect and the
insufficiency effect.
2 GitHub repository: https://github.com/blairw/litbaskets
7
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
Figure 3. How www.litbaskets.io manages the recall-precision trade-off.
To exemplify the usefulness of www.litbaskets.io for dealing with both the insufficiency effect (when
too little literature is found) and the impracticality effect (when researchers are overwhelmed with too
many results), we conduct some exemplary searches. Since the actual judgement of whether a paper is
relevant will depend on what a researcher seeks to achieve, we chose to use the number of documents
retrieved as a proxy for judging how encompassing the results retrieved by our artifact are in
comparison to existing alternatives. To do this, we selected four broad search terms that we believe are
representative for what may be initially used by researchers conducting exploratory searches (Table 3).
Table 3. Number of results for different search tools using example searches based on title,
abstract and keywords and publication year > 2013.
Search
Term
AIS
Bo8 AISeL Litbaskets.io
Core Set (‘L5’)
Litbaskets.io
Full Set (‘L1’) WoS Google
Scholar
blockchain 4 154 51 692 2,974 ~24,100
crowdfunding 25 144 77 204 1,302 ~20,400
e-health 2 135 104 830 6,187 ~78,900
outsourcing 63 243 338 1,452 18,338 ~84,300
*Searches in Google Scholar could not be limited to abstracts, titles, keywords only.
As aforementioned, relying on a set of ‘top journals’ is problematic when conducting literature reviews.
Further looking at the number of documents retrieved using titles, abstracts and keywords for the AIS
Bo8, it is seems that for many searches too few documents are retrieved for a comprehensive literature
review. When limiting the search to titles, abstracts and subjects in the AISeL the number of
documents retrieved are more promising. However, looking more closely at the results almost all of
them are conference papers. We thus believe that the AISeL can offer a good additional resource to
litbaskets.io for identifying conference papers. In contrast, the number of search results retrieved from
Web of Science (WoS) and Google Scholar indicate an impracticality effect as researchers would have
to traverse through thousands of documents of which many will be irrelevant to the research topic.
While the usefulness of the ability for IS researchers to search through a specific set of IS journals
should be evident, the number of results for www.litbaskets.io further exemplifies this. As can be seen
in Table 3, users are able to interactively manage the focus of their literature searches and thereby
control the number of documents retrieved. Our core set (‘L5’) in Table 3 is for searches across
journals listed in Figure 2, i.e., those appearing in the majority of all eleven of our reference lists. As
the range of journals through which a search is conducted increases, so does the number of documents
retrieved. This is illustrated by the increase in the number of documents retrieved for the full set of all
847 journals indexed by Scopus. Thus our artifact allows researchers to manage the recall-precision
8
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
trade-off as searchers can increase the number of journals if the recall in response to their query is too
low. However, too many documents are retrieved for an initial engagement with the literature
researchers can easily apply filters such as ‘review articles only’ or reduce the number of journals
through which a search is conducted.
6 Discussion
6.1 Contributions
The contributions of our research are related to the bibliometric work underpinning the creation of our
list of IS journals as well as the IT artifact we created. Firstly, our artifact contributes an important
new means for discovering IS knowledge. Many established disciplines have a primary resource for
discovering publications, yet no comparable resource exists for IS. Our tool thus addresses an
important need as discipline-specific literature reference databases fulfil notable functions: (1) They
make literature accessible through searches; (2) They support researchers in making knowledge claims
based on a disciplines discourse by ensure good coverage of established knowledge; (3) They select
specific sources to filter out predatory journals and any sources not central to disciplinary knowledge.
Moreover, by enabling a search through an extensive set of journals in which research of relevance to
IS may be published, we enable IS researchers to improve the effectiveness of their literature searches.
Recall and precision and therefore the effects of insufficiency as well as the impracticality are related to
both the range of sources considered for a literature search as well as the search terms chosen for the
search. We manage and optimise precision by reducing the sources considered for searching, from the
broad coverage offered by large multidisciplinary sources such as Google Scholar or WoS to a more
targeted and specific set of journals. Furthermore, Scopus further supports the literature identification
process as it allows researchers to employ elaborate search strategies such as successive fractions or
snowballing through its ‘refine search’ function (Boell and Cecez-Kecmanovic 2014). Hence,
researchers are less likely to be overwhelmed by huge numbers of irrelevant documents. We also
manage and optimise recall, as searches through a more targeted set of journals enable a more
thorough search within this set, for instance, by using single word searches or common language in
addition to specialised terminology. Hence, researchers are less likely to miss relevant publications as
they can use more inclusive search terms in their searches without risking that too many irrelevant
documents (‘false positives’) are retrieved.
Our tool therefore is useful to IS researcher in three ways: (1) By enabling targeted searches and filters,
we empower researchers to quickly identify a highly relevant article to be added on top of their reading
list. Based on the reading of such an article, researchers discover further relevant terminology,
constructs, concepts, theories, models, etc. that then inform further literature searches (Boell and
Cecez-Kecmanovic 2014); (2) By using our tool, researchers can also compile lists of relevant articles
related to a topic that provide them with an overview of what is well known and researched within the
wider IS community, thus better frame the contribution of their work to the disciplinary discourse
(Boell and Hovorka 2019); (3) By enabling a targeted search, our tool supports editors by facilitating
identification of potential reviewers.
Finally, the methodology we used for creating the journal list demonstrates that skewed distributions
underpinning bibliometric phenomena also apply to how frequently journals are mentioned across
journal listings. In other words, inclusion of journals in journal lists follows a similar distribution
pattern as, for instance, citations to articles. This finding has implications for journal analysis in
bibliometric studies more widely as it questions the validity of studies underpinned by a single journal
list. For instance, Stewart et al. (2017) make important observation about the cohesion of the IS
discipline. Yet when looking at the set of journals used for the analysis, 30% of the 98 journals did not
appear in any of the other ten journal listings identified by us. Likewise, Burgess et al. (2017)
exploration of the diversity of editorial boards in IS journals was based on the CABS journal listing.
However, the CABS ranking not only includes 6 journals not mentioned in any of the other journal
lists, it also excludes most journals of our journal set of 51 journals appearing in the majority of all
journal lists. Hence our research encourages caution when looking at findings from earlier bibliometric
studies not only for IS, but for bibliometric studies related to journals in general.
6.2 Limitations and Next Steps
Although our search tool enables most IS researchers to begin comprehensive literature reviews on
most IS topics, it is not perfect and has some limitations. Firstly, users must have access to Scopus.
Secondly, while we identified an extensive set of IS journals, depending on the research undertaken
9
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
additional journals should be searched. Thirdly, our search tool enables searching though journal
article and excludes other publications such as conference proceedings or book chapters. These
limitations do, however, have mitigating factors. Firstly, Scopus access is very common. We inspected
the institutional login page for Scopus and found that, for example, every university that has ever
hosted the Australasian Conference on Information Systems (ACIS) currently has institutional access
to Scopus (Scopus 2019). Secondly, we argue that by starting with journal papers – i.e., documentation
of completed research projects – in their first literature review iteration, IS researchers can have a
better appreciation for conference papers that they discover in later iterations. Furthermore our
artifact can be paired with AISeL that offers much better coverage of IS conference, but lacks coverage
of IS journals. We thus recognise the value of potentially including additional databases in future to
reduce the high level of reliance on Scopus, and we are currently investigating the technical feasibility
of connecting to APIs available for dblp.org and Google Scholar.
One of the reasons for making our codebase entirely open-source is to encourage other IS academics to
contribute, such that the project as it is ‘by the community, for the community’. We take inspiration
from other significant open-source projects started in academia and a view that the IS community has
a critical mass of members who possess the programming skills required to participate. For instance,
we would welcome an initiative to extend the coverage of www.litbaskets.io to also include the
proceedings of leading IS conferences in its searches.
7 Conclusion
To make a meaningful contribution, research needs to be framed in relation to the prevailing discourse
of an academic community (Boell and Hovorka 2019; Hovorka and Boell 2015). The literature
identification process is central for understanding share knowledge and hence for expressing new
knowledge claims. However, unlike Medline for Medicine or Psycinfo for Psychology, there is currently
no database that IS researchers can go to as a starting point for conducting exploratory literature
searches, leading to either too little literature to review (the insufficiency effect) or too much literature
to review (the impracticality effect). Our study addresses this deficit by identifying a set of journals
that can be considered as receptive to IS research. By building www.litbaskets.io as an interface for
searching through these journals we hope to not only make literature searches more effective for IS
researchers, but also to facilitate the IS discipline uniting around a core of shared knowledge.
8 References
Alvesson, M., and Sandberg, J. 2014. “Habitat and Habitus: Boxed-in versus Box-Breaking Research”,
Organization Studies (35:7), pp. 967–987.
Association for Information Systems (AIS) 2011. “Senior Scholars’ Basket of Journals” [online].
https://aisnet.org/general/custom.asp?page=SeniorScholarBasket
Australian Business Deans Council (ABDC) 2016. “ABDC Journal Quality List” [online].
https://abdc.edu.au/research/abdc-journal-list/
Baskerville, R.L., and Myers, M.D. 2002. “Information Systems as a Reference Discipline”, MIS
Quarterly (26:1), pp. 1–14.
Bates, M.J. 1989. “The design of browsing and berrypicking techniques for the online search interface”,
Online Review (13:5), pp. 407–424.
Boell, S.K., and Cecez-Kecmanovic, D. 2014. “A Hermeneutic Approach for Conducting Literature
Reviews and Literature Searches”, Communications of the AIS (34:1), pp. 257–286.
Boell, S.K., and Cecez-Kecmanovic, D. 2015. “On Being ‘systematic’ in Literature Reviews in IS”,
Journal of Information Technology (30:2), pp. 161–173.
Boell, S.K. and Hovorka, D.S. 2019. “Writing, Arguing, Contributing - A Cogent Argumentation
Framework for Identifying, Specifying, and Evaluating Research Contribution”, Australasian
Journal of Information Systems (23). 10.3127/ajis.v23i0.1857.
Buckland, M., and Gey, F. 1994. “The relationship between Recall and Precision”, Journal of the
American Society for Information Science (45:1), pp. 12–19.
Burgess, T.F., Grimshaw, P., and Shaw, N.E. 2017. “Research Commentary—Diversity of the
Information Systems Research Field: A Journal Governance Perspective”, Information Systems
Research (28:1), pp. 5–21.
Cabanac, G. 2012. “Shaping the Landscape of Research in Information Systems From the Perspective
of Editorial Boards: A Scientometric Study of 77 Leading Journals”, Journal of the American
Society for Information Science and Technology (63:5), pp. 977–996.
Chan, H.C., Guness, V., and Kim, H.-W. 2015. “A Method for Identifying Journals in a Discipline: An
Application to Information Systems”, Information and Management (52:2), pp. 239–246.
10
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
Chartered Association of Business Schools (CABS) 2018. “Academic Journal Guide 2018” [online].
https://charteredabs.org/academic-journal-guide-2018/
Diederich, S., Brendel, A.B., and Kolbe L.M. 2019. “On Conversational Agents in Information Systems
Research: Analyzing the Past to Guide Future Work”, Proceedings of the 14th International
Conference on Wirtschaftsinformatik, p. 1550-1564.
Gillenson, M., and Stafford, T. 2008. “Journal Rankings 2008: A Synthesis of Studies”, Americas
Conference on Information Systems (AMCIS), Paper 58.
Greenhalgh, T., and Peacock, R. 2005. “Effectiveness and efficiency of search methods in systematic
reviews of complex evidence: audit of primary sources”, British Medical Journal (331:7524), pp.
1064–1065.
He, J., Ping, Q., Lou, W., and Chen, C. 2019. “PaperPoles: Facilitating Adaptive Visual Exploration of
Scientific Publications by Citation Links,” Journal of the Association for Information Science
and Technology (70:8), pp. 843-857.
Holsapple, C.W. 2009. “A new map for knowledge dissemination channels”, Communications of the
ACM (52:3), pp. 117–125.
Hovorka, D.S., and Boell, S.K. 2015. “Cogency and Contribution in IS Research”, International
Conference on Information Systems, Fort Worth (USA), Paper 1574.
Lamp, J. W. 2004. Index of Information Systems Journals. Geelong, Deakin University [online].
http://lamp.infosys.deakin.edu.au/journals/
Larsen, K.R., Hovorka, D.D., Dennis, A.R., and West, J.D. 2019. “Understanding the Elephant: The
Discourse Approach to Boundary Identification and Corpus Construction for Theory Review
Articles”, Journal of the AIS (20:7), pp. 887-916.
Lewis, B., Templeton, G., and Luo, X. 2007. “A Scientometric Investigation into the Validity of IS
Journal Quality Measures”, Journal of the AIS (8:12), pp. 619–633.
Lowry, P.B., Moody, G.D., Gaskin, J., Galletta, D.F., Humpherys, S.L., Barlow, J.B., and Wilson, D.
2013. “Evaluating Journal Quality and the Association for Information Systems Senior Scholars’
Journal Basket Via Bibliometric Measures: Do Expert Journal Assessments Add Value?”, MIS
Quarterly (37:4), pp. 993–1012.
Marchionini, G. 2006. “Exploratory search: From finding to understanding”, Communications of the
ACM, (49:4), pp. 41–46.
Paré, G., Trudel, M.-C.C., Jaana, M., and Kitsiou, S. 2015. “Synthesizing Information Systems
Knowledge: A Typology of Literature Reviews,” Information and Management (52:2), pp. 183–
199.
Salton, G., and McGill, M.J. 1983. Introduction to modern information retrieval, New York: McGraw-
Hill.
Savolainen, R. 2017. “Berrypicking and information foraging: Comparison of two theoretical
frameworks for studying exploratory search”, Journal of Information Science (43:3), pp. 1–14.
Scopus 2019. Login via your Institution [online], https://www.scopus.com/checkaccess.uri
Schuster, R., Wagner, G., and Schryen, G. 2018. “Information Systems Design Science Research and
Cumulative Knowledge Development: An Exploratory Study”, International Conference on
Information Systems, San Francisco (USA).
Stewart, A., Cotton, J. L., and Adya, M. 2017. “Information Systems: A House Divided?”,
Communications of the AIS (41:24), pp. 544–586.
Tate, M., Furtmueller, E., Evermann, J., and Bandara, W. 2015. “Introduction to the special issue: The
literature review in information systems”, Communications of the AIS (37), Paper 5.
vom Brocke, J., Simons, A., Riemer, K., Niehaves, B., Plattfaut, R., and Cleven, A. 2015. Standing on
the shoulders of giants: Challenges and recommendations of literature search in information
systems research, Communications of the AIS (37), Paper 9.
Walstrom, K.A., and Hardgrave, B.C. 2001. “Forums for information systems scholars: III,”
Information and Management (39:2), pp. 117–124.
Webster, J., and Watson, R.T. 2002. Analyzing the past to prepare for the future: Writing a literature
review. MIS Quarterly (26:2), pp. 13-23.
Willcocks, L., Whitley, E. A., and Avgerou, C. 2008. “The ranking of top IS journals: A perspective
from the London School of Economics”, European Journal of Information Systems (17:2), pp.
163–168.
Wilson, C.S. 1999. “Informetrics”, Annual Review of Information Science and Technology (34), pp.
107–247.
Winkler, T. J., and Wessel, M. 2018. “A Primer on Decision Rights in Information Systems: Review
and Recommendations”, International Conference on Information Systems, San Francisco
(USA).
11
Australasian Conference on Information Systems Boell & Wang
2019, Perth Supporting Exploratory IS Literature Searches
Copyright
Copyright: © 2019 Boell & Wang. This is an open-access article distributed under the terms of the
Creative Commons Attribution-NonCommercial 3.0 Australia License, which permits non-commercial
use, distribution, and reproduction in any medium, provided the original author and ACIS are
credited.
12
