Content uploaded by Sean B. Maynard
Author content
All content in this area was uploaded by Sean B. Maynard on Dec 10, 2015
Content may be subject to copyright.
Australasian Conference on Information Systems Zurita et al.
2015, Adelaide, South Australia Evaluating Research Articles for InfoSec Management Teaching
1
Evaluating the Utility of Research Articles for Teaching
Information Security Management
Harry Zurita
Department of Computing and Information Systems
The University of Melbourne
Parkville, Victoria, Australia
Email: harry.zurita@gmail.com
Sean B. Maynard
Department of Computing and Information Systems
The University of Melbourne
Parkville, Victoria, Australia
Email: sean.maynard@unimelb.edu.au
Atif Ahmad
Department of Computing and Information Systems
The University of Melbourne
Parkville, Victoria, Australia
Email: atif@unimelb.edu.au
Abstract
Research articles can support teaching by introducing the latest expert thinking on relevant topics and
trends and describing practical real-world case studies to encourage discussion and analysis.
However, from the point of view of the instructor, a common challenge is identifying the most suitable
papers for classroom teaching amongst a very large pool of potential candidates that are not typically
written for teaching purposes. Further, even in practice-oriented disciplines such as Information
Security Management (ISM), high-quality journals emphasise theoretical contribution and research
method rather than relevance to practice. Our review of the relevant literature did not find a
comprehensive set of criteria to assist instructors in evaluating the suitability of research articles to
teaching. Therefore, this research-in-progress paper presents a framework to support academics in
the process of evaluating the suitability of research articles for their teaching programs.
Keywords: Information Security, Research Article Evaluation, Multi Criteria Decision Making
1 Introduction
The selection of research articles for use in teaching by academics can be challenging. Research
articles are written with a very specific scope and many articles may be required in a course, possibly
even one or more articles for each topic area. This requires academics to review vast amounts of
literature to identify individual research articles for their utility in teaching. This is a time consuming
process, for academics that are already overloaded with administrative tasks, teaching responsibilities
and research activities (Benbasat and Zmud 1999; Shkedi 1998).
The use of research articles in teaching provides a number of benefits. Research articles assist
academics by showing students how to make fact-based decisions (Hemsley-Brown and Sharp 2003),
can support teaching programs with "free discussion, short questioning, to improve students learning"
(Abawajy 2009) and provide case studies that “promote problem solving and analysis” because “since
cases are often utilized in a group setting, they provide an opportunity for students to develop
teamwork, interpersonal and communications skills” (Cappel and Schwager 2002).
However, many research articles are not suitable for use in classrooms for a number of reasons. First,
the academic rigour required in high quality journals impacts the usefulness of these articles in
practice and teaching (Benbasat and Zmud 1999). Second, some articles are written in a manner (e.g.
structure) that is simply not conducive to teaching (Lindskog et al. 1999). Third, students find some
articles hard or unpleasant to read (Taylor 2007). Finally, the lifespan of research articles can be
limited, especially in dynamic disciplines such as those influenced by technology (Crowley 2003).
Based on the aforementioned points discussed, this paper pursues the research question of: “How can
the suitability of research articles to information security management teaching be evaluated?”
Australasian Conference on Information Systems Zurita et al.
2015, Adelaide, South Australia Evaluating Research Articles for InfoSec Management Teaching
2
For the purpose of this paper we define information security management as the process of applying
formal, informal and technical controls with the objective of protecting the confidentiality, integrity
and availability of information in the physical and digital environment whilst maintaining strategic
alignment with the organisational mission.
This paper is structured as follows. First we introduce the background to the area before describing
the research methodology undertaken. We then develop a framework with categories of criteria to
evaluate the suitability of research articles to a generic subject. Third, we develop a methodology using
the framework so that academics can be more efficient in assessing the suitability of research articles,
and subsequently show the utility of the methodology by describing a prototype application. In the
discussion section we suggest how the criteria can be used in ISM to address existing deficiencies in
available guidance from textbooks. Finally we conclude and offer suggestions for future work.
2 Background
The value of including research within teaching is widely recognised (Abawajy 2009; Benbasat and
Zmud 1999; Cappel and Schwager 2002) and is often mandated by universities. Research articles
often support the teaching process by promoting active learning and in-depth exploration of material
(Fisher 2006; Peck 2004). However, research articles are often not written in a classroom-friendly
format, or with teaching in mind. They are written for other researchers, contain technical jargon, and
use complex writing styles (Lindskog et al. 1999). They may also be hard, or unpleasant to read
(Taylor 2007) because they contain rigorous research approaches aimed “to establish credibility, to
publish in high quality journals, to attain tenure and promotion, and to compete for research funding”
(Rosemann and Vessey 2008).
Similarly, articles written in quality journals tend to lack relevance to practice. These articles focus on
academic rigor over practical relevance as specified by many publication outlets (Benbasat and Zmud
1999). As a result, researchers tend to focus on explaining their rigorous research approach, making
their exposure to practice-based activities infrequent and insufficient (Taylor 2007). Subsequently,
research articles are less engaging for practitioners because they lack insights from real-world
practice.
Furthermore, in dynamic disciplines, the lifespan of research articles decreases considerably because
due to newer methods and technologies introduced continuously (Crowley 2003; Jewels et al. 2003).
Therefore, there is no cumulative research tradition (Benbasat and Zmud 1999) and articles thus,
become rapidly outdated. Having access to recent research is important because it is more engaging
for students (Cappel and Schwager 2002) and allows them to get practice-oriented experience that
will support them in the changing environment of the real world (Hsu and Blackhouse 2002).
3 Research Methodology
The first part of this research was to conduct a conceptual study on how to assess the suitability of
research articles for teaching. We draw on guidance from Neuman (2006) on how to conduct a
systematic literature review to identify articles that address this issue. Since the aim of the paper is to
develop an evaluation criteria for use in Information Security Management, our approach seeks to
explore literature in the related disciplines of Information Systems and Information Security literature
using Google Scholar, the AIS Digital Library, and various publisher databases (e.g. Elsevier,
Emerald). We searched these using a range of terms (see Figure 1).
Figure 1: Search Terms Used
In addition to the papers identified using the search terms above, we also included those articles that
were cited in these papers. Overall we identified 48 papers. Subsequently, 36 papers were discarded
Australasian Conference on Information Systems Zurita et al.
2015, Adelaide, South Australia Evaluating Research Articles for InfoSec Management Teaching
3
because they did not provide guidance or recommendations on how to make research articles and
teaching cases more suitable to the student or academic community, leaving a total of twelve papers.
We analysed these papers in line with Neuman’s (2006) open, axial and selective coding approach. We
then developed a framework of criteria for assessing the suitability of research articles for teaching.
We operationalised the framework using a multi criteria methodology as per Maynard et al. (2001).
The approach determines the scores of parents (Categories) based on the scores gathered from the
bottom level (Criterion). Additionally, we used the approaches in (Adelman et al. 1985) and
(Goicoechea et al. 1992) to determine weightings in the two levels (Categories and Criteria) of the
hierarchy.
4 Systematic Literature Review
Our review of the literature did not identify any articles in the IS or Information Security domains that
presented a comprehensive method assessing the suitability of research articles for teaching.
However, several articles presented criteria that can be used to evaluate research articles for this
purpose. These criteria were aimed at the authors of research papers rather than classroom
instructors. They advised on how to: (1) incorporate research into teaching; (2) increase articles’
relevance to practice; and (3) write good case studies.
A number of researchers present characteristics of good research articles (Cappel and Schwager 2002;
Farhoomand 2004; Kim et al. 2006). These include writing in clear and simple English (Farhoomand
2004), giving real world examples (Hackney et al. 2003), being timely (Yue 2012) and having a “hook”
to motivate readership (Cappel and Schwager 2002). These publications also try to address the
acknowledged problem that research articles can be unpleasant to read and lack relevance to practice.
The lack of relevance to practice in research articles makes them less accessible and less interesting to
the reader. This discourages the reader from using these articles for teaching purposes (Taylor 2007).
Lack of relevance to practice may be caused for a number of reasons. For instance, it is suggested that
article length and complexity make it difficult for students to understand. Furthermore, because the
focus of many articles is on the rigorous research approach rather than the findings (Benbasat and
Zmud 1999; Rosemann and Vessey 2008) they often become hard to read as the reader gets bogged
down in the detail of the rigour.
Articles used for teaching purposes need to focus more on practice-based factors around problems and
topics relevant for practitioners (Rosemann and Vessey 2008). Articles can address this by providing
an implementable approach to resolve practice-based problems. Furthermore, they can challenge
readers’ casual assumptions, paradigms or trends on practice-based areas (Benbasat and Zmud 1999).
One way of achieving this is to involve practitioners during the research (Hemsley-Brown and Sharp
2003; Rosemann and Vessey 2008). Practitioners give a practice perspective to research articles by
increasing the exposure researchers have to practice (Benbasat and Zmud 1999).
Teaching cases provide an alternate method of using research in teaching and are an effective tool as
they allow students to develop real-life decision making, problem solving, higher-order reasoning,
teamwork and communication skills (Cappel and Schwager 2002; Farhoomand 2004; Hackney et al.
2003). These skills are developed using the active learning methodology where students ‘learn by
doing’ which is characterized as being highly motivational (Cappel and Schwager 2002).
Although teaching cases allow the development of the aforementioned skills, there is a recognised
paucity of teaching cases in the Information Systems discipline (Cappel and Schwager 2002) and even
more so in the information security management discipline. At the same time, the lack of cumulative
research and the dynamism of this discipline cause that the few teaching cases available become
rapidly outdated (Benbasat and Zmud 1999).
Literature provides guidance on how to write teaching cases for the Rotterdam School of Management
(Yue 2012) and the Journal of Information Systems Education (Cappel and Schwager 2002), amongst
others. Although these guides have a specific focus, they also provide general guidance for teaching
case development. Kim et al. (2006) reviewed 100 teaching cases from multiple disciplines identifying
strategies and core attributes of good cases. They identified five core attributes of good cases: relevant,
realistic, engaging, challenging and instructional.
4.1 Framework to evaluate research articles for teaching purposes
From our analysis of the literature we developed a framework of criteria for the assessment of the
suitability of research articles for teaching (Table 1). We analysed the literature using the three rounds
Australasian Conference on Information Systems Zurita et al.
2015, Adelaide, South Australia Evaluating Research Articles for InfoSec Management Teaching
4
of coding defined by (Neuman 2006). In the first round, open coding, we scanned the selected
publications identifying recommendations on how to make research articles and teaching cases more
suitable for use in classes. In the second, axial coding, we categorized the criteria according to theme.
In the third round each category was divided into one or more criteria according to their specific
focus.
Category & Criteria References
1. Clarity
1.1: How simple is the article narrative (i.e. avoiding unnecessary
words, jargon, technical language, and the extended used of
citations)?
1.2: To what extent does the article use a top down structure where
the initial paragraph provides the setting and main issues of the
research article?
Benbasat and Zmud (1999);
Farhoomand (2004); Kavan
(1998); Kim et al. (2006);
Rosemann and Vessey (2008);
Yue (2012)
2. Succinctness
2.1: To what extent does the article length match the effort required
by students, as stipulated by the course, to allow them to
conduct an optimal analysis of it?
2.2: To what extent does the article provide sufficient information to
allow students to develop coherent conclusions?
2.3: To what extent does the article focus on the findings, rather
than the inputs such as the literature review or the research
methodology?
Benbasat and Zmud (1999);
Cappel and Schwager (2002); Kim
et al. (2006); Robey and Markus
(1998); Rosemann and Vessey
(2008); Senn (1998); Taylor
(2007); Yue (2012)
3. Objectiveness
3.1: To what extent is the article written in a neutral, unbiased
manner, allowing students to develop their own opinion?
Cappel and Schwager (2002);
Farhoomand (2004); Robey and
Markus (1998); Rosemann and
Vessey (2008); Taylor (2007); Yue
(2012)
4. Realism
4.1: To what extent does the article incorporate real world
examples?
4.2: How authentic does the article seem given the level of evidence
and facts presented?
4.3: To what extent does the article cite participants to increase its
realism?
Benbasat and Zmud (1999);
Farhoomand (2004); Hackney et
al. (2003); Jewels et al. (2003);
Kim et al. (2006); Rosemann and
Vessey (2008); Senn (1998);
Taylor (2007); Yue (2012)
5. Timeliness
5.1: To what extent are the research article’s findings up-to-date? Cappel and Schwager (2002);
Taylor (2007); Yue (2012)
6. Teaching friendliness
6.1: To what extent has the article been previously assessed for use
in other teaching programs?
Cappel and Schwager (2002); Kim
et al. (2006); Taylor (2007)
7. Depth
7.1: To what extent does the article provide multiple perspectives
from different stakeholders?
7.2: To what extent does the article provide distractors (non-
pertinent features) to challenge students’ analytical skills?
7.3: To what extent does the complexity of data (qualitative and
qualitative) presented by the article help to develop students’
problem solving skills?
7.4: To what extent does the article contain teaching aids to support
student learning?
7.5: To what extent does the article let students make their own
decisions by not providing a diagnosis of the problem?
7.6: To what extent does the article provide feedback on the possible
actions of students?
7.7: To what extent does the article synthesize an existing body of
research for the area of study?
Cappel and Schwager (2002);
Farhoomand (2004); Kim et al.
(2006); Taylor (2007); Yue (2012)
Australasian Conference on Information Systems Zurita et al.
2015, Adelaide, South Australia Evaluating Research Articles for InfoSec Management Teaching
5
Category & Criteria References
8. Engagement
8.1: To what extent does the article’s storyline have a ‘hook’ to
engage students?
8.2: To what extent does the article have an engaging storyline?
8.3: To what extent does the article include human factors such as
cultural, socio-political factors, and ethical issues?
8.4: To what extent does the article include controversy, contrast,
conflict, dilemma, or other dramatic elements?
8.5: To what extent does the article gradually disclose the content?
8.6: To what extent does the article allow students to ‘learn by
doing’?
Farhoomand (2004); Hackney et
al. (2003); Jewels et al. (2003);
Kim et al. (2006); Rosemann and
Vessey (2008); Senn (1998);
Taylor (2007); Yue (2012)
9. Relevance to practice
9.1: To what extent does the article describe current practitioner
issues?
9.2: To what extent does the article contribute with an
implementable approach to resolve a practical issue?
9.3: To what extent does the article stimulate a reader’s casual
assumptions by identifying emerging trends, structural changes
or paradigms?
9.4: To what extent does the article reflect collaboration between
researchers and practitioners?
Benbasat and Zmud (1999); Kavan
(1998); Rosemann and Vessey
(2008)
10. Teaching objectives focus
10.1: To what extent is the article applicable to the subject area?
10.2: To what extent does the article fit into the teaching objectives of
the subject?
10.3: To what extent does the difficulty of the article match the ability
of students in the subject?
Cappel and Schwager (2002);
Jewels et al. (2003); Kim et al.
(2006); Taylor (2007); Yue (2012)
11. Thinking skills development
11.1: To what extent does the article enable students to develop
problem solving skills?
11.2: To what extent does the article enable students to develop
critical thinking skills?
Benbasat and Zmud (1999);
Farhoomand (2004); Hackney et
al. (2003); Jewels et al. (2003);
Kim et al. (2006)
Table 1: Category Framework
5 Article Evaluation Methodology
The process for the evaluation of article suitability for teaching is described in this section. The
process consists of two phases ‘individual teaching program’ and ‘individual article’ and within each
phase are the steps required to define the weight and importance of criteria as well as to score the
criteria (see Figure 2). The evaluator rates subjectively different factors on three steps (blue boxes),
the rest of the steps are to make calculations to produce the final article’s rating. This methodology is
based on that suggested by (Adelman et al. 1985; Goicoechea et al. 1992; Maynard 1997; Maynard et
al. 2001).
Figure 2: Article Evaluation Methodology
In addition, a MS Excel prototype was developed to support the process by calculating automatically
the scores involved in the methodology. Thus, the evaluator is required only to introduce three values:
Australasian Conference on Information Systems Zurita et al.
2015, Adelaide, South Australia Evaluating Research Articles for InfoSec Management Teaching
6
the category importance (CatImp) in Step 1, the criteria importance (CriImp) in Step 3, and the
criteria score (CriSco) in Step 5 to determine article ratings.
Step 1: Define the rating for each category considered in the evaluation process
The evaluator rates the importance of each category (CatImp) to their teaching. A rating of five
represents the highest importance rating; one represents the lowest importance rating; and zero
represents categories that are not considered. Let us consider a scenario where an evaluator rates
Clarity as important, whereas Succinctness is rated as somewhat important. For the purpose of this
example the remaining criteria are rated as “not applicable” (Figure 3).
Figure 3: Step 1
Step 2: Evaluate the normalised weight of each category
The tool determines the value of the normalised weight of each category (CatNorWei) by calculating
the category importance (CatImp) divided by the sum of all the categories importance.
=
∑
In our scenario, the tool would determine the value of the normalised weight of categories rated with
non-zero values (see Figure 4).
For Clarity, =
, … = 0.
For Succinctness: =
, … = 0.
Figure 4: Step 2
Step 3: Define the rating for each criterion considered in the evaluation process
The evaluator rates the importance of each criterion (CriImp) to the teaching program. A rating of five
represents the highest rating; one represents the lowest rating; and zero represents criterion that are
not considered. In the case of the scenario, the evaluator rates the criterion in the categories that were
rated with non-zero values (Clarity and Succinctness) (Figure 5).
Australasian Conference on Information Systems Zurita et al.
2015, Adelaide, South Australia Evaluating Research Articles for InfoSec Management Teaching
7
Figure 5: Step 3
Step 4: Evaluate the normalised weight of each criterion
The tool determines the value of the normalised weight of each criterion (CriNorWei) by calculating
the criterion importance (CriImp) divided by the sum of all the criterion importance.
=
∑
In the scenario, the tool would determine the value of the normalised weight of both categories and
criteria for those rated with non-zero values (See Figure 6).
For Clarity (1 Criterion), =
, … = 1.00 (100%)
For Succinctness (2 criteria), =
, … = 0. (44.44%)
=
, … = 0. (55.56%)
Figure 6: Step 4
Step 5: Define to what extent the article address each criterion
The evaluator rates the score of how well the article addresses each criterion (CriSco). A score of five
represents the highest score; one represents the lowest score; and zero represents criterion that are
not considered. In the scenario, the evaluator rates the criterion score in the categories or criteria that
received non-zero importance values (Figure 7).
Figure 7: Step 5
Step 6: Evaluate the Articles rating in each category
Australasian Conference on Information Systems Zurita et al.
2015, Adelaide, South Australia Evaluating Research Articles for InfoSec Management Teaching
8
The tool determines the final score of each category (CatSco) by calculating the sum of each criterion
score (CriSco from step 5) multiplied to the criterion normalized weight (CriNorWei step 4).
= ∑CriSco CriNorWei
5
In the scenario, the tool would determine the value of the score of both categories that were rated in
importance as different from zero (See Figure 8)
In Clarity (1 Criterion), =
, … = 0.80 (80%)
In Succinctness (2 criteria), = ..
, … = 0.67 (66.67%)
Figure 8: Step 6
Step 7: Evaluate the Article’s Rating
The tool determines the final rating of the article (ArtRat) by calculating the sum of each category
rating (CatSco from Step 6) multiplied to the category normalized weight (CatNorWei Step 2).
=
From the Step 6 Example: the tool determines the final rating of the article by calculating the sum of
each category rating multiplied by their weight (Figure 9). Considering only non-zero rated categories
(2 Categories), = (0.80 0.67 0.67 0.33, … = 0.7556, … ArtRat = 75.56 %
Figure 9: Step 7
This example shows the complete process of the two mentioned parts of the methodology. First, rating
the importance of the categories and criteria for the evaluator’s teaching program. Second, rating one
article based on the first part. In a real case example, the evaluator would rate more articles to
compare their ratings. To get the total score of another article, the evaluator needs to repeat the
methodology from Step 5.
6 Discussion: The Utility of the ‘Suitability of Research Paper
Criteria’ to Information Security Management
In this section we suggest how the suitability criteria can be used to address the gaps in ISM guidance
from available textbooks. In order to do this we focus on the criteria related to suitability to teaching
Australasian Conference on Information Systems Zurita et al.
2015, Adelaide, South Australia Evaluating Research Articles for InfoSec Management Teaching
9
topics (categories 9, 10 and 11), rather than the criteria aimed at evaluating the suitability to classroom
teaching.
6.1 Relevance to Practice across Industries and Contexts (Category 9)
ISM instructors may be interested in exposing students to ISM practices in a range of contexts. For
example, relating to industry sectors ISM plays a key role in: (1) Critical Infrastructure Protection
(CIP) from National Security threats (see Theoharidou et al. (2007) for an argument for CIP to be
included in curricula and Beraud and Ahmad (2011)) for a discussion of why risk methods should
consider CIP); (2) protecting private enterprise from local and international competitors (see a range
of ISM controls to protect competitive advantage in Ahmad et al. (2014a); and (3) privacy of personnel
information in public organisations (Bélanger and Crossler 2011). Further, security issues in Small to
Medium-sized Enterprise (SMEs) are frequently different to that of larger organisations (Barlette and
Fomin 2008; Ng et al. 2013). ISM is strongly influenced by differentiators among employees such as
national culture (Ifinedo 2009) and behavioural archetypes (Crossler et al. 2013). ISM is also
influenced by organisational differentiators such as organisational culture (Lim et al. 2010; Lim et al.
2009) and governance (Koh et al. 2005). All of these issues are critical for ISM practitioners to
consider when developing effective security strategy in organisations.
ISM instructors may also look at deficiencies in the way organisations implement security guidance
e.g. from industry standards. For example, studies have pointed out deficiencies in the
implementation of information security risk assessment (ISRA) methods (e.g. see Shedden et al.
(2010a). Similarly, there have been a number of studies looking at deficiencies in the incident
response process (Ahmad et al. 2012; Ahmad et al. 2015; Tøndel et al. 2014). Although these case
studies are hard to find because organisations rarely give access to their sensitive information and
functions (see Kotulic and Clark (2004) and Tøndel et al. (2014)), however they provide valuable
insights for students that relate to real-world ISM challenges.
Instructors will find that the discussion of ISM in most textbooks tends to take a narrow view of the
range of formal and informal controls that fall under ISM (see Dhillon (2006) for a discussion on the
distinctions between formal, informal and technical controls and Whitman and Mattord (2014) as an
example of a management-oriented textbook that covers a range of security controls).
First, the emphasis continues to remain on traditional controls such as Policy, Risk and SETA whilst
neglecting other critical areas such as intra-organisational liaison (communication, collaboration and
coordination) between ISM and other parts of the organisation (Alshaikh et al. 2014), as well as the
core security strategy process (see Baskerville and Dhillon (2008)). It is also unclear whether ISM
should include Digital Forensic Readiness (see Elyas et al. (2015) for a management perspective on
Digital Forensics and commentary on security contributions). Further, the discussion of managerial
activities remains at a high-level, which does not provide enough detailed guidance for organisations
seeking to implement the functions internally (Alshaikh et al. 2014).
6.2 Teaching Objectives: Imparting the Management Perspective of
Information Security (Category 10)
The primary teaching objectives for instructors in Information Security Management is how to
prepare students for a career in the discipline by providing: (1) an understanding of the management
perspective of Information Security; and (2) access to knowledge that is relevant to real-world practice
across a range of industries and contexts where ISM may be applied (see 6.2) (Ahmad and Maynard
2014; Martini and Choo 2014).
Regarding the first objective, ISM instructors will struggle to impart an authentic management
perspective of Information Security without falling into the conventional ‘IT Security’ discourse typical
in ISM textbooks. The ISM instructor can find a number of recent papers that depart from the
traditional view of information assets as being discrete, enumerated and situated in the formal
business process (e.g. see Shedden et al. (2009); Shedden et al. (2011); Shedden et al. (2010b) for a
distributed cognitive view of information within informal business practice). Further, a number of
recent studies focus on the security of ‘tacit knowledge’ in ‘human containers’ (see Ahmad et al.
(2014a); Manhart and Thalmann (2015). These papers espouse the idea that enterprise security must
adopt an ‘information-centric’ rather than ‘IT-centric’ view (see Ahmad and Ruighaver (2005);
Winkler (1996)).
Australasian Conference on Information Systems Zurita et al.
2015, Adelaide, South Australia Evaluating Research Articles for InfoSec Management Teaching
10
6.3 Thinking skills to support ISM practice (Category 11)
A key topic that has been largely neglected is how ISM practitioners should ‘strategize’ by leveraging
their resources to best advantage to address security risk. Security strategy literature has pointed out
that ISM managers facing unpredictable and transient threats in the shape of intelligent adversaries
(e.g. in cases of industrial espionage and cyber terrorism) must adopt a ‘warfare’ mindset (this view
was first presented in Baskerville (2005) and then tested in Baskerville et al. (2014) which implies
security situation awareness (see Webb et al. (2014) for a model of situation awareness) must be
developed in combination with tactical speed and agility.
Some research has looked at the range of strategic and tactical paradigms that can be implemented in
organisations (e.g. see Tirenin and Faatz (1999) and Ahmad et al. (2014b)) however there has not
been much discussion on the particular thinking skills such as game-theoretic approaches needed to
employ a combination of strategies effectively especially in asymmetric situations (e.g. see game-
theoretic approaches in Cavusoglu et al. (2008), and discussion of asymmetry in cyber-physical
situations in Ahmad (2010)).
7 Conclusion
This paper has identified criteria for the evaluation of the applicability of research articles for use in
teaching and has operationalised these criteria into a methodology for article assessment. This
methodology enables academics evaluating, in seven steps, one or more articles to determine their
suitability for teaching use. It allows comparisons of papers to occur to enable the choice of the most
suitable papers for teaching programs.
We consider that the framework and methodology provided answers the proposed research question:
“How can research articles applicability be evaluated for use in teaching?” The
framework and methodology can be used to evaluate research articles suitability for teaching. The
framework provides a set of comprehensive categories to be considered in the assessment of research
articles applicability to teaching. The criteria, formatted as questions, provided evaluators with a set of
requirement to meet in the research articles. Finally, the methodology allows evaluators to follow a
descriptive process to complete this task.
The importance of the criteria is linked to the discipline of the teaching program. For example, we
consider that in practice-base disciplines the category “relevance to practice” is going to be a key
category. The reason is that practice-oriented issues are important for this type of discipline and they
are not sufficiently discussed in literature.
This paper contributes to theory as it has analysed the literature to identify 11 areas and 33 criteria to
analyse papers with the purpose of selecting papers to use in teaching. This is the first study, that the
authors are aware of that has analysed and synthesised this literature to from an analysis framework
which is then used by an evaluation methodology.
The paper also has contributed to practice, especially in the area of paper assessment by academics for
teaching purposes. Using the criteria and methodology proposed will enable academics to make quick
decisions on the suitability of teaching cases and research papers to be used in teaching. This will
reduce the time required for the selection of articles.
This paper has focused mainly on the information systems domain in its search for literature. There
may be research conducted in other domains about incorporation of research papers into teaching.
Future work will identify these and incorporate them into our framework and methodology. The
methodology has limitations as it relies on unbiased assessment of articles and required self-
assessment of the criteria by academics. This requires that academics read the articles very carefully
and that they are very well-versed in the articles reviewed.
There are a number of future projects that build on this work. First, the applicability of text books to
the classroom environment may be able to be assessed. Second, a research project is underway to
assess the literature in the Information Security Domain using this tool, with the aim of selecting
relevant articles for a Masters level Information Security Management course that the authors teach.
8 Bibliography
Abawajy, J.H. 2009. "Design and Delivery of Undergraduate It Security Management Course,"
Advances in Information Security and Assurance (5576), pp 402-411.
Australasian Conference on Information Systems Zurita et al.
2015, Adelaide, South Australia Evaluating Research Articles for InfoSec Management Teaching
11
Adelman, L., Rook, F.W., and Lehner, P.E. 1985. "User and R&D Specialist Evaluation of Decision
Support Systems," IEEE Transactions on Systems, Man, and Cybernetics (15:3), May-Jun, pp
334-342.
Ahmad, A., and Ruighaver, A.B. 2005. "An Information-Centric Approach to Data Security in
Organizations," in: Proceedings of Tencon 2005: 2005 IEEE Region 10. 1-5, R. Harris (ed.).
Melbourne, Australia: Swinburne University.
Ahmad, A. 2010. "Tactics of Attack and Defense in Physical and Digital Environments: An
Asymmetric Warfare Approach," Journal of Information Warfare. (9:1), pp 46-57.
Ahmad, A., Hadjkiss, J., and Ruighaver, A.B. 2012. "Incident Response Teams - Challenges in
Supporting the Organizational Security Function.," Computers & Security (31:5), pp 643-652.
Ahmad, A., and Maynard, S.B. 2014. "Teaching Information Security Management: Reflections and
Experiences," Information Management & Computer Security (22:5), pp 513-536.
Ahmad, A., Bosua, R., and Scheepers, R. 2014a. "Protecting Organizational Competitive Advantage: A
Knowledge Leakage Perspective," Computers & Security (42), pp 27-39.
Ahmad, A., Maynard, S.B., and Park, S. 2014b. "Information Security Strategies: Towards an
Organizational Multi-Strategy Perspective," Journal of Intelligent Manufacturing (25:2), pp
357-370.
Ahmad, A., Maynard, S.B., and Shanks, G. 2015. "A Case Analysis of Information Systems and
Security Incident Responses," International Journal of Information Management (35:6), pp
717-723.
Alshaikh, M., Ahmad, A., Maynard, S.B., and Chang, S. 2014. "Towards a Taxonomy of Information
Security Management Practices in Organisations," in: 25th Australasian Conference on
Information Systems. Auckland, New Zealand.
Barlette, Y., and Fomin, V.V. 2008. "Exploring the Suitability of Is Security Management Standards
for Smes," in: Hawaii International Conference on System Sciences, Proceedings of the 41st
Annual Conference. IEEE, pp. 308-317.
Baskerville, R. 2005. "Information Warfare: A Comparative Framework for Business Information
Security," Journal of Information System Security (1:1), pp 23-50.
Baskerville, R., and Dhillon, G. 2008. "Information Systems Security Strategy: A Process View," in:
Information Security: Policy, Processes, and Practices. Advances in Management
Information Systems, D.W. Straub, S.E. Goodman and R. Baskerville (eds.). Armonk, NY: M.
E. Sharpe., pp. 15-45.
Baskerville, R., Spagnoletti, P., and Kim, J. 2014. "Incident-Centered Information Security: Managing
a Strategic Balance between Prevention and Response," Information & Management (51:1),
pp 138-151.
Bélanger, F., and Crossler, R.E. 2011. " Privacy in the Digital Age: A Review of Information Privacy
Research in Information Systems," MIS quarterly (35:4), pp 1017-1042.
Benbasat, I., and Zmud, R. 1999. "Empirical Research in Information Systems: The Practice of
Relevance," MIS Quarterly (23:1), pp 3-16.
Beraud, P., and Ahmad, A. 2011. "A Process for the Identification of Security Risks from Critical
Infrastructure Interdependencies," Journal of Information Warfare (10:1), pp 48-66.
Cappel, J.J., and Schwager, P.H. 2002. "Writing Is Teaching Cases: Guidelines for Jise Submission,"
Journal of Information Systems Education (13:4), pp 287-294.
Cavusoglu, H., Raghunathan, S., and Yue, W.T. 2008. "Decision-Theoretic and Game-Theoretic
Approaches to It Security Investment," Journal of Management Information Systems (25:2),
pp 281-304.
Crossler, R.E., Johnston, A.C., Lowry, P.B., Hu, Q., Warkentin, M., and Baskerville, R. 2013. "Future
Directions for Behavioral Information Security Research.," Computer & Security (23), pp 90-
101.
Crowley, E. 2003. "Information System Security Curricula Development," in: Proceedings of the 4th
conference on Information technology curriculum ACM, pp. 249-255.
Dhillon, G. 2006. Principles of Information Systems Security. John Wiley and Sons.
Elyas, M., Ahmad, A., Maynard, S.B., and Lonie, A. 2015. "Digital Forensic Readiness: Expert
Perspectives on a Theoretical Framework," Computers & Security (52), pp 70-89.
Farhoomand, A. 2004. "Writing Teaching Cases: A Quick Reference Guide," Communications of the
Association for Information Systems (13:1), p 9.
Fisher, D. 2006. "Using Research Papers as a Tool in Teaching Introductory Statistics," Journal of
Interdisciplinary Mathematics (9), pp 287-295.
Goicoechea, A., Stakhiv, E.Z., and Li, F. 1992. "A Framework for Qualitative Experimental Evaluation
of Multiple Criteria Decision Support Systems," Procedings of the Ninth International
Australasian Conference on Information Systems Zurita et al.
2015, Adelaide, South Australia Evaluating Research Articles for InfoSec Management Teaching
12
Conference Proceedings on Multiple Criteria Decision Making: Theory and Application in
Business, Industry and Government, A. Goicoechea, S. Zionts and L. Duckstein (eds.), New
York: Springer-Verlag, pp 1-17.
Hackney, R.A., McMaster, T., and Harris, A. 2003. "Using Cases as a Teaching Tool in Is Education,"
Journal of Information Systems Education (14:3), pp 229-234.
Hemsley-Brown, J., and Sharp, C. 2003. "The Use of Research to Improve Professional Practice: A
Systematic Review of the Literature.," Oxford Review of Education, (29:4), pp 449-470.
Hsu, C., and Blackhouse, J. 2002. "Information Systems Security Education: Redressing the Balance
of Theory and Practice," Journal of Information Systems Education (13:3), pp 211-218.
Ifinedo, P. 2009. "Information Technology Security Management Concerns in Global Financial
Services Institutions: Is National Culture a Differentiator? ," Information Management &
Computer Security (17:5), pp 372-387.
Jewels, T., Jones, W., and Ford, M. 2003. "A Study of Cases: Evaluating Requirements," in: ACIS
2003 Proceedings. Perth, Western Australia.
Kavan, C.B. 1998. "Profit through Knowledge: The Application of Academic Research to Information
Technology Organizations," Information Resources Management Journal (IRMJ) (11:1), pp
17-22.
Kim, S., Phillips, W.R., Pinsky, L., Brock, D., Phillips, K., and Keary, J. 2006. "A Conceptual
Framework for Developing Teaching Cases: A Review and Synthesis of the Literature across
Disciplines," Medical Education (40:9), pp 867-876.
Koh, K., Ruighaver, A.B., Maynard, S.B., and Ahmad, A. 2005. "Security Governance: Its Impact on
Security Culture," Proceedings of the 3rd Australian Information Security Management
Conference, Perth.
Kotulic, A.G., and Clark, J.G. 2004. "Why There Aren't More Information Security Research Studies,"
Information and Management (41), pp 597-607.
Lim, J.S., Chang, S., Maynard, S., and Ahmad, A. 2009. "Exploring the Relationship between
Organizational Culture and Information Security Culture," in: 7th Australian Information
Security Management Conference. Churchlands, Australia: Edith Cowan University: pp. 88-
97.
Lim, J.S., Ahmad, A., Chang, S., and Maynard, S.B. 2010. "Embedding Information Security Culture
Emerging Concerns and Challenges," in: PACIS 2010 Proceedings. Brisbane, Australia: pp.
463-474.
Lindskog, S., Lindqvist, U., and Jonsson, E. 1999. "It Security Research and Education in Synergy," in:
Proceedings of the 1st World Conference on Information Security Education. Stockholm,
Sweden.
Manhart, M., and Thalmann, S. 2015. "Protecting Organizational Knowledge: A Structured Literature
Review," Journal of Knowledge Management (19:2), pp 190-211.
Martini, B., and Choo, K.-K.R. 2014. "Building the Next Generation of Cyber Security Professionals,"
in: Twenty Second European Conference on Information Systems. Tel Aviv: pp. 1-13.
Maynard, S. 1997. "A Multiple-Constituency Approach for the Evaluation of Decision Support
Systems," Masters thesis, Dept of Information Systems. Monash University.
Maynard, S.B., Burstein, F., and Arnott, D. 2001. "A Multi-Faceted Decision Support System
Evaluation Approach," Journal of decision systems (10:3-4), pp 395-428.
Neuman, W.L. 2006. Social Research Methods: Qualitative and Quantitative Approaches, (Sixth
ed.).
Ng, Z.X., Ahmad, A., and B., M.S. 2013. "Information Security Management: Factors That Influence
Security Investments in Sme's," in: 11th Australian Information Security Management
Conference. Churchlands, Australia: Edith Cowan University.
Peck, W.H. 2004. "Teaching Metastability in Petrology Using a Guided Reading from the Primary
Literature," Journal of Geoscience Education (25:3), pp 284-288.
Robey, D., and Markus, M.L. 1998. "Beyond Rigor and Relevance: Producing Consumable Research
About Information Systems," Information Resources Management Journal (11:1), pp 7-15.
Rosemann, M., and Vessey, I. 2008. "Toward Improving the Relevance of Information Systems
Research to Practice: The Role of Applicability Checks," MIS Quarterly (32:1), pp 1-22.
Senn, J. 1998. "The Challenge of Relating Is Research to Practice," Information Resources
Management Journal (11:1), pp 23-28.
Shedden, P., Scheepers, R., Smith, M., and Ahmad, A. 2009. "Towards a Knowledge Perspective in
Information Security Risk Assessments – an Illustrative Case Study," in: Proceedings of the
20th Australasian Conference on Information Systems. Melbourne, Australia: Monash
University: pp. 74-84.
Australasian Conference on Information Systems Zurita et al.
2015, Adelaide, South Australia Evaluating Research Articles for InfoSec Management Teaching
13
Shedden, P., Ruighaver, A.B., and Ahmad, A. 2010a. "Risk Management Standards – the Perception of
Ease of Use.," Journal of Information Systems Security (6:3).
Shedden, P., Smith, W., and Ahmad, A. 2010b. "Information Security Risk Assessment: Towards a
Business Practice Perspective," in: Proceedings of the 8th Information Security Management
Conference. Perth, Australia: Edith Cowan University: pp. 127-138.
Shedden, P., Scheepers, R., Smith, W., and Ahmad, A. 2011. "Incorporating a Knowledge Perspective
into Security Risk Assessments," VINE Journal of Knowledge Management (61:2).
Shkedi, A. 1998. "Teachers' Attitudes Towards Research: A Challenge for Qualitative Researchers,"
International Journal of Qualitative Studies in Education (11:4), pp 559-577.
Taylor, R.G. 2007. "Making Molehills out of Mountains: Bringing Security Research to the
Classroom," Journal of Digital Forensics, Security and Law (2:4), pp 43-58.
Theoharidou, M., Stougiannou, E., and Gritzalis, D. 2007. "A Cbk for Information Security and
Critical Infrastructure Protection," in: Fifth World Conference on Information Security
Education. Springer US, pp. 49-56.
Tirenin, W., and Faatz, D. 1999. "A Concept for Strategic Cyber Defense," Military Communications
Conference Proceedings, 1999. MILCOM 1999. IEEE: IEEE, pp. 458-463.
Tøndel, I.A., Line, M.B., and Jaatun, M.G. 2014. "Information Security Incident Management: Current
Practice as Reported in the Literature.," Computers & Security (45), pp 42-57.
Webb, J., Ahmad, A., Maynard, S.B., and Shanks, G. 2014. "A Situation Awareness Model for
Information Security Risk Management.," Computers & Security (44), pp 391-404.
Whitman, M.E., and Mattord, H.J. 2014. Principles of Information Security. Course Technology,
Cengage Learning.
Winkler, I.S. 1996. "Information Security Is Information Security," IBM Systems Journal, (35).
Yue, T. 2012. "How to Write a Good Teaching Case," RSM Case Development Centre, p. 8.
Copyright: © 2015 Zurita, Maynard, Ahmad. This is an open-access article distributed under the
terms of the Creative Commons Attribution-NonCommercial 3.0 Australia License, which permits
non-commercial use, distribution, and reproduction in any medium, provided the original author and
ACIS are credited.
