Simulated Phishing Attack and Embedded Training Campaign

Abstract

Phishing attacks are costly for both organizations and individuals, yet existing academic research has provided little guidance on how to strategize and implement a combined phishing awareness and training campaign. Drawing on operant conditioning theory, we conduct an in-depth case study on a large phishing awareness campaign and reveal that phishing awareness is a learning process through which individuals’ behavior can be strengthened by reinforcement and punishment. Based on the case study findings, we present several propositions for cybersecurity stakeholders. This study contributes to the phishing awareness literature and has implications for research and practice. This paper is useful for organizations planning or in the process of implementing or reviewing a phishing awareness and education program.

Full text

Full Terms & Conditions of access and use can be found at
https://www.tandfonline.com/action/journalInformation?journalCode=ucis20
Journal of Computer Information Systems
ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/ucis20
Simulated Phishing Attack and Embedded Training
Campaign
William Yeoh, He Huang, Wang-Sheng Lee, Fadi Al Jafari & Rachel Mansson
To cite this article: William Yeoh, He Huang, Wang-Sheng Lee, Fadi Al Jafari & Rachel Mansson
(2021): Simulated Phishing Attack and Embedded Training Campaign, Journal of Computer
Information Systems, DOI: 10.1080/08874417.2021.1919941
To link to this article: https://doi.org/10.1080/08874417.2021.1919941
Published online: 30 Aug 2021.
Submit your article to this journal
Article views: 80
View related articles
View Crossmark data

Simulated Phishing Attack and Embedded Training Campaign
William Yeoh
a
, He Huang
a
, Wang-Sheng Lee
b
, Fadi Al Jafari
a
, and Rachel Mansson
a
a
Deakin University, Burwood, Australia;
b
Monash University, Clayton, Australia
ABSTRACT
Phishing attacks are costly for both organizations and individuals, yet existing academic research
has provided little guidance on how to strategize and implement a combined phishing awareness
and training campaign. Drawing on operant conditioning theory, we conduct an in-depth case
study on a large phishing awareness campaign and reveal that phishing awareness is a learning
process through which individuals’ behavior can be strengthened by reinforcement and punish-
ment. Based on the case study ndings, we present several propositions for cybersecurity stake-
holders. This study contributes to the phishing awareness literature and has implications for
research and practice. This paper is useful for organizations planning or in the process of imple-
menting or reviewing a phishing awareness and education program.
KEYWORDS
Cybersecurity; phishing
awareness; phishing
education; simulated
phishing attack; embedded
training
1. Introduction
Phishing attacks are causing multibillion-dollar losses to
businesses and individuals.
1
Phishing is the process of
attempting to obtain valuable information, such as login
credentials and credit card details, and can result in
prominent payoffs for cybercriminals in terms of
money or valuable data.
2
Most people have at least one
online account that contains sensitive data, such as
financial, health, or personal information.Olmstead
and Smith
3
Since the COVID-19 pandemic, the threat
of phishing has become even more serious because the
distractions of home offices can make employees more
susceptible to phishing e-mails. Recently, three higher
education institutions in the United States were targeted
by cybercriminals using ransom tactics.
4
Many of these
ransomware attacks began with a phishing e-mail.
4
Phishing is an attack against people
5
; thus, it is crucial
to raise individual awareness about phishing to protect
sensitive information from unauthorized access.
6,7
Consequently, phishing awareness campaigns are
becoming increasingly urgent.
8
Such campaigns repre-
sent the best defense tool for organizations against cyber
threats.
9,10
However, phishing awareness is a constantly
changing process that must maintain its standards by
being continuously measured and managed.
11
Some
organizations strengthen individuals’ phishing aware-
ness by employing continuous phishing awareness and
educational campaigns.
9,10,12
The aim of the phishing
awareness strategy is to educate users so they can recog-
nize phishing e-mails and take appropriate action.
13
Sun, Yu, Lin and Tseng
14
stated that increasing users’
security knowledge can reduce the possibility of phish-
ing e-mails deceiving them. Aleroud and Zhou
10
recom-
mended using phishing intelligence quotient testing,
such as simulated phishing attack, and educating indi-
viduals as two main ways to increase cybersecurity
awareness. The study of Kumaraguru, Cranshaw,
Acquisti, Cranor, Hong, Blair and Pham
15
suggested
that embedded training is an important means to
improve phishing awareness, which can gain additional
effects after repeated executions.
The extant literature shows some prior studies on
simulated phishing attacks, phishing education, and
the effectiveness of phishing awareness education (as
outlined in Section 2.3), but hitherto there is limited
research focused on combined phishing awareness
campaigns that comprise simulated phishing attacks
and embedded phishing education. Thus, this paper
provides an in-depth case study of a large organiza-
tion’s combined simulated phishing attacks and
embedded training campaign. Further, we adopt oper-
ant conditioning theory to study the effect of the com-
bined campaign on individuals, as phishing awareness
is a learning process through which individuals’ beha-
vior can be strengthened by operant of reinforcement
and punishment.
16–18
Existing studies have focused on
methods for generating phishing awareness, but over-
looked the individual’s stimulus–response associations.
Hence, we conduct this case study from the theoretical
lens of operant conditioning theory.
CONTACT William Yeoh william.yeoh@deakin.edu.au Department of Information Systems and Business Analytics, Deakin University, Burwood,
Australia
JOURNAL OF COMPUTER INFORMATION SYSTEMS
https://doi.org/10.1080/08874417.2021.1919941
© 2021 International Association for Computer Information Systems

The remainder of this report is organized as follows.
Section 2 reviews the related literature, while Section 3
outlines the simulated phishing campaign, and Section 4
presents the phishing awareness and embedded training
framework. Section 5 presents the case study findings,
and Section 6 presents the propositions and implications
for research and practice. Finally, Section 7 concludes
the paper and suggests ideas for future research.
2. Literature review
2.1. Theoretical foundation
According to Skinner, the father of operant condition-
ing theory, operant conditioning is a behaviorism
learning method that occurs through rewards and
punishments.
16–18
Based on Thorndike’s law of effect,
which considers that behavior with pleasant conse-
quences is likely to be repeated, and vice versa,
Skinner introduced an element called ‘reinforcement’.
Skinner’s experiments revealed that an organism’s
behavior could be modified through ‘reinforcement’ –
that is, behavior that is reinforced tends to be strength-
ened, whereas behavior that is not reinforced tends to
be weakened.
16–18
The operant conditioning theory is useful to this
research because a phishing awareness campaign is an
arrangement of contingencies of reinforcement. In this
case study, the cybersecurity team organizes special con-
tingencies (i.e., phishing awareness and training campaign)
that expedite learning and increase phishing awareness,
which would otherwise be acquired slowly.
19
Drawing on
Skinner’s operant conditioning theory, Table 1 depicts the
four quadrants of operant conditioning. In the case orga-
nization, a congratulatory message was sent when
a phishing e-mail was detected and reported by an indivi-
dual, thereby reinforcing and rewarding the correct beha-
vior. In contrast, when a phishing e-mail was opened, the
individual was punished via embedded training require-
ments (more detail is presented in Section 4).
2.2. Social engineering
Mann
20
defined social engineering as a method of trick-
ing people into disclosing information or performing an
operation. Generally, attackers rely on a variety of
psychosocial techniques to manipulate victims, such as
impersonating authority or using curiosity.
5
There are
several types of cybersecurity attacks through social
engineering, with phishing e-mails being the most com-
mon example of social engineering attack.
21
Phishing
attackers use deceptive methods to trick individuals
into revealing confidential information by imitating
trusted individuals and/or organizations.
22
That is,
a phishing attack applies social engineering strategies
to structure messages to trick victims into opening
attachments, clicking embedded hyperlinks, or entering
information.
23
To examine the fraudulent operations and skills
used in phishing, Atkins and Huang
23
collected numer-
ous phishing e-mails from the MillerSmiles website and
found that eight types of social engineering techniques
were applied to the e-mails. The techniques are author-
ity, attraction/excitement, urgency, fear/threat, tradi-
tion, pity, politeness, and formality. Among these, the
most popular technique is urgency, with 71% of the
researched phishing e-mails containing an emergency
statement.
23
Cybercriminals use emergency statements
to make recipients feel a sense of urgency and convince
them to respond to e-mails rapidly. E-Mail subject lines
containing urgent keywords are often used to attract
readers’ attention. The purpose of using fear/threat is
to make victims worry about the consequences if they
do not respond rapidly – for example, their account can
no longer be used. Cybercriminals also make polite
statements in e-mails in an attempt to build friendly
relationships with potential victims to gain their
trust.
23
Social engineering attacks are recognized as easy to
implement and difficult to defend against.
5
The attacks
focus on the individual’s psychology, and technical pro-
tection measures often cannot effectively resist them.
5
Individuals who tend to think that phishing will not
occur to them are actually more vulnerable to social
engineering attacks.
23
Hence, making individuals face
simulated phishing attacks can increase their awareness
of the cyber threat.
24
2.3. Phishing awareness education
As outlined in Table 2 below, education is the best way
to resist social engineering attacks because it reduces the
Table 1. Four quadrants of operant conditioning.
Positive stimulus Negative stimulus
Reinforcement
(increase behavior)
Positive reinforcement: receive congratulatory message when
phishing e-mail is reported
Negative reinforcement: embedded training not needed
when phishing e-mail is reported
Punishment (decrease
behavior)
Positive punishment: undergo embedded training when phishing
e-mail is opened to decrease behavior
Negative punishment: not applicable in this case
2YEOH ET AL.

Table 2. Summary of phishing training studies.
Type Sub-type Advantage Disadvantage Reference Context Method Result
Anti-phishing
delivery
method:
Advertising
and publicity
Paper or
electronic
flyers,
newsletters,
posters
Wide coverage Easily ignored by users.
10
Xiong
25
Publicity of anti-phishing message Compared the phishing
warnings distributed by
Chrome with two built-in
training warnings
Embedded training can make up for the
lack of phishing webpage detection
skills
Bada, Sasse &
Nurse
26
Existing cybersecurity campaigns
in the United Kingdom and
Africa
Campaign cybersecurity Need to provide feedback while
providing safety information;
continuous training and feedback
Training course Onsite lecturer
guidance
Instructors interact
with students and
teach flexibly
9
Expensive,
27
boring and
ineffective,
28
tends to
fail because it is based
on rote.
29
Reinheimer
30
Among 2,200 employees of the
German State Office for
Geoinformation and State
Survey (SOGSS)
After onsite for six months, three
methods of text, video, and
interactive examples used for
intensive training
Those who watched video quizzes or
interactive examples had a higher
level of safety awareness and
knowledge
Nicholson
31
83 teenagers aged 12 to 17 years Onsite instruction by instructor,
and test conducted
immediately afterward
Training can effectively educate people
about phishing attacks
Carella
32
150 undergraduates from
Northern Kentucky University
Onsite training and embedded
training
Onsite training works best in the short
term; persistence of the embedded
training method has greatest effect on
phishing campaign
Education
videos
Friendliness and
flexibility; users
can obtain
maximum
freedom
Monotonous, not
challenging, and unable
to provide users with
further dialogue
33
Valentine
27
Employee training for retail chain
stores
Education videos only Education videos training is effective
Cone et al.
29
Students from United States Naval
Graduate School
Video tutorials, and CBT; CBT is
a web-based slide
presentation
Users try to complete a CBT session with
the least amount of time or thought
Karumbaiah
34
422 undergraduate students at
Northeastern University in the
United States
(1) Video training; (2) simulated
phishing training with just-in-
time (JIT) training; (3)
phishing rankings
Both video training and rankings reduce
click-through rate of phishing e-mails
Gordon
35
5,416 employees of a tertiary
academic medical center
Online training courses Online learning may not be the most
effective way to provide phishing
training
(Continued)
JOURNAL OF COMPUTER INFORMATION SYSTEMS 3

Table 2. (Continued).
Type Sub-type Advantage Disadvantage Reference Context Method Result
For personal
training
Game Can challenge,
motivate and
engage
participants
Not suitable for everyone
and all ages
Canova
et al.
36
Smartphone context Developed a game-based
smartphone application –
NoPhish
It is effective
Wen
37
In universities What.Hack simulated fishing
game, the non-gamified
training and another game
[Anti-Phishing Phil]
What.Hack is better than the other two
Sheng et al.
38
Gamification Game, text, and multimedia Game is better
Embedded
training
based on
simulation
environment
The easiest for users
to focus and
learn
33
May increase employee
stress
13
;causes safety
fatigue effect
39
Caputo
et al.
40
1,359 participants from a medium-
sized organization in
Washington
Embedded training and onsite
training
Embedded training used in the study is
more effective
Kumaraguru
et al.
41
Embedded training Embedded training with text and
picture, and method of
advertising and publicity
(security notifications)
Embedded training is better than
sending security notifications; comic
intervention is most effective
approach
Kumaraguru
et al.
42
Embedded training Embedded training and method
of advertising and publicity
(sending training materials via
e-mail)
Embedded training is more effective
than sending the same training
materials via e-mail
Carella
32
150 undergraduates from
Northern Kentucky University
Onsite training and embedded
training
Onsite training works best in the short
term; persistence of the embedded
training method has greatest effect on
phishing campaign
Form of training
information:
Text
Easy to deliver, uses
fewer resources,
has a wide range
of applications
Monotonous Kumaraguru
15
515 participants Text and picture for embedded
training
Manga-based training method achieves
better results; embedded training
retains knowledge even after 28 days
Jensen
7
355 American university students
and staff familiar with phishing
attacks and receiving regular
rule-based guidance
Text and text plus graphics Text format and text plus graphics format
are equally effective
Abawajy
9
60 volunteer participants Text, game, and video Combined delivery methods are better;
game and video are effective
Picture Intuitive, easy to see,
suitable for a wide
range of people
Limited delivery channels
and devices
Jensen
7
355 American university students
and staff familiar with phishing
attacks and receiving regular
rule-based guidance
Text and text plus graphics Text format and text plus graphics format
are equally effective
Kumaraguru
15
515 participants Text and picture for embedded
training
Manga-based training method achieves
better results; embedded training
retains knowledge even after 28 days
Multimedia More intuitive and
clear explanation,
can carry more
information
Limited delivery channels
and devices, needs more
resource space
Karumbaiah
34
422 undergraduate students at
Northeastern University in the
United States
(1) Video training, (2) simulated
phishing training with JIT
training, (3) phishing rankings
Both video training and rankings
reduced click-through rate of phishing
e-mails
Abawajy
9
60 volunteer participants Text, game, and video Combined delivery methods are better;
game and video are effective
Game Interesting and easy
to attract users’
attention
There are requirements for
users’ interests and skills,
and the scope of
application is narrow
Abawajy &
Kim
33
30 voluntary participants, 25%
received formal safety training;
62% of participants like to play
games
Text, game, and video Combined delivery methods are better
Abawajy
9
60 volunteer participants Text, game and video Combined delivery methods are better;
game and video are effective
(Continued)
4YEOH ET AL.

Table 2. (Continued).
Type Sub-type Advantage Disadvantage Reference Context Method Result
Effective time
dimension:
30 minutes
In-class and
online
education
videos
Effect is better than
other training
methods within
30 minutes
No persistence Karumbaiah
34
422 undergraduate students at
Northeastern University in the
United States
(1) Video training, (2) simulated
phishing training with JIT
training, (3) phishing rankings
Both video training and rankings reduce
click-through rate of phishing e-mails
Carella
32
150 undergraduates from
Northern Kentucky University
Onsite training and embedded
training
Onsite training works best in the short
term; persistence of the embedded
training method has greatest effect on
phishing campaign
7 days Embedded
training
Enduring Easy to cause user safety
fatigue
39
Kumaraguru
43
311 employees of Portuguese
companies
Embedded training Decisions made by embedded trained
participants are much better; at least
seven days after training, users retain
the knowledge gained
Pars
44
36 participants Based on embedded design of
game
Improves users’ ability to identify
phishing e-mails, and users retain this
feature for at least one week
16 days Jackson
45
27 users Embedded training Users retain their knowledge after
16 days
28 days Kumaraguru
15
515 participants Text and picture for embedded
training
Manga-based training method achieves
better results; embedded training
retains knowledge even after 28 days
JOURNAL OF COMPUTER INFORMATION SYSTEMS 5

risk of the weakest part of the security chain – the
human factor.
46
Reinheimer, Aldag, Mayer, Mossano,
Duezguen, Lofthouse, von Landesberger and
Volkamer
30
asserted that education can increase indivi-
duals’ understanding of how and why to protect their
networks, and increase the likelihood of identifying and
reporting cybersecurity threat activities.
Multiple approaches to phishing education have been
proposed. Jensen, Dinger, Wright and Thatcher
7
sug-
gested a mindfulness training approach that teaches
individuals to pay attention to message evaluation, con-
text awareness, and suspicious messages. Arachchilage,
Love and Beznosov
47
developed a game prototype in
mobile devices as an educational tool, and through
experiments found that participants’ threat perception
and perceived severity factors could prompt individuals
to evade threats. Kumaraguru, Sheng, Acquisti, Cranor
and Hong
48
developed a phishing education system
based on e-mail and an online game. Wen, Lin, Chen
and Andersen
37
developed a simulated fishing game
named “What.Hack” and conducted a comparative
study with this game. Canova, Volkamer, Bergmann
and Borza
36
built a game-based smartphone application
called “NoPhish” that teaches people to distinguish
between legitimate and illegal network addresses.
Tseng, Chen, Lee and Weng
49
proposed a phishing
attack framework and developed a phishing educational
game that can verify its effectiveness. Sheng, Magnien,
Kumaraguru, Acquisti, Cranor, Hong and Nunge
38
compared games to reading phishing tutorials and
found that games are an effective way to better teach
people to fight phishing and other security attacks.
Kumaraguru, Rhee, Sheng, Hasan, Acquisti, Cranor and
Hong
42
and Xiong, Proctor, Yang and Li
25
found that it is
more effective to provide embedded training after an indi-
vidual has been attacked than to send the same education
materials via e-mail. When individuals realize that they
cannot identify phishing e-mails, they are more likely to
value the subsequent education and training.
22
Hence,
phishing simulation represents an effective awareness exer-
cise to trigger individual training need.
6
For embedded
training, Kumaraguru, Rhee, Acquisti, Cranor, Hong and
Nunge
41
and Jensen, Dinger, Wright and Thatcher
7
recom-
mended the use of a combination of text and graphic notes
about phishing. Eminağaoğlu, Uçar and Eren
50
and
Reinheimer, Aldag, Mayer, Mossano, Duezguen,
Lofthouse, von Landesberger and Volkamer
30
revealed
that education through interactive content is effective in
raising awareness about phishing security. In line with
behavioral science principles,
51
Kumaraguru, Cranshaw,
Acquisti, Cranor, Hong, Blair and Pham
15
suggested that
embedded training allows subjects to retain phishing
knowledge for up to 28 days.
Although studies on social engineering and phishing
awareness education exist, there is little research that
specifically studies simulated phishing attacks combined
with embedded training campaigns, and their effect on
individuals. Thus, this study seeks to enhance the con-
textual understanding of the underexplored phishing
awareness campaign together with an embedded train-
ing program through an in-depth case study.
3. Case study
This case study was conducted at a large higher educa-
tion institution in Australia. The main goals of the
phishing campaign were to: (1) reduce the number of
people responding to phishing e-mails; (2) increase the
number of people reporting phishing e-mails to allow
the cybersecurity response team to contain any damage
that could be caused by those e-mails, and (3) identify
any groups within the organization that are relatively
more vulnerable to phishing attacks.
The case organization has over 10,000 individuals
across several campuses and offices around the world.
The large staff size ensures that the study results repre-
sent all kinds of individuals from senior management to
casual workers. The total participants included 10,928
individuals (full-time, part-time, casual workers and
visiting scholars) from 16 departments. The department
name has been anonymized at the request of the case
organization. In this paper, P1 to P10 represent profes-
sional departments, while A11 to A16 are academic
departments. Professional departments include such
functional units as finance, information technology,
planning, and research administration, whereas aca-
demic departments include faculties/schools such as
health, engineering, information technology, arts, busi-
ness, and education. As a result of personnel changes in
such a large organization, the participants fluctuated
slightly each month.
To compare the effectiveness of the phishing aware-
ness campaign, the case organization first conducted
a baseline exercise in April 2019. Using a commercial
simulated phishing platform, a total of 14 different types
of phishing e-mails were distributed in the baseline
exercise. From July 2019 to January 2020, a total of six
cycles of phishing awareness exercises were conducted,
with a total of six different types of phishing e-mail
attacks, with each cycle one month apart (about
28 days – see Table 3). The behavioral science principle
suggests that a habit can be cultivated with further
reinforcement within 28 days
51
; hence, the training
cycle was set once every month, except for August
because this was the semester break with a low number
of individuals.
6YEOH ET AL.

The type of simulated phishing attack that an indivi-
dual would receive via e-mail depended on the month
that the phishing attack was distributed. Each phishing
e-mail contained components that recorded whether an
individual acted with the e-mail, such as replying, open-
ing, clicking on an embedded link, or reporting. If an
individual fell victim to the phishing e-mail, the indivi-
dual would be led to a phishing video education page
created by the cybersecurity team. The campaign did not
collect or store individuals’ passwords or other sensitive
information, even if the subjects entered them into the
form fields.
The phishing training suggested that if individuals
landed on the training page immediately after making
a mistake, they would not behave similarly in the future.
In this case, two changes in individual behavior resulting
from the training were observed and measured: (1)
reduced unsafe behavior, and (2) increased phishing
reporting rates. During the campaign, each individual
would receive a phishing attack e-mail each month. The
details of the types of simulated phishing attacks are
provided in the appendix. As depicted in the appendix,
the typical technique was to induce subjects to click
suspicious links or open phishing attachments via
e-mail. Subjects should identify phishing e-mails
through the following four elements:
●the name and address in the e-mail sender field do
not match
●errors exist in the e-mail, such as misspellings,
grammatical errors, or incorrect spaces
●the e-mail encourages immediate action
●when hovering one’s mouse between the link text
and the displayed link address, the link text does
not match.
4. Phishing awareness and embedded training
framework
Drawing on operant conditioning theory and the social
engineering and phishing education literature, Figure 1
depicts our phishing awareness and embedded training
integrative framework. The framework begins with each
individual in the e-mail system receiving an e-mail invit-
ing them to click on a link to an external website.
Depicted as an insecure transaction in Figure 1, if the
individual insecurely responds to the phishing e-mail,
opens an .exe attachment message, or ignores the e-mail,
the individual’s insecure transaction data will be col-
lected. This insecure behavior directs the individuals to
a “redirect to login” page and they receive “embedded
training” on that webpage to make them aware of their
unsafe behavior (see Figure 2). That is, the insecure
subjects are forwarded to a webpage containing the
phishing education information shown in Figure 2.
The webpage contains an educational video on how to
identify phishing patterns and a phishing e-mail exam-
ple. They also receive e-mails with links to
a cybersecurity education website and a compulsory
phishing awareness training module. This action is con-
sidered ‘punishment’ in operant conditioning theory.
However, if an individual reports a simulated phishing
e-mail through a “phishing alert” (a report button
embedded in the organization’s e-mail system), the but-
ton will display “Congratulations, this is a phishing
simulation test.” In addition, individuals reporting real
phishing e-mails will receive an e-mail from the cyber-
security team to thank them and indicate that the
reported phishing e-mail has been blacklisted (see
Figure 3). Hence, this act positively reinforces and
rewards the individual’s correct behavior (i.e., ‘reinfor-
cement’ in operant conditioning theory).
The data from the whole process are recorded in
the database of the commercial simulated phishing
platform (see Figure 4), which is used to study the
effects of simulated cyberattacks and embedded train-
ing. However, confidential or sensitive information
entered by individuals is not recorded and stored.
This phishing awareness and embedded training fra-
mework can also be used as a basic framework for
cybersecurity stakeholders in planning their phishing
awareness program. The framework is universal, is
applicable to any organization, and can be easily
used to implement organizational phishing campaigns.
5. Case study ndings
By comparing a single individual’s behavior between the
baseline and subsequent cycles, it can be determined
whether the single individual learned to take more
secure action against phishing attacks in the new acqui-
sition cycle. Thus, the organization can examine
whether it is possible to cultivate the phishing resistance
of individuals by simulating phishing attacks and com-
bining embedded training.
5.1. Baseline
At the case organization, the baseline phishing e-mail
was sent between April 2 and 5, 2019, and click tracking
was maintained for another five days to attain accurate
information about whether individuals regularly
checked their e-mails. A total of 14 different types of
phishing e-mails were distributed in the baseline exer-
cise. The results are shown in Figure 5 below.
JOURNAL OF COMPUTER INFORMATION SYSTEMS 7

During the baseline period, the count of e-mails deliv-
ered without bouncing was 10,928. Notably, 3,733 indivi-
duals (32.4% of the total) opened the e-mail. Of these, 1,315
(12% of the total) clicked a link in the e-mail (failed the test)
and 15 individuals (0.1% of the total) responded to phish-
ing e-mails. Another 904 (8.3% of the total) individuals
reported the phishing e-mail using the “Phish Alert” button
or by forwarding it to their spam reporting mailbox. In
addition, 7,195 individuals were noted as not taking any
action with the simulated phishing e-mails.
5.2. Six practice cycles
In the next six cycles, the number of active individuals
fluctuated by about 6%, mainly because of ongoing
casual contract employment or termination. Table 4
depicts the overall simulated phishing statistics.
Notably, the number of people who reported phishing
e-mails increased over time, except in October 2019. The
number of people who clicked the phishing links also
decreased. Further, the total number of individuals who
Table 3. Summary of monthly treatments and manipulation method.
Cycle Phishing e-mail
Social engineering
technique
Delivered
(#)
Opened
(#)
Opened (% to
Delivered)
Clicked
(#)
Clicked (% to
Opened)
1 Microsoft: Your Password has Expired a) Authority
b) Credibility
c) Emergency
d) Politeness
11,216 2,922 26.05% 1,159 40%
2 Australian Taxation Office: Failure to Submit Tax
Refund Notice
a) Authority
b) Credibility
c) Emergency
d) Attractive
e) Politeness
11,454 3,893 34% 1,023 26%
3 Dropbox Business is safe a) Authority
b) Credibility
c) Attractive
d) Politeness
11,553 1,527 13% 113 7%
4 Change of Password Required Immediately a) Authority
b) Credibility
c) Emergency
d) Fear/threat
e) Politeness
11035 3151 29% 677 21%
5 Your Flight Ticket Order a) Authority
b) Credibility
c) Fear/threat
d) Form
e) Politeness
7957 2520 32% 610 24%
6 Urgent help required a) Authority
b) Credibility
c) Pathetic
d) Emergency
e) Politeness
10919 3079 28% 804 26%
Figure 1. Phishing awareness and embedded training framework.
8YEOH ET AL.

Figure 2. Embedded training page.
Figure 3. Congratulatory e-mail.
JOURNAL OF COMPUTER INFORMATION SYSTEMS 9

opened phishing e-mails was much lower than the num-
ber of individuals who ignored phishing e-mails. This
occurred because of the organization’s e-mail client pre-
view function, which allows individuals who have not
closed their e-mail account to directly preview most
e-mail content without performing an open operation.
Nonetheless, while on the preview mode of the e-mail
client, the individuals did not choose to click on the
phishing e-mails, so this can be noted as the ability of
the individuals to distinguish between legitimate e-mails
and phishing e-mails.
Figure 6 below shows the phishing click rate by
departments in different months. The phishing click
rate (which represents an insecure behavior) of all
departments showed a downward trend from the base-
line to the following six activity cycles, except for depart-
ment A15. October 2019 was the lowest month for click
rates across all departments, with click-through rates
ranging from 0% to 4%. Among the 16 departments,
33% of individuals in the P9 department clicked on the
external links in phishing e-mails – the department with
the highest click rate at baseline. Interestingly, the fastest
decline was also P9. After the baseline, the probability of
this department’s unsafe response to phishing e-mails
was 0%. In P1, 33% of individuals performed unsafe
behaviors in November. This was the highest level of
unsafe data recorded in the entire phishing campaign.
P2 was the best performing department and did not
experience any insecure behavior. The unsafe probabil-
ity of all other departments was less than 20%.
The number of people reporting phishing e-mails to
the case organization’s “Phish Alert” reporting system
showed an overall upward trend from the baseline per-
iod to the end of the simulated phishing campaign. In all
departments, except the P9 department, the probability
of reporting phishing e-mails reached the highest level in
the final cycle of the campaign (see Figure 7). Only an
average of 8% of the 16 departments during the baseline
period reported phishing e-mails, but this figure reached
the highest average of 35% at the end of the phishing
awareness campaign. In particular, P4 and P10 were the
best performing departments at the baseline period.
These two departments had very similar values of 18%
and 17%. In contrast, P1, P2, and P9 were the three
worst-performing departments in the baseline period,
with no one reporting phishing e-mails.
5.3. Subject analysis
A total of 8,189 individuals were consistently exposed to
the baseline and all six cycles of the simulated phishing
attacks. They represented a group of people with
0
2000
4000
6000
8000
10000
12000
Delivered Didn't Open Opened Ignored after Opening Reported Clicked Replied
10922
7806, 65.84%
3116, 34.16%
901, 40.16% 897, 24.22%
1309, 35.22%
9, 0.4%
Number of subjects
Bar chart of Baseline Phishing Simulation Statistics
Figure 5. Baseline phishing simulation statistics.
Figure 4. Individuals’ behavior records.
10 YEOH ET AL.

a continuous presence at the organization and allowed
us to track the effect of the campaign on a sample that
was fixed and unchanging over time. For this group of
individuals, casual staff clicked on 8.3% of the phishing
e-mails they received, visitors (visiting scholars or
researchers with a case university e-mail account)
clicked on 6.9% of the e-mails, and individuals who
were neither casual staff nor visitors clicked on 6.6% of
the e-mails. A total of 66 individuals in the organization
clicked four or more times (range 0 to 7) on a simulated
phishing e-mail. Of these 66 individuals, the majority of
them were casual staff (54.6%). Thus, this analysis high-
lights other vulnerable subgroups within the organiza-
tion that are easy to overlook and not a focus of
cybersecurity training programs.
With regards to reporting suspicious e-mails, only
137 individuals from this group of 8,189 individuals
(1.7%) reported a simulated phishing e-mail six or
more times (range 0 to 7). Early reporting of such
phishing e-mails can allow the cybersecurity team to
undertake precautionary measures to help protect the
organization; thus, methods to encourage and improve
on this low reporting rate require further attention.
Given that casual staff and visitors comprise
a significant proportion of the population of individuals
who have access to the organization’s network, this is
a vulnerability. A comprehensive cybersecurity program
should consider including training for all vulnerable
subgroups within the organization. This point is parti-
cularly relevant if a malicious individual could use an
account from a casual employee or visitor to infiltrate
the network in the same way as using an account from
an academic or professional staff member.
6. Discussion
Compared with the baseline period, the number of indi-
viduals who responded unsafely in all six cycles showed
a downward trend. The reporting rate of phishing e-mails
also increased substantially in all departments at the end
of the campaign. The October figures were abnormal. The
number of individuals who chose to ignore phishing
e-mails in October was the largest in the entire phishing
campaign, accounting for 86.78% of the total number of
active individuals. This is because the simulated phishing
e-mails in October came from Dropbox, yet the shared
storage drive provided by the case organization is One
Drive; hence, the vast majority of individuals did not trust
the phishing e-mail. This shows that the success rate of
phishing e-mails has a positive correlation with their
relevance to individuals.
There were still some individuals who responded to
phishing e-mails over the six cycles. This occurred
because, when individuals responded to these e-mails,
they did not receive system feedback – that is, they were
not included in the training program. This caused these
individuals to fail to realize they had performed unsafe
behaviors. This shows that if individuals are only exposed
to phishing attacks without training, this will not help
improve their anti-phishing capabilities.
Table 4. Phishing simulation statistics.
Baseline (July 2019) (Sep 2019) (Oct 2019) (Nov 2019) (Dec 2019) (Jan 2020)
Delivered (#) 10,922 11,216 11,454 11,553 11,035 7,957 10,919
Opened (#) 3,116 2,922 3,893 1,527 3,151 2,520 3,079
Opened (% to delivered) 28.53% 26.05% 33.99% 13.22% 28.55% 31.67% 28.20%
Clicked (#) 1,309 1,159 1,023 113 677 610 804
Clicked (% to delivered) 11.98% 10.33% 8.93% 0.98% 6.14% 7.67% 7.36%
Reported (#) 897 955 2,060 422 1,551 1,540 2,794
Reported (% to opened) 28.79% 32.68% 52.92% 27.64% 49.22% 61.11% 90.74%
Replied (#) 9 4 114 1 23 12 0
Replied (% to opened) 0.29% 0.14% 2.93% 0.07% 0.73% 0.48% 0.00%
Did not open 7,806 8,294 7,561 10,026 7,884 5,437 7,840
Did not open (%) 71.47% 73.95% 66.01% 86.78% 71.45% 68.33% 71.80%
0%
5%
10%
15%
20%
25%
30%
35%
P1 P2 P3 P4 P 5 P6 P7 P8 P9 P10 A11 A12 A13 A14 A15 A16
Percentage of Clicked (%)
Department
MultiBar Chart of Clicked(%) VS. Departments in different months
BaseLine
07/2019
09/2019
10/2019
11/2019
12/2019
01/2020
Figure 6. Phishing click rate by department.
JOURNAL OF COMPUTER INFORMATION SYSTEMS 11

The number of individuals who replied to phishing
e-mails in September was the highest in all cycles. This
occurred because the phishing e-mails in September
were impersonating a trusted government organization
(the tax office), which was relevant to many individuals
at the time because it was tax season. In addition, the
phishing e-mail contained several social engineering
techniques that pushed the individuals to respond,
including authority, emergencies, threats, and attractive-
ness, causing individuals to treat this e-mail with less
caution, with a total of 114 people responding to the
e-mail – far exceeding other cycles. In addition, after five
cycles of decline in clicks, there was a slight increase in
clicks on phishing e-mails in the sixth cycle (Table 3).
Our analysis is due to the use of “pathetic” factors in the
phishing attack, so we believe that different persuasive
factors have different effects on users.
Notably, there was no significant difference in the
likelihood of unsafe behavior across all departments.
Even in a faculty that included the Department of
Information Technology, academics with extensive
computer skills and information security knowledge
did not show better anti-phishing capabilities than the
other departments. This shows that phishing is equally
threatening to the cybersecurity of all individuals.
The simulated phishing attacks and embedded train-
ing campaign resulted in increases in the reporting of
actual phishing e-mails by individuals. Moreover, this
campaign had spillover effects on the organization’s
cybersecurity response team. As a result of increased
frequencies in dealing with such incidents, the cyberse-
curity team had a chance to learn how to deal with actual
phishing e-mails and improve their practices and pro-
cesses. Subsequently, this considerably increased their
efficiency in response times.
6.1. Propositions from the case study
The findings from this case study offer several proposi-
tions for cybersecurity stakeholders, as follows.
Phishing awareness education is necessary for every-
one. First, the case study shows that all individuals are
potentially vulnerable to phishing attacks. The percen-
tage of phishing victims at the university’s Department
of Information Technology (highly-educated informa-
tion technology academics) was no lower than in other
departments. This suggests that phishing awareness edu-
cation is necessary for everyone. Hence, we propose that
all individuals who can access the organization’s com-
mon network, including casual workers, must be
included in the phishing awareness and training
campaign.
Multiple cycles of training to reinforce phishing aware-
ness. Second, from the perspective of human behavior,
the case study revealed that multiple 28-day cycles are
essential for individuals to form a habit and be more
alert to phishing attacks. The case organization continu-
ously changed the type of phishing e-mails over six
cycles, and the proportion of individuals who performed
unsafe behaviors declined. Thus, phishing campaign
managers need to organize multiple successive simula-
tion/training cycles to cultivate a phishing awareness
culture. We propose that at least six cycles of training
are required to effectively reinforce phishing awareness.
Phishing templates should be relevant. Third, as not
many individuals in the case opened the “Dropbox”
phishing e-mail, this shows that phishing can be very
sensitive to the template used. Poorly targeted phishing
attacks will not work well, as this case’s Dropbox example
shows, because One Drive was the organization’s default
storage drive. As such, phishing campaign managers
need to employ an authentic phishing template that can
catch alert and highly-educated individuals. Hence, we
propose that the phishing templates should be relevant
and customized to the individuals and the organization.
For instance, the above-mentioned tax refund reminder
is a timely template for the tax return period.
Simulated phishing attacks should always be bundled
with embedded training. Fourth, imparting phishing
awareness and patterns through embedded training is
more efficient and effective because individuals are more
0%
20%
40%
60%
80%
100%
120%
P1 P2 P3 P4 P 5 P6 P7 P8 P9 P10 A11 A12 A13 A14 A15 A16
Percentage of Report(%)
Departments
MultiBar Chart of Report(%) VS. Departments in different months
BaseLine
07/2019
09/2019
10/2019
11/2019
12/2019
01/2020
Figure 7. Phishing report by departments in different months.
12 YEOH ET AL.

appreciative of “education on-demand.” Thus, we pro-
pose that simulated phishing attacks should always be
bundled with embedded training and education, which
also helps organizations reduce the costs of training
employees.
6.2. Implications for research
This research advances the knowledge base in several
ways. First, it contributes to phishing awareness
research focusing on combined simulated phishing
attacks and embedded training campaigns. Previous
studies mainly focused on phishing simulation or
phishing education separately, and did not delve into
the implementation process of the combined initia-
tives, or the effect of the combined initiatives on indi-
viduals. To our best knowledge, this research
represents the first rigorous study to focus on the entire
implementation of the combined simulated phishing
attacks and embedded training campaigns, as well as
their effect on individuals. It extends our contextual
understanding of how to implement a systematic
phishing awareness and embedded training campaign.
The six-cycle campaign discussed above provides
insights into the entire implementation process.
Second, this research adopted the operant condition-
ing theory
17,18
as the theoretical lens to explain the effect
of the combined simulated phishing attacks and
embedded training campaign on individuals. In prior
phishing awareness studies, such as,
52,53
the attention
was on the methods for generating phishing awareness,
while ignoring the role of stimulus–response associa-
tions made by the learner. We confirmed that individual
behavior is guided by stimuli, as an individual chooses
one response instead of another mainly because of the
“operant” of prior conditioning.
28
The case findings
revealed that some individuals reacted unsafely during
the baseline period; however, after undergoing the
embedded training and further simulation cycles, the
probability of individuals’ unsafe behavior decreased,
and the phishing reporting rate increased significantly.
Hence, this study extends the use of operant condition-
ing theory to the phishing awareness and training
context.
Third, this research contributes to the phishing
awareness literature – particularly the literature pertain-
ing to the implementation of phishing simulation and
training. This study presents an integrative phishing
simulation and embedded training framework (as
shown in Figure 1). The framework supplements the
body of research on phishing awareness and offers sys-
tematic guidance for future cybersecurity stakeholders
to plan and implement their own phishing campaign.
6.3. Implications for practice
This paper has implications for practice. First, in
alignment with the practical contribution of
Hatakka, Thapa and Sæbø,
54
this paper offers
a practical base from which cybersecurity stake-
holders can learn. Based on the case study learnings,
we present several propositions for cybersecurity
practitioners and phishing campaign managers. The
propositions represent best practice and are especially
useful for cybersecurity stakeholders who are
involved in planning, reviewing, and implementing
phishing awareness initiatives and training cam-
paigns, as well as cybersecurity communities who
are engaged with anti-phishing initiatives.
Second, we provide an integrative framework on
combined simulated phishing attack and embedded
training (see Figure 1). Cybersecurity practitioners and
decision makers can use the framework as a reference
guide while designing phishing awareness and training
programs. Also, embedded training and educational
videos should be taken into account when designing
phishing awareness programs, as the effect of timely
phishing training and education is considered more
significant when individuals recognize that they have
failed to distinguish phishing e-mails.
25,42
7. Conclusion
Cultivating phishing awareness is becoming increas-
ingly urgent and important in fighting phishing
attacks. Drawing on operant conditioning theory,
this paper examined how combined simulated phish-
ing attacks and embedded training campaign can
increase individuals’ resistance to phishing attacks.
The evidence from the case study confirmed the
applicability of operant conditioning theory and con-
cluded that a structured phishing simulation cam-
paign together with embedded training enhances
individuals’ awareness of phishing.
Like all other studies, this paper has some limita-
tions, which offer opportunities for future research.
First, this study focused on the education sector;
hence, future research can investigate other industry
sectors and compare the results. Second, the indivi-
duals of this study were mostly highly educated. Thus,
future research can investigate this framework in other
educational qualification settings and compare the
findings.
ORCID
William Yeoh http://orcid.org/0000-0002-2964-4518
JOURNAL OF COMPUTER INFORMATION SYSTEMS 13

References
1. Investigation, F. B. O. Business E-mail Compromise
The 12 Billion Dollar Scam. [accessed 2020 Dec 12].
https://www.ic3.gov/Media/Y2018/PSA180712 .
2. Pienta D, Thatcher JB, Johnston A. Protecting a whale
in a sea of phish. J Inf Technol. 2020;35:214–31.
doi:10.1177/0268396220918594.
3. Olmstead K, Smith A. Americans and cybersecurity.
Pew Res Center. 2017;26:311–27.
4. McKenzie L. Cyberextortion Threat Evolves. [accessed
2020 Dec 12]. https://www.insidehighered.com/news/
2020/06/11/colleges-face-evolving-cyber-extortion-
threat .
5. Krombholz K, Hobel H, Huber M, Weippl E. Advanced
social engineering attacks. J Inform Secur Appl.
2015;22:113–22. doi:10.1016/j.jisa.2014.09.005.
6. Jansson K, Von Solms R. Phishing for phishing
awareness. Behav Inf Technol. 2013;32:584–93.
doi:10.1080/0144929X.2011.632650.
7. Jensen ML, Dinger M, Wright RT, Thatcher JB.
Training to mitigate phishing attacks using mindfulness
techniques. J Manage Inform Sys. 2017;34:597–626.
doi:10.1080/07421222.2017.1334499.
8. Zwilling M, Klien G, Lesjak D, Wiechetek Ł, Cetin F,
Basim HN. Cyber security awareness, knowledge and
behavior: a comparative study. J Comput Inf Syst.
2020;1–16. doi:10.1080/08874417.2020.1712269.
9. Abawajy J. User preference of cyber security awareness
delivery methods. Behav Inf Technol. 2014;33:237–48.
doi:10.1080/0144929X.2012.708787.
10. Aleroud A, Zhou L. Phishing environments, techniques,
and countermeasures: a survey. Com Secur.
2017;68:160–96. doi:10.1016/j.cose.2017.04.006.
11. Kruger HA, Kearney WD. A prototype for assessing
information security awareness. Comp Secur.
2006;25:289–96.
12. Shahbaznezhad H, Kolini F, Rashidirad M. Employees’
behavior in phishing attacks: what individual, organiza-
tional, and technological factors matter?. J Comput Inf
Syst. 2020;1–12. doi:10.1080/08874417.2020.1812134.
13. Jampen D, Gür G, Sutter T, Tellenbach B. Don’t click:
towards an effective anti-phishing training.
A comparative literature review. Human-centric Comp
Inform Sci. 2020;10:1–41. doi:10.1186/s13673-020-
00237-7.
14. Sun JC-Y, Yu S-J, Lin SS, Tseng -S-S. The mediating
effect of anti-phishing self-efficacy between college stu-
dents’ internet self-efficacy and anti-phishing behavior
and gender difference. Comput Human Behav.
2016;59:249–57. doi:10.1016/j.chb.2016.02.004.
15. Kumaraguru P, Cranshaw J, Acquisti A, Cranor L,
Hong J, Blair MA, Pham T. School of phish: a
real-world evaluation of anti-phishing training. In:
Proceedings of the 5th symposium on usable privacy
and security. Mountain View California (USA): ACM;
2009. p. 3.
16. Skinner BF. The technology of teaching. New York
(USA): Meredith Corporation; 1968.
17. Skinner BF. The science of learning and the art of
teaching. USA: Harvard Educational Review; 1954.
18. Skinner BF. Operant behavior. Am Psychol.
1963;18:503. doi:10.1037/h0045185.
19. Eugenio FC Jr, Ocampo AJT. Assessing classcraft as an
effective gamification app based on behaviorism learn-
ing theory. In: Proceedings of the 2019 8th international
conference on software and computer applications.
New York (NY); 2019. p. 325–29.
20. Mann MI. Hacking the human: social engineering tech-
niques and security countermeasures. Aldershot (UK):
Gower Publishing, Ltd.; 2012.
21. Btoush M, Alarabeyat A, Zboon M, Ryati O, Hassan M,
Ahmad S. Increasing information security inside orga-
nizations through awareness learning for employees.
J Theore Appl Inform Tech. 2011;24(2).
22. Alazri AS. The awareness of social engineering in informa-
tion revolution: techniques and challenges. In: 2015 10th
International Conference for Internet Technology and
Secured Transactions (ICITST); London (UK): IEEE;
2015. p. 198–201.
23. Atkins B, Huang W. A study of social engineering in
online frauds. Open J Soc Sci. 2013;1:23. doi:10.4236/
jss.2013.13004.
24. Al-Hamar M, Dawson R, Al-Hamar J. The need for
education on phishing: a survey comparison of the UK
and Qatar. Campus-Wide Inform Sys. 2011;28:308–19.
doi:10.1108/10650741111181580.
25. Xiong A, Proctor RW, Yang W, Li N. Embedding train-
ing within warnings improves skills of identifying
phishing webpages. Hum Factors. 2019;61:577–95.
doi:10.1177/0018720818810942.
26. Bada M, Sasse AM, Nurse JR. Cyber security awareness
campaigns: why do they fail to change behaviour?
(Working paper). Global Cyber Security Capacity
Centre, University of Oxford; 2014. https://discovery.
ucl.ac.uk/id/eprint/1468954/1/Awareness%
20CampaignsDraftWorkingPaper.pdf.
27. Valentine JA. Enhancing the employee security aware-
ness model. Comput Fraud Secur. 2006;2006:17–19.
doi:10.1016/S1361-3723(06)70370-0.
28. Leach J. Improving user security behaviour. Com Secur.
2003;22:685–92. doi:10.1016/S0167-4048(03)00007-5.
29. Cone BD, Irvine CE, Thompson MF, Nguyen TD.
A video game for cyber security training and
awareness. Comp Secur. 2007;26:63–72. doi:10.1016/j.
cose.2006.10.005.
30. Reinheimer B, Aldag L, Mayer P, Mossano M,
Duezguen R, Lofthouse B, Von Landesberger T,
Volkamer M. An investigation of phishing awareness
and education over time: when and how to best remind
users. In: Sixteenth Symposium on Usable Privacy and
Security ({SOUPS} 2020); Virtual Event; 2020. p.
259–84.
31. Nicholson J, Javed Y, Dixon M, Coventry L, Ajayi OD,
Anderson P. Investigating teenagers’ ability to detect
phishing messages. In: 2020 IEEE European
Symposium on Security and Privacy Workshops
(EuroS&PW). Genoa (Italy): IEEE; 2020. p. 140–49.
32. Carella A, Kotsoev M, Truta TM. Impact of security
awareness training on phishing click-through rates. In:
2017 IEEE international conference on Big Data (Big
Data). Boston (MA): IEEE; 2017. p. 4458–66.
14 YEOH ET AL.

33. Abawajy J, Kim T-H. Performance analysis of cyber
security awareness delivery methods. In: Security tech-
nology, disaster recovery and business continuity.
Berlin, Heidelberg: Springer; 2010. p. 142–48.
34. Karumbaiah S, Wright RT, Durcikova A, Jensen ML.
Phishing training: a preliminary look at the effects of
different types of training. In: WISP 2016 proceedings.
Dublin (Ireland); 2016. p. 11.
35. Gordon WJ, Wright A, Glynn RJ, Kadakia J,
Mazzone C, Leinbach E, Landman A. Evaluation of
a mandatory phishing training program for high-risk
employees at a US healthcare system. J Am Med Inform
Assoc. 2019;26:547–52. doi:10.1093/jamia/ocz005.
36. Canova G, Volkamer M, Bergmann C, Borza R.
NoPhish: an anti-phishing education app. In: Mauw S,
Jensen CD, editors. Security and Trust Management.
STM 2014. Lecture notes in computer science. Vol.
8743. Cham: Springer. https://doi.org/10.1007/978-3-
319-11851-2_14 .
37. Wen ZA, Lin Z, Chen R, Andersen E. What. hack:
engaging anti-phishing training through a role-playing
phishing simulation game. In: Proceedings of the 2019
CHI conference on human factors in computing sys-
tems. Glasgow (UK); 2019. p. 1–12.
38. Sheng S, Magnien B, Kumaraguru P, Acquisti A,
Cranor LF, Hong J, Nunge E. Anti-phishing phil: the
design and evaluation of a game that teaches people not
to fall for phish. In: Proceedings of the 3rd symposium on
Usable privacy and security. Pittsburgh (PA); 2007. p.
88–99.
39. Stanton B, Theofanos MF, Prettyman SS, Furman S.
Security fatigue. IT Prof. 2016;18:26–32. doi:10.1109/
MITP.2016.84.
40. Caputo DD, Pfleeger SL, Freeman JD, Johnson ME.
Going spear phishing: exploring embedded training
and awareness. IEEE Secur Priv. 2013;12:28–38.
doi:10.1109/MSP.2013.106.
41. Kumaraguru P, Rhee Y, Acquisti A, Cranor LF,
Hong J, Nunge E. Protecting people from phishing:
the design and evaluation of an embedded training
email system. In: Proceedings of the SIGCHI confer-
ence on human factors in computing systems. San
Jose (CA): ACM; 2007. p. 905–14.
42. Kumaraguru P, Rhee Y, Sheng S, Hasan S, Acquisti A,
Cranor LF, Hong J. Getting users to pay attention to
anti-phishing education: evaluation of retention and
transfer. In: Proceedings of the anti-phishing working
groups 2nd annual eCrime researchers summit.
Pittsburgh, (PA); 2007. p. 70–81.
43. Kumaraguru P, Sheng S, Acquisti A, Cranor LF, Hong J.
Lessons from a real world evaluation of anti-phishing
training. In: 2008 eCrime Researchers Summit. Atlanta
(GA): IEEE; 2008; p. 1–12.
44. Pars C. PHREE of Phish: the effect of anti-phishing
training on the ability of users to identify phishing
emails. Enschede (Netherlands): University of Twente;
2017.
45. Jackson C, Simon DR, Tan DS, Barth A. An evaluation of
extended validation and picture-in-picture phishing
attacks. In: International conference on financial crypto-
graphy and data security. Berlin, Heidelberg: Springer;
2007. p. 281–93.
46. Bissell K, LaSalle R, Cin P Accenture’s ninth annual cost
of cybercrime study: unlocking the value of improved
cybersecurity protection, 2019.
47. Arachchilage NAG, Love S, Beznosov K. Phishing threat
avoidance behaviour: an empirical investigation.
Comput Human Behav. 2016;60:185–97. doi:10.1016/j.
chb.2016.02.065.
48. Kumaraguru P, Sheng S, Acquisti A, Cranor LF, Hong J.
Teaching Johnny not to fall for phish. ACM TOIT.
2010;10:1–31. doi:10.1145/1754393.1754396.
49. Tseng -S-S, Chen K-Y, Lee T-J, Weng J-F. Automatic
content generation for anti-phishing education game
In: 2011 international conference on electrical and con-
trol engineering. Yichang (China): IEEE; 2011. p.
6390–94.
50. Eminağaoğlu M, Uçar E, Eren Ş. The positive outcomes
of information security awareness training in compa-
nies–A case study. Inform Secur Techn Rep.
2009;14:223–29. doi:10.1016/j.istr.2010.05.002.
51. Judah G, Gardner B, Aunger R. Forming a flossing habit:
an exploratory study of the psychological determinants of
habit formation. Br J Health Psychol. 2013;18:338–53.
doi:10.1111/j.2044-8287.2012.02086.x.
52. Elbashir MZ, Collier PA, Sutton SG, Davern MJ, Leech SA.
Enhancing the business value of business intelligence: the
role of shared knowledge and assimilation. J Inf Syst.
2013;27:87–105. doi:10.2308/isys-50563.
53. Elbashir MZ, Collier PA, Sutton SG. The role of orga-
nizational absorptive capacity in strategic use of busi-
ness intelligence to support integrated management
control systems. Account Rev. 2011;86:155–84.
doi:10.2308/accr.00000010.
54. Hatakka M, Thapa D, Sæbø Ø. Understanding the role of
ICT and study circles in enabling economic opportunities:
lessons learned from an educational project in Kenya.
Inform Sys J. 2020;30:664–98. doi:10.1111/isj.12277.
JOURNAL OF COMPUTER INFORMATION SYSTEMS 15

Appendix
1. Microsoft: Your Password has Expired
‘Your Password Expired’ e-mail (see Figure A1) pretends to be an e-mail sent by Microsoft Corporation to a customer
stating that their password has expired, urging them to change their password. The e-mail reminds the individual that
they will not be able to access their profile and e-mail after the password expires, and the password is valid for
90 days. It also provides phishing links to entice individuals to click and leak their usernames and passwords on unsafe
websites.
This simulated phishing e-mail sent in July 2019 uses several social engineering techniques to gain user trust, as
follows:
(a) Contact Information. The e-mail uses Microsoft’s logos and fake Microsoft e-mail addresses and names, making the e-mail
look trustworthy.
(b) Format. Mail has the appearance of a formal letter, increasing credibility.
(c) Emergency situations. The e-mail highlighted the emergency (password expired), requiring individuals to respond quickly
and imply threats in a negative tone (passwords can only be used for 90 days).
(d) Politeness. Construct the author as a real person and be friendly to the individual (ends in friendly terms such as sincerely).
Individuals should be vigilant because the e-mail gives external links and asks individuals to change their password. At the same
time, this message has other characteristics of phishing e-mails: incorrect e-mail address, and redirect URL.
Figure A1. Your password has expired.
16 YEOH ET AL.

2. Australian Taxation Office: Failure to Submit Tax Refund Notice
The ATO e-mail (see Figure A2) informs individuals that they fail to complete the tax refund submission, and gives an external
link asking the individuals to click to submit relevant information. At the same time, individuals are reminded that failure to
submit on time may face fines. This message sent in September 2019 uses multiple social engineering techniques to trick
individuals, as follows:
(a) Authority: The e-mail uses the Australian Tax Office logo and a spoofed e-mail address and name to make the e-mail look
trustworthy.
(b) Format. Mail has the appearance of a formal letter, which increases credibility.
(c) Emergency situations. The e-mail highlighted an emergency situation (a tax refund submission has expired and needs to be
submitted within 7 days), requiring individuals to respond quickly and imply threats in a negative tone (non-execution will
result in a fine of $ 5000).
(d) Attraction: The message is implicitly attractive, and the tax refund can get some economic benefits.
(e) Politeness. Build the author as a real person and user-friendly (ending in sincere and friendly terms).
Individuals should be vigilant because the e-mail provides external links and requires individuals to fill out private
information. At the same time, this message has another feature of phishing e-mails: incorrect e-mail addresses and
redirected URL.
3. Dropbox Business is safe
The ‘Dropbox Business is safe’ e-mail (see Figure A3) reminds individuals that Dropbox is safe again, and gives an external link to
tell individuals that they can restore user files after clicking. This message sent in October 2019 uses multiple social engineering
techniques to trick individuals, as follows:
(a) Authority: The e-mail uses the Dropbox logo and a spoofed e-mail address and name to make the e-mail look
trustworthy.
(b) Format. Mail has the appearance of a formal letter, which increases credibility.
(c) Attraction: The message contains appeal and comes from a file that can help individuals recover.
(d) Politeness. Build the author as a real person and be user-friendly (address individuals in a friendly way).
Individuals should be vigilant because the e-mail provides external links and requires individuals to fill out private
information. At the same time, this message has another feature of phishing e-mails: incorrect e-mail addresses and
redirected URL.
Figure A2. Failure to submit tax refund notice.
JOURNAL OF COMPUTER INFORMATION SYSTEMS 17

4. Change of Password Required Immediately
The ‘Change of Password Required Immediately’ e-mail (see Figure A4) reminds the individual that a security breach has occurred
in the organization’s system and needs to immediately change the password and provide an external link. This message sent in
November 2019 uses multiple social engineering techniques to trick individuals, as follows:
(a) Authority: This message uses an e-mail address and name that looks like the institution, making the e-mail look trustworthy.
(b) Format. Mail has the appearance of a formal letter, which increases credibility.
(c) Emergency: This e-mail reminds the individual in an urgent tone that the password needs to be changed immediately.
(d) Fear/threat: This e-mail highlights a security breach in the system that caused the individual to panic.
(e) Politeness. Build the author as a real person and be user-friendly (end with a friendly greeting).
Individuals should be vigilant because the e-mail provides an external link and asks the individual to change their
password. At the same time, this message has another feature of phishing e-mails: incorrect e-mail addresses and
redirected URL.
Figure A3. Dropbox business is safe.
Figure A4. Change of password required immediately.
18 YEOH ET AL.

5. Your Flight Ticket Order
Your ‘Flight Ticket Order’ e-mail (see Figure A5) prompts the individual’s credit card to be charged and asks the individual to
download malicious attachments. If the individual is in doubt, the e-mail gives an external link that impersonates a legitimate URL
to induce the individual to click and enter private information. This phishing e-mail sent in December 2019 uses multiple social
engineering techniques to trick individuals, as follows:
(a) Authority: The message uses the Virgin Australia logo and a fake e-mail address and name to make the e-mail look
trustworthy.
(b) Format. Mail has the appearance of a formal letter, which increases credibility.
(c) Fear/threat: This e-mail reminds the individual that the credit card was charged, causing the individual to panic, causing the
individual to download a malicious attachment or click a fake external link without thinking.
(d) Form: The technical terms such as the fabricated ticket number and departure airport are used to convince the reader that the
e-mail is authentic.
(e) Politeness. Build the author as a real person and be user-friendly (end with a friendly greeting).
Individuals should be vigilant because the e-mail not only provides external links but also requires individuals to download
attachments. At the same time, this message has another feature of phishing e-mails: incorrect e-mail addresses and redirected
URL.
6. Urgent help required
The ‘Urgent help’ required e-mail (see Figure A6) comes from within the organization, prompting for an urgent help request from
a colleague, and enticing individuals to download the attachment. This phishing e-mail sent in January 2020 uses multiple social
engineering techniques to trick individuals, as follows:
(a) Authority: This message uses an e-mail address and name similar to those in the organization’s internal mail, making the
e-mail seem trustworthy.
(b) Format. Mail has the appearance of a formal letter, which increases credibility.
(c) Pity: This e-mail uses the tone of a colleague to ask the individual for help, which arouses the sympathy and compassion of the
individual.
Figure A5. Your flight ticket order.
JOURNAL OF COMPUTER INFORMATION SYSTEMS 19

(d) Emergencies: At the same time, this e-mail highlights emer-
gencies, and individuals often think less in this situation.
(e) Politeness. Build the author as a real person and be user-
friendly (end with a friendly greeting).
Individuals should be alert because the e-mail requires indivi-
duals to download attachments. At the same time, this mes-
sage also has other features of phishing e-mails: wrong e-mail
address, putting download button in the body.
Figure A6. Urgent help required.
20 YEOH ET AL.