Content uploaded by Christopher Mayhorn
Author content
All content in this area was uploaded by Christopher Mayhorn on Mar 01, 2016
Content may be subject to copyright.
KEEPING UP WITH THE JONESES:
ASSESSING PHISHING SUSCEPTIBILITY IN AN EMAIL TASK
Kyung Wha Hong1, Christopher M. Kelley2, Rucha Tempa2,
Emerson Murphy-Hill1 & Christopher B. Mayhorn2
1Department of Computer Science, 2Department of Psychology,
North Carolina State University, Raleigh, NC
Most prior research on preventing phishing attacks focuses on technology to identify and
prevent the delivery of phishing emails. The current study supports an ongoing effort to
develop a user-profile that predicts when phishing attacks will be successful. We sought
to identify the behavioral, cognitive and perceptual attributes that make some individuals
more vulnerable to phishing attack than others. Fifty-three participants responded to a
number of self-report measures (e.g., dispositional trust) and completed the ‘Bob Jones’
email task that was designed to empirically evaluate phishing susceptibility. Over 92% of
participants were to some extent vulnerable to phishing attacks. Additionally, individual
differences in gender, trust, and personality were associated with phishing vulnerability.
Application and implications for future research are discussed.
INTRODUCTION
Cybersecurity involves a complex interaction
between users and technology. While security
threats might take a variety of forms such as viruses
or worms delivered via nefarious websites or USB
drives, theft using social engineering tactics such as
phishing are becoming increasingly common and
costly. Loss of time and increased stress levels are
the immediate personal costs (Hardee, West, &
Mayhorn, 2006). Long term personal costs are
likely as well, such as decreased trust and usage of
the internet for banking, shopping, and other
conveniences (Dhamija & Tygar, 2005; Kelley,
Hong, Mayhorn, & Murphy-Hill, 2012). In terms of
economic losses, a recent survey (Gartner, 2007)
indicates phishing attacks caused a loss of 3.2
billion dollars based on a sample of 4500 adults
with an average of $866 lost per phishing
occurrence. Moreover, phishing targeted at
administrators can compromise entire systems and
user communities (Schwartz, 2011).
The goal of this research is to develop a user-
profile that predicts when and where phishing
attacks will be successful. Such a user-profile could
be useful to help identify behavioral, cognitive, and
perceptual differences that make some users more
susceptible to phishing than others. For instance,
individual differences in trust and cognitive and
attentional capacity have been identified separately
as contributing to phishing susceptibility. However,
no one has constructed a unified user-profile that
combines individual differences to proactively
identify individual users who are prone to being
successfully phished.
METHOD
Participants
Fifty-three undergraduate students were
recruited to complete an experiment (Table 1).
Participants were tested individually in sessions that
lasted approximately two hours and given extra-
credit as compensation.
Materials
The experiment was completed in two stages
such that participants completed an online survey
and then a laboratory session.
Self-report measures. Participants completed a
survey that measured demographic characteristics
such as age, gender, and primary language as well
as previous experiences with phishing, online
purchasing behavior, and general computing
behavior (based on Eveland, Shah, & Kwak, 2003;
Yoshioka, Washizaki, & Maruyama, 2008).
Participants also responded to measures of
dispositional trust (Merritt & Ilgen, 2008),
impulsivity (Neyste & Mayhorn, 2009), and
personality (Gosling, Rentfrow, & Swann, 2003).
Table 1
Participant Characteristics
MRange
Age 20.20 18 - 27
Gende r
Male
Female
Race
Caucasian/Non-hispanic
Language
English Primary
Major
Computer Science
Psychology
Freque ncie s
60%
40%
80%
96%
SD
2.33
34%
66%
Behavioral measures. To empirically assess
phishing susceptibility, participants completed an
email task where they were asked to access a
Google Mail account for a character named “Bob
Jones” and categorize 14 email messages (Table 2).
Table 2
Email Messages Divided by Category
Email Category n
Phishing
7
Spam 1
Malware 1
Legitimate 5
Total 14
Figure 1 shows one of the phishing emails we used
as stimuli in this experiment. This email appears to
be from careerbuilder.com, a legitimate website
representing a real company (even with their logo).
Also it seems to give useful information to the user.
However, if a user clicks on the links included in
the email, it actually leads them to a website that is
not related to careerbuilder’s official website.
Disguising the sender or source of an email by
making it look like a legitimate company is a
typical tactic used to create phishing emails.
Figure 1
Example Phishing Email
Participants were given the following instructions:
When you are going through each email, do
as you normally do. For example, if you
normally read each email carefully do as
you usually do. Or if you usually skim
through each message quickly that’s also
fine, too. After going through an email you
have to make a decision about the email. If
you think email is legitimate and you’d like
to respond (e.g., reply, click on a link,
download a file) to the email, then mark
‘Important’. If you think email is legitimate
but doesn’t need any response and would
like to just archive, leave it as it is. If you
think email is not legitimate, suspicious, or
spam, then ‘Delete’.
Procedure
After providing informed consent and
completion of the self-report measures delivered
online, participants visited the laboratory where a
battery of cognitive tests and the Bob Jones email
task were administered. The cognitive tests included
a measure of working memory capacity (WMC)
(Unsworth, Heitz, Schrock, & Engle, 2005),
crystallized intelligence (Shipley, 1986), spatial
ability (Peters et al., 1995; Vandenberg & Kuse,
1978), and sustained attention (Temple et al., 2000).
Upon completion of the cognitive tests, instructions
for the Bob Jones email task were delivered.
Finally, participants were debriefed and dismissed.
RESULTS
Responses to self-report measures were
captured via an online survey tool, Qualtrics, and
the results of the cognitive tests and the Bob Jones
email task were entered into SPSS for analysis.
Survey Results
Prior phishing experience. Many respondents
indicated that they had previous phishing
experience via email. For instance, 25% reported
glancing at the contents of a phishing email whereas
36% admitted to completely reading a phishing
message. Thirty percent were compelled to ask
someone else whether they thought the email was
authentic whereas 11% reported contacting an
authority (e.g., bank). The most severe phishing
consequences seemed to be relatively rare with 15%
clicking on a link, 8% installing a virus/malware,
and 6% entering personal information. Of those
who entered personal information, name (6%) and
password (6%) comprised the information provided
to phishers. Most frequent consequences of worst
experience included “noticed unusual activity in an
online account” (15%) and “reduced online
activity” (15%). Based on this previous experience,
89% agreed that they were “confident that they can
tell the difference between a legitimate email and
one sent by a scammer.”
Behavioral Results
Bob Jones email task performance. To
ascertain phishing susceptibility, a score that ranged
from 0 (perfect ability) to 100 (no ability) was
calculated for participant’s ability to identify
phishing emails. The data suggested more than 92%
of participants were susceptible to phishing with
only 4 participants (7.5% of the sample)
successfully identifying all of the phishing emails
and approximately 52% misclassifying more than
half of the phishing emails. Since phishing also
impacts the ability of people to identify legitimate
emails, the number of authentic emails that were
incorrectly deleted was assessed. Fifty-four percent
deleted at least one authentic email.
Individual differences correlated with
accuracy. The ability to correctly identify phishing
emails revealed gender, trust, and personality were
correlated with phishing vulnerability. For example,
women were less likely than men to correctly
identify phishing emails, t(51) = -2.15, p < .036.
Dispositional trust, extraversion and openness to
new experience were correlated with deleting
legitimate emails. Specifically, less trusting
individuals, r(52) = -.30, p < .034, introverts, r(53)
= -.29, p < .054, and those less open to new
experiences, r(53) = -.435, p < .002, were more
likely to delete legitimate emails.
Severity of email misclassification. In
addition, because misclassifying some emails could
have more severe consequences than others, five
classes of email severity were created that ranged
from 1 to 5. (Class 1:legitimate email—no danger,
Class 2:spam email or email sent to numerous
recipients—no danger but less useful, Class
3:phishing email redirecting to unexpected site—no
danger, Class 4:phishing email with a danger of
loosing less critical information, Class 5: phishing
email with a danger of losing money or critical
information). Thus, when an email was
misclassified a severity score was assigned based on
the participant’s response (e.g., their classification)
and the consequence of misclassifying that
particular email (Table 3). For example, if a
participant responded with ‘important’ for a
phishing email in email severity class 4, the severity
score for this response was assigned a score of 4.
However, if this participant responded with ‘delete’
for a phishing email in email severity class 5, the
severity score for this response was assigned a score
of 0. A total severity score due to misclassification
was calculated as the sum of severity scores for
each email response and ranged from 0 (no
consequence) to 23 (severe consequence).
Table 3
The Severity Score based on Email Severity Class and
Participants’ responses
Results revealed an average severity score of
14.24. What’s more, only 2% of participants
correctly classified all emails indicating
approximately 98% would have experienced
adverse consequences resulting from email
misclassification.
DISCUSSION
While the topic of phishing and social
engineering is not new, the current focus on the
human side of the HCI equation promises to expand
our knowledge in this area. The preliminary results
of the current study illustrate a number of findings.
First, results suggest a disconnect between
participants’ self-reported data and the empirical
data collected from the Bob Jones email task.
Specifically, approximately 92% of participants
misclassified phishing emails even though 89%
indicated they were confident of their ability to
identify phishing emails. These results suggest a
majority of participants were not only susceptible to
phishing attacks, but overconfident in their ability to
protect themselves from such attacks. Second, only
2% of the participants suffered no adverse
consequences due to misclassification of emails
during the task. Third, individual differences such
as gender, dispositional trust, and personality appear
to be associated with the ability to correctly
categorize emails as either legitimate or phishing.
Limitations
While these results are interesting, they should
be interpreted with caution given several potential
methodological and analytical limitations. For
instance, reliance on self-report of prior behavior
may be subject to memory biases. Likewise, the
behavioral measure (Bob Jones email task) could be
described as artificial because participants were
asked to role play; however, this methodology has
been validated with prior research (Sheng et al.,
2010). Moreover, analysis of the consequences of
participants’ email misclassification severity was
based on a preliminary coding scheme developed by
an individual rater. Current efforts are underway to
provide inter-rater reliability for this measure and
additional measures used in the Bob Jones email
task. The sample recruited for the current study
consisted of college students. However, efforts are
currently underway to recruit a more diverse set of
participants (i.e., a non-student sample of working
professionals). Recently, we collected data from
volunteers employed at a government agency.
Future analyses will compare the students and non-
students to determine whether there are similarities
that are common to the two groups and more
importantly, how they vary in terms of phishing
susceptibility.
Future Research and Application
These results contribute to an ongoing effort to
develop a user profile that identifies those most at
risk of being phished. One implication might be the
ability to recommend a tailored anti-phishing
training tool to a user who is determined to be
vulnerable to phishing attack. Moreover, our efforts
to investigate individual differences in phishing
susceptibility are exemplified in a recent paper that
describes how people from different cultures
conceptualize phishing (Tembe, Hong, Murphy-
Hill, Mayhorn, & Kelley, 2013).
Further research will focus on refining this
profiling procedure and using it to inform the design
of a usable and effective tool to help users combat
phishing attacks. Our plan is to develop a training
tool that includes training contents reflecting the
results from this study in addition to conventional
training tools’ contents (e.g., disguised email
source, poor grammar, urgency cues, etc.).
Moreover, we will analyze how our anti-phishing
tool contributes to protecting users from the severe
consequences of phishing attacks compared to other
tools that are currently on the market.
ACKNOWLEDGEMENTS
This research was supported by a National
Security Agency Grant to the fourth and fifth
authors.
REFERENCES
Dhamija, R., & Tygar, J. D. (2005). The battle against
phishing: Dynamic security skins. Paper presented at the
ACM International Conference Proceeding Series.
Eveland, W. P., Shah, D. V., & Kwak, N. (2003). Assessing
causality in the cognitive mediation model: A panel study
of motivations, information processing, and learning
during campaign 2000. Communication Research, 30(4),
359-386. doi: 10.1177/0093650203253369
Gartner. (2007). Gartner survey shows phishing attacks
escalated in 2007; more than $3 billion lost to these
attacks. Retrieved from
http://www.gartner.com/newsroom/id/565125
Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003). A
very brief measure of the big-five personality domains.
Journal of Research in personality, 37(6), 504-528.
Hardee, J. B., West, R., & Mayhorn, C. B. (2006). To
download or not to download: An examination of
computer security decision making. interactions, 13(3),
32-37.
Kelley, C. M., Hong, K. W., Mayhorn, C. B., & Murphy-Hill,
E. (2012). Something smells phishy: Exploring
definitions, consequences, and reactions to phishing.
Proceedings of the Human Factors and Ergonomics
Society Annual Meeting, 56(1), 2108-2112. doi:
10.1177/1071181312561447
Merritt, S. M., & Ilgen, D. R. (2008). Not all trust is created
equal: Dispositional and history-based trust in human-
automation interactions. Human Factors: The Journal of
the Human Factors and Ergonomics Society, 50(2), 194-
210.
Neyste, P. G., & Mayhorn, C. B. (2009). Perceptions of
cybersecurity: An exploratory analysis. Proceedings of
the 17th world congress of the international ergonomics
association. Beijing, China.
Peters, M., Laeng, B., Latham, K., Jackson, M., Zaiyouna, R.,
& Richardson, C. (1995). A redrawn vandenberg and kuse
mental rotations test-different versions and factors that
affect performance. Brain and cognition, 28(1), 39-58.
Schwartz, M. J. (2011). Spear phishing attacks on the rise,
InformationWeek. Retrieved from
http://www.informationweek.com/security/attacks/spear-
phishing-attacks-on-the-rise/230500025
Sheng, S., Holbrook, M., Kumaraguru, P., Cranor, L. F., &
Downs, J. (2010). Who falls for phish?: A demographic
analysis of phishing susceptibility and effectiveness of
interventions. Proceedings of the 28th international
conference on Human factors in computing systems.
Atlanta, Georgia, USA
Shipley, W. C. (1986). Shipley institute of living scale. Los
Angeles, CA: Western Psychological Services.
Tembe, R., Hong, K. W., Murphy-Hill, E., Mayhorn, C. B., &
Kelley, C. M. (2013). American and Indian
Conceptualizations of Phishing. Proceedings of the 3rd
Workshop on Socio-Technical Aspects in Security and
Trust.
Temple, J. G., Warm, J. S., Dember, W. N., Jones, K. S.,
LaGrange, C. M., & Matthews, G. (2000). The effects of
signal salience and caffeine on performance, workload,
and stress in an abbreviated vigilance task. Human
Factors: The Journal of the Human Factors and
Ergonomics Society, 42(2), 183-194. doi:
10.1518/001872000779656480
Unsworth, N., Heitz, R. P., Schrock, J. C., & Engle, R. W.
(2005). An automated version of the operation span task.
Behavior Research Methods, 37(3), 498-505.
Vandenberg, S. G., & Kuse, A. R. (1978). Mental rotations, a
group test of three-dimensional spatial visualization.
Perceptual and motor skills, 47(2), 599-604. doi:
10.2466/pms.1978.47.2.599
Yoshioka, N., Washizaki, H., & Maruyama, K. (2008). A
survey on security patterns. Progress in Informatics, 5(5),
35-47.
