Content uploaded by Althea Gamble Gamble Blakey
Author content
All content in this area was uploaded by Althea Gamble Gamble Blakey on Feb 17, 2021
Content may be subject to copyright.
Full Terms & Conditions of access and use can be found at
https://www.tandfonline.com/action/journalInformation?journalCode=imte20
Medical Teacher
ISSN: 0142-159X (Print) 1466-187X (Online) Journal homepage: https://www.tandfonline.com/loi/imte20
Measuring exposure to bullying and harassment
in health professional students in a clinical
workplace environment: Evaluating the
psychometric properties of the clinical workplace
learning NAQ-R scale
Kelby Smith-Han, Emma Collins, Mustafa Asil, Althea Gamble Blakey, Lynley
Anderson, Elizabeth Berryman & Tim J. Wilkinson
To cite this article: Kelby Smith-Han, Emma Collins, Mustafa Asil, Althea Gamble Blakey, Lynley
Anderson, Elizabeth Berryman & Tim J. Wilkinson (2020) Measuring exposure to bullying and
harassment in health professional students in a clinical workplace environment: Evaluating the
psychometric properties of the clinical workplace learning NAQ-R scale, Medical Teacher, 42:7,
813-821, DOI: 10.1080/0142159X.2020.1746249
To link to this article: https://doi.org/10.1080/0142159X.2020.1746249
View supplementary material Published online: 14 Apr 2020.
Submit your article to this journal Article views: 394
View related articles View Crossmark data
Measuring exposure to bullying and harassment in health professional
students in a clinical workplace environment: Evaluating the psychometric
properties of the clinical workplace learning NAQ-R scale
Kelby Smith-Han
a
, Emma Collins
b
, Mustafa Asil
c
, Althea Gamble Blakey
d
, Lynley Anderson
e
,
Elizabeth Berryman
f
and Tim J. Wilkinson
g
a
Department of Anatomy and Otago Medical School, University of Otago, Dunedin, New Zealand;
b
School of Nursing, Otago
Polytechnic, Dunedin, New Zealand;
c
Educational Assessment Research Unit (EARU), College of Education, University of Otago, Dunedin,
New Zealand;
d
Otago Medical School, University of Otago, Dunedin, New Zealand;
e
Department of Bioethics Centre, University of
Otago, Dunedin, New Zealand;
f
Waitemata District Health Board, Auckland, New Zealand;
g
Dean’s Department and Department of
Medicine, Otago Medical School, University of Otago, Christchurch, New Zealand
ABSTRACT
Background: Instruments that measure exposure to bullying and harassment of students learning
in a clinical workplace environment (CWE) that contain validity evidence are scarce. The aim of this
study was to develop such a measure and provide some validity evidence for its use.
Method: We took an instrument for detecting bullying of employees in the workplace, called the
Negative Acts Questionnaire –Revised (NAQ-R). Items on the NAQ-R were adapted to align with
our context of health professional students learning in a CWE and added two new factors of sexual
and ethnic harassment. This new instrument, named the Clinical Workplace Learning NAQ-R, was
distributed to 540 medical and nursing undergraduate students and we undertook a Confirmatory
Factor Analysis (CFA) to investigate its construct validity and factorial structure.
Results: The results provided support for the construct validity and factorial structure of the new
scale comprising five factors: workplace learning-related bullying (WLRB), person-related bullying
(PRB), physically intimidating bullying (PIB), sexual harassment (SH), and ethnic harassment (EH).
The reliability estimates for all factors ranged from 0.79 to 0.94.
Conclusion: This study provides a tool to measure the exposure to bullying and harassment in
health professional students learning in a CWE.
KEYWORDS
Education environment;
undergraduate; work-based
Introduction
Bullying and harassment of students in health professional
education is a significant, ongoing and widespread prob-
lem, with evidence from medicine (Fnais et al. 2014), nurs-
ing (Clarke et al. 2012), dentistry (Rowland et al. 2010),
physiotherapy (Whiteside et al. 2014), and pharmacy
(Knapp et al. 2014). In addressing bullying and harassment
in a clinical workplace environment (CWE), it is likely to be
useful to examine the extent of the exposure –both to
establish the extent, and to also measure the impact of
any intervention. In health professional education, the
prevalence of bullying and harassment is highly variable
ranging from 6.3% (Wolf et al. 1998) to 87.4% (Owoaje
et al. 2012) in medical students; 45.1% (Ferns and
Meerabeau 2008) to 90% (Foster et al. 2004) in nursing stu-
dents; 34.6% in dentistry students (Rowland et al. 2010);
and 25% in physiotherapy students (Stubbs and Soundy
2013). Reasons for this variability are due to the different
definitions of what constitutes bullying and harassment,
along with different instruments used to measure the
phenomena (Einarsen et al. 2011).
A significant amount of learning in health professional
education occurs in the CWE, which may account for
Practice points
An instrument to measure the prevalence of bul-
lying and harassment of health professional stu-
dents in a clinical workplace environment (CWE)
that has validity evidence is scarce.
This study provides some validity evidence for an
instrument that measures the prevalence of bully-
ing and harassment of health professional stu-
dents learning in a CWE. The instrument contains
five factors of workplace learning-related bullying
(WLRB), person-related bullying (PRB), physically
intimidating bullying (PIB), sexual harassment
(SH), and ethnic harassment (EH).
The Clinical Workplace Learning NAQ-R scale is
quickly administered and provides health profes-
sional educators information into the negative
experiences students may be facing learning in a
CWE. Information may also assist educators in
developing specific interventions to target par-
ticular negative experiences faced by health pro-
fessional students learning in a CWE.
CONTACT Kelby Smith-Han kelby.smith-han@otago.ac.nz Department of Anatomy, School of Biomedical Sciences, University of Otago, PO Box 56,
Dunedin 9054, New Zealand
ß2020 Informa UK Limited, trading as Taylor & Francis Group
MEDICAL TEACHER
2020, VOL. 42, NO. 7, 813–821
https://doi.org/10.1080/0142159X.2020.1746249
students in the health professions reporting more bullying
and harassment than other higher education students
(Rautio et al. 2005). Health professional students are also
placed in a setting which is not just a learning environ-
ment, but also a working environment delivering health-
care to patients who are in need of care and treatment. It
is in this environment that learning about the profession
interacts with the provision of the health service, and is
also where abuse of health professional students occurs.
This is illustrated in a study by Rees et al. (2015) on clinical
workplace abuse narratives of students in a variety of
health professional education institutions, where many
examples of verbal and physical abuse along with sexual
and ethnic harassment were described.
When looking at addressing bullying and harassment of
health professional students, the leaders of health profes-
sional education institutions first need to ask the questions:
do we have bullying and harassment occurring at our insti-
tution when students are learning in CWE’s? If so, to what
extent? And lastly, what specific forms of bullying and har-
assment are occurring? Answering these questions is key to
informing the development of a response to bullying and
harassment of health professional students. Furthermore, as
effective interventions are developed, for example the
Creating a Positive Learning Environment (CAPLE) initiative
(Gamble Blakey et al. 2019a,2019b,2019c), we need reli-
able and valid measures to determine impact.
To answer these questions, reliability and validity evi-
dence is needed for an instrument used to measure the
exposure to bullying and harassment of health professional
students in a CWE. To our knowledge, no studies have
been published explicitly demonstrating an instrument’s
validity and psychometric properties measuring the preva-
lence and type of bullying and harassment (that includes
sexual and ethnic harassment), in health professional edu-
cation specifically within a CWE. In order to address this
issue, we took the following approach to develop such
an instrument.
The NAQ-R
An instrument already developed to investigate bullying in
the workplace is the Negative Acts Questionnaire –Revised
(NAQ-R). The questionnaire has been previously researched
to provide validity and reliability evidence for its use and is
widely used in measuring exposure to workplace bullying
of employees (Einarsen et al. 2009). The NAQ-R contains 22
questions with three factors: work-related bullying, person-
related bullying, and physically intimidating bullying.
Previous literature has illustrated that sexual and ethnic
harassment are significant factors associated with the bully-
ing and harassment of health professional students
(Gaughran et al. 1997; Richardson et al. 1997; White 2000;
Rautio et al. 2005; Witte et al. 2006; Wilkinson et al. 2006;
Garbin et al. 2010; Premadasa et al. 2011; Rees and
Monrouxe 2012; Bruce et al. 2015; Rees et al. 2015), how-
ever, the NAQ-R does not include behaviours associated
with these factors. Therefore, it was determined appropri-
ate to include two additional factors of sexual and ethnic
harassment, to provide a more comprehensive account of
bullying and harassment.
Purpose of the study
We have called the modified questionnaire the Clinical
Workplace Learning NAQ-R scale. Specifically, this study
attempts to answer the research question of:
To what extent are the factors present in the modified version
of the questionnaire, applicable to a CWE; and what is the
effect of adding two new factors on the validity of
the instrument? To this end, the psychometric properties of the
Clinical Workplace Learning NAQ-R scale were evaluated.
Method
Participants and procedures
The Clinical Workplace Learning NAQ-R questionnaire was
distributed to all undergraduate medical students who
were in their clinical years (years 4–6) at the University of
Otago’s six-year medical degree, and all undergraduate
nursing students in their final two years of the Otago
Polytechnic’s Bachelor of Nursing degree. Years 4–6 of the
medical curriculum entails the immersion of medical stu-
dents in different CWEs for their learning, predominately
hospital and general practice learning environments. Year 2
of the nursing degree is when nursing students begin to
learn in CWEs, with final year students (year 3) spending
the largest amount of time learning in CWEs. The CWEs for
nursing students consist of Primary Healthcare settings,
hospitals and residential care facilities, with the predomin-
ant amount of clinical learning conducted in the hospital
environment.
For the medical students, hardcopy questionnaires for
year 4 and 5 were administered during whole class ses-
sions. Year 6 students are more geographically dispersed
so on-line questionnaires were used. For the nursing stu-
dents hardcopy questionnaires were distributed during a
whole class teaching session. Ethical approval of the study
was obtained from the University of Otago Human Ethics
Committee and by the Otago Polytechnic
Ethics Committee.
Theoretical underpinnings of the original NAQ-R
In the original NAQ-R bullying is defined as ‘a situation in
which hostile and aggressive actions are systematically
directed at one or more persons in such a way that they are
stigmatized and victimized’(Mikkelsen and Einarsen 2001,p.
394). Additionally, describing the dimensions of what con-
stitutes bullying in the workplace in which the NAQ-R is sit-
uated is necessary to understand its construction. Bullying
in the workplace constitutes negative and unwanted
behaviours (Einarsen et al. 2011) along with ‘…evolving
and often escalating hostile workplace relationships rather
than discrete and disconnected events and is associated
with repetition (frequency), duration (over a period of
time), and patterning (of a variety of behaviours involved)
as its most salient features’(Einarsen et al. 2009, p. 25).
The NAQ-R contains three factors: work-related bullying,
person-related bullying, and physically intimidating bully-
ing. Work-related bullying consists of behaviours targeted
at an individual’s working role and activities such as being
given unreasonable deadlines, or meaningless tasks
(Einarsen et al. 2011). Person-related bullying consists of
814 K. SMITH-HAN ET AL.
behaviours that are targeted at the individual themselves
for example, spreading gossip or rumours about you, hav-
ing insulting or offensive remarks made about your person
(Einarsen et al. 2011). As stated by Einarsen et al. (2011),
the behaviours associated with person-related bullying
‘…are by and large, independent of the work organisation’
(p. 13). Physically intimidating bullying is associated with
behaviours targeting the individual with explicit acts of
physical aggression or violence or threats of violence
(Einarsen et al. 2011).
Instrument development
Demographic data collected in the survey included age,
ethnicity, sex and sexual orientation. Sexual orientation was
included purposively as previous literature indicates that
students who are in a minority regarding sexual orientation
are more likely to be bullied and/or harassed (Przedworski
et al. 2015).
To develop the instrument, we undertook the following
processes. The original concept underpinning the NAQ-R
(bullying and harassment of employees in a workplace)
was modified in order to fit our context (bullying and har-
assment of students learning in a CWE). Items used in the
original NAQ-R were then edited to align with our new
context. Finally, we added two new factors of sexual and
ethnic harassment. Both processes were conducted while
maintaining the original instrument’s behavioural design.
Modifying the concept of bullying and harassment in
the workplace
The NAQ-R consists of a three factor model of workplace
related bullying, person-related bullying, and physically
intimidating bullying. The NAQ-R was originally designed in
the context of workplace bullying and harassment of
employees and therefore, the items in the questionnaire
related to the definition of bullying and harassment in the
context of working as an employee. In order to make the
NAQ-R effective for measuring exposure of bullying and
harassment of students learning in a CWE we undertook the
following conceptual modifications.
Firstly, we modified the existing definition of bullying
used for the NAQ-R of ‘a situation in which hostile and
aggressive actions are systematically directed at one or more
persons in such a way that they are stigmatized and victi-
mized’(Mikkelsen and Einarsen 2001, p. 394) to fit our con-
text ‘a situation in which hostile and aggressive actions are
systematically directed at one or more students in such a
way that they are stigmatized and victimized in a clinical
workplace learning environment.’
Then, we edited specific items in the original question-
naire so they would align with this new modified concept
of bullying and harassment, from employees working, to stu-
dents learning in a CWE. Two statements that did not fit
the context of students learning in a clinical environment
were removed. One statement from the work-related bully-
ing factor: ‘Pressure not to claim something to which by
right you are entitled (e.g. sick leave, holiday entitlement,
travel expenses),’as it relates to the role of being
employed which is not relevant to students. The second
statement that was omitted was from the person-related
bullying factor: ‘Practical jokes carried out by people you
don’t get along with.’As health professional students gen-
erally move around different clinical workplaces to experi-
ence different areas of healthcare practice, we concluded
that the behaviour indicated in the statement may occur
less frequently. We also re-analysed the remaining 20 state-
ments in the original NAQ-R associated with its three fac-
tors, and re-worded 10 statements to align with our
modified concept. The changes to the wording of particular
items can be seen in Table 1:
Table 1. Re-worded statements from the NAQ-R used in the clinical workplace learning NAQ-R scale.
Original statement in NAQ-R Wording change/omissionRationale for change
Work-related bullying factor
Someone withholding information which
affects your performance
Someone withholding information which affects
your learning
To fit our context of students learning in a
workplace environment
Being ordered to do work below your level
of competence
Being ordered to do tasks above your level
of competence
This reflects the importance of students learning
tasks to be within their level of competence in
order to keep themselves safe, and their
patients safe during the learning process.
Having your opinions ignored Having your opinions and views ignored Aides understanding of the statement to broaden
it out to more than just having your
‘opiniated comments’ignored, but whatever
your contribution happens to be at the
time ignored.
Being given tasks with unreasonable deadlines Being given tasks with unreasonable or
impossible targets or deadlines
Aides understanding of the statement.
Person-related bullying factor
Being humiliated or ridiculed in connection
with your work
Being humiliated or ridiculed in connection with
your learning
To fit our context of students learning in a
workplace environment
Having key areas of responsibility removed or
replaced with more trivial or
unpleasant tasks
Having key areas of your student role removed or
replaced with more trivial or unpleasant tasks
Highlights significant change in what the student
should be doing at the level they are currently
at, which changes as they progress.
Being ignored or excluded Being ignored or excluded from the clinical team To fit our context of students learning in a
workplace environment
Having insulting or offensive remarks made
about your person, attitudes or your
private life
Having insulting or offensive remarks made
about your person (i.e. habits and background),
attitudes or your private life
Aides understanding of the statement.
Hints or signals from others that you should
quit your job
Hints or signals from others that you should quit
studying your profession
To fit our context of students learning in a
workplace environment
Persistent criticism of your errors or mistakes Persistent criticism of your work and effort To make sure students didn’t get this confused
with patient safety literature (where errors etc.
have specific definitions).
Change/addition of wording shown in italics.
MEDICAL TEACHER 815
Two statements obtained from previous literature, were
also added to more accurately and thoroughly reflect the
context of experiencing bullying and harassment by stu-
dents learning in a clinical environment. Items added were:
Being assigned work for punishment rather than for educa-
tional value; and having learning opportunities blocked or
withheld by others. These statements were added to the
workplace related bullying factor and was re-named the
workplace learning-related bullying factor to fit our new
modified concept, (see Supplement Appendix, Table A1, for
all Clinical Workplace Learning NAQ-R Scale Items).
Adding two new factors
Two new factors were introduced in our modified version
of the questionnaire: sexual harassment and ethnic harass-
ment (Supplement Appendix, Table A1).
To include sexual harassment in our CWE context we
adopted Till’s definition of ‘academic sexual harassment’as
the ‘use of authority to emphasise the sexuality or sexual
identity of a student in a manner which prevents or impairs
that student’s full enjoyment of educational benefits, climate,
or opportunities’(Till 1980, p. 7). The items used for this fac-
tor that aligned with this definition drew on the general
sexual harassment literature such as the Sexual Experiences
Questionnaire (SEQ) (Fitzgerald et al. 1995); along with
questionnaires used in the bullying and harassment litera-
ture focused on professional working environments
(Crebbin et al. 2015); and health professional students
(Sheehan et al. 1990; Baldwin et al. 1991; Uhari et al. 1994;
Wilkinson et al. 2006; Woolley et al. 2006; Rowland et al.
2010; Clarke et al. 2012; Knapp et al. 2014; Whiteside et al.
2014; AAMC 2018).
We adopted the conceptualisation of ethnic harassment
as described by Schneider et al, defined as ‘threatening ver-
bal conduct or exclusionary behaviour that has an ethnic
component and is directed at a target because of his or her
ethnicity’(Schneider et al. 2000, p. 3). The items that were
used for this factor aligning with this definition were
derived from a combination of the Ethnic Harassment
Experiences scale (EHE) (Schneider et al. 2000); and the bul-
lying and harassment literature in the workplace (Keashly
1997; Einarsen et al. 2011; Crebbin et al. 2015) and of
health professional students (Sheehan et al. 1990; Baldwin
et al. 1991; Uhari et al. 1994; Wilkinson et al. 2006; Woolley
et al. 2006; Rowland et al. 2010; Clarke et al. 2012; Knapp
et al. 2014; Whiteside et al. 2014; AAMC 2018).
Maintaining instrument design
Two main aspects in the design of the original NAQ-R
ensures all items are written using specific behavioural
statements, and there are no definitions given about or
words mentioning ‘bullying’or ‘harassment’when partici-
pants undertake the questionnaire (Einarsen et al. 2009).
For example, the questionnaire asks participants how often
they have been subjected to the following negative acts
and gives a list of the terms such as ‘Having insulting or
offensive remarks made about your person (i.e. habits and
background), your attitudes or your private life,’as
opposed to asking how the participant feels about the
behaviour. Asking about specific behaviours without pro-
viding terms or definitions assists in minimising
misinterpretations by participants, specifically when devel-
oping an instrument investigating incidence and/or preva-
lence (Arvey and Cavanaugh 1995), such as is the focus of
this research. Taking this behavioural approach to the
design of the instrument ‘is considered to provide a more
objective estimate of exposure to bullying behaviours than
self-labelling approaches, as respondents’need for cognitive
and emotional processing of information would be reduced’
(Einarsen et al. 2009, p. 27). Therefore, when modifying the
original NAQ-R, we kept the same approach of using
behavioural terminology and not mentioning the words
‘bullying’or ‘harassment.’
The frequency rating of behaviours in the original NAQ-
R was kept, because we determined that it would still fit
our context of students learning in a CWE. For this meas-
urement, participants are asked to rate how often they
experienced the behaviours listed on a 1–5 point Likert
scale where 1 ¼Never,2¼Now and then,3¼Monthly,
4¼Weekly,5¼Daily.
While in the NAQ-R, participants are asked to rate their
experiences over the past 6 months, we modified this to
8 months for the current questionnaire, as this gives the
students the opportunity to reflect on their experiences for
the majority of their clinical year.
In summary, in light of these modifications the newly
formed 31-item Clinical Workplace Learning NAQ-R scale
has been developed to measure health professional stu-
dents’exposure to negative inter-personal interactions in a
CWE. It is comprised of five hypothesised factors: workplace
learning-related bullying (WLRB), person-related bullying
(PRB), physically intimidating bullying (PIB), sexual harass-
ment (SH), and ethnic harassment (EH) (the Clinical
Workplace Learning NAQ-R questionnaire is located in
Supplement Appendix A2).
Statistical analyses
We developed and examined the psychometric characteris-
tics of the Clinical Workplace Learning NAQ-R scale using a
structural equation modelling approach to explore the val-
idity evidence. A Structural Equation Modelling (SEM)
approach, specifically Confirmatory Factor Analysis (CFA),
was carried out using MPlus 7 (Muth
en and Muth
en 2012).
We conducted the data analyses in two stages. Initially, we
examined the data for outliers and missing cases. Then, the
factorial structure of the Clinical Workplace Learning NAQ-R
scale was examined by means of Confirmatory Factor
Analysis (CFA). To provide support for the validity evidence
for the hypothesised factor structure of the Clinical
Workplace Learning NAQ-R scale, we investigated and com-
pared the goodness-of-fit of different competing models as
suggested by Noar (2003) and Strauss and Smith (2009).
Confirmatory factor analysis (CFA)
The hypothesized model (five-factor) model was compared
to three other alternative competing models. The hypothe-
sized model included three factors reported by Einarsen
et al. (2009) as well as two new factors that are sexual har-
assment (SH) and ethnic harassment (EH) all of which are
related to each other. The competing models included: (a)
a one-factor (unidimensional) model that assumed all
816 K. SMITH-HAN ET AL.
manifest variables loaded on a single factor, (b) a three-fac-
tor model that suggests WLRB, PRB, and PIB items loaded
on a single factor, and (c) a second-order (higher-order)
factor model with the five scale factors subordinating to a
single second-order factor.
One-factor model means that what we are measuring is
a unidimensional construct and university students are not
differentiating the bullying and harassment factors.
Evidence for a three-factor model indicates that all three
factors reported by Einarsen et al. (2009) are not distinct
from each other. Support for second-order model would
suggest that these related five factors can be accounted
for by an underlying higher order construct. Support for
the hypothesized five-factor correlated (oblique) model
would suggest that medical and nursing students differen-
tiate between the five bullying and harassment factors that
are related to each other.
Since the data were ordered-categorical, the weighted
least squares mean and variance adjusted (WLSMV) estima-
tion procedure was used for CFA analyses. The WLSMV is a
robust estimation technique that is suggested for model-
ling ordinal data (Flora and Curran 2004; Brown 2006). The
consequences of treating ordinal responses as continuous
which may lead to reporting inaccurate results are well-
established in the literature (Lubke and Muth
en 2004).
A number of different indices were examined to com-
pare the different models and evaluate model-data fit
(Cheung and Rensvold 2002; Fan and Sivo 2005,2007).
Each of these measures reflects a different aspect of model
fit and may not perform equally well under different types
of model conditions (Fan and Sivo 2007). Thus, it is import-
ant to use multiple indices rather than relying on one
measure (Hair et al. 2010). Indices reported in this study
included the Root Mean Square Error of Approximation
(RMSEA), the Comparative Fit Index (CFI), the Tucker-Lewis
index (TLI), and the Weighted Root Mean Square Residual
(WRMR). The chi-square (v
2
) values were also reported but
not used for model fit decisions as this statistic and its sig-
nificance are inflated with large sample sizes. The com-
monly accepted cut-offs for ‘acceptable’or ‘good’fit
(Browne and Cudeck 1992; MacCallum et al. 1996; Hu and
Bentler 1998;Yu2002; Hair et al. 2010) included: a non-sig-
nificant chi-square (v
2
), RMSEA with values <0.08 indicating
an acceptable fit and values <0.05 indicating a good fit,
CFI and TLI with values >0.90 being indicative of reason-
able fit and values >0.95 indicating a good fit, and WRMR
with values being close to 1. The limitations of coefficient
alpha (a) as a measure of reliability estimate is well docu-
mented in the literature (Sijtsma 2009; Teo and Fan 2013).
Therefore, using polychoric correlations, we calculated and
reported McDonald’s omega (x) (McDonald 1999) as a bet-
ter estimate of reliability.
Results
Participant and demographic information
A total of 428 from an eligible 852 medical students com-
pleted the questionnaire giving a response rate of 50%
(428/852). A total of 69 nursing students in year 2 and
43 year 3 completed the questionnaire, from an eligible
212 nursing students, a giving a response rate of 53%
(112/212). The questionnaire took approximately 5 minutes
to complete.
Therefore, 540 medical and nursing students completed
the questionnaire giving an overall response rate of 51%
(540/1064). Of the participants, 65.2% (n¼352) were
females and 34.4% (n¼186) were males. Only one student
did not report their sex and one student identified them-
selves as transgender. Mean age was 23.7 years (range
19–53 years, SD ¼4.35). Self-reported ethnic composition
was reported as; New Zealand European (67.8%), M
aori
(9.4%), Chinese (8.3%), Indian (3.0%), Samoan (1.3%), Cook
Island M
aori (0.7%), Tongan (0.4%), other ethnicities (20%),
and not stated (0.7%). Because individuals can be of more
than one ethnicity, these totals are greater than 100%.
Sexual orientation was reported as heterosexual (91.7%),
homosexual (3.3%), bisexual (2.4%), other (1.5%), and not
stated (1.1%).
Descriptive statistics
No univariate outliers were identified to have an effect on
the results. The proportion of missing cases for each item
was trivial ranging from mostly zero to one percent. The
Expectation Maximization (EM) algorithm which assumes
that observations are missing at random (MAR) was utilized
to impute the missing cases rather than listwise deletion.
The means and standard deviations for the five factors of
the Clinical Workplace Learning NAQ-R scale are summar-
ized in Table 2.
Factor means ranged from 1.15 to 1.44, suggesting that
most students endorsed ‘never’or ‘now and then’with the
statements. However, examination of the item means most
of which ranged from 1.00 to 5.00 revealed that there were
some students who had been subjected to negative acts
on a daily basis. The standard deviations ranged from 0.28
to 0.44 indicating that the dispersion of responses for each
factor were somewhat similar.
Confirmatory factor analysis (CFA)
The goodness-of-fit measures of hypothesized and alterna-
tive models are summarized in Table 3.
Evaluation of the unidimensional model revealed that
this model was not representing the sample data
Table 2. Descriptive statistics of clinical workplace learning NAQ-R scale factors.
Number of items M SD Min Max
Workplace learning-related bullying (WLRB) 8 1.44 0.44 1.00 3.75
Person-related bullying (PRB) 11 1.43 0.43 1.00 4.09
Physically intimidating bullying (PIB) 3 1.18 0.35 1.00 3.67
Sexual harassment (SH) 5 1.15 0.28 1.00 3.20
Ethnic harassment (EH) 4 1.15 0.38 1.00 4.00
MEDICAL TEACHER 817
sufficiently. The RMSEA, CFI, TLI and WRMR values did not
meet the commonly acceptable fit criteria. The chi-square
statistic and fit indices (RMSEA ¼0.045; CFI ¼0.955;
TLI ¼0.955; and WRMR ¼1.232) suggested that the
hypothesized five-factor correlated model provided the
best model fit with the data. The three-factor and second-
order models had also good fit close to the hypothesized
model. However, the chi-square difference test (DIFFTEST)
between the second-order and the hypothesized model
indicated that adding a higher-order dimension signifi-
cantly worsened the fit. Thus, the hypothesized model was
retained as the model of best fit with the five factors of
workplace learning-related bullying (WLRB), person-related
bullying (PRB), physically intimidating bullying (PIB), sexual
harassment (SH), and ethnic harassment (EH).
For this model, factor correlations and reliabilities are
provided in Table 4.
The correlations between the Clinical Workplace
Learning NAQ-R factors ranged from 0.48 to 0.95. The high-
est correlations were found between workplace learning-
related bullying (WRB), person-related bullying (PRB), and
physically intimidating bullying (PIB). These results were
closely consistent with findings of the Einarsen et al. (2009)
study. However, we observed moderate correlations
between the new factors sexual harassment (SH), ethnic
harassment (EH) - and other factors indicating that the evi-
dence of discriminant validity improved with the addition
of SH and EH factors.
All of the omega reliability estimates were greater than
the recommended level (0.70). Standardized factor loadings
for the hypothesized model are provided in Table 5.
All the unstandardized factor loadings were significant.
Standardized factor loadings for the hypothesized model
ranged from 0.63 to 0.96 providing support for convergent
validity. All of the items were strong indicators of the fac-
tors they were related to.
In summary, the CFA analyses and reliability estimates
provided support for the validity evidence of the factorial
structure of the Clinical Workplace Learning NAQ-R.
Discussion
The aim of this research was to construct and provide val-
idity evidence for a self-report instrument to measure
health professional students’experience of bullying and
harassment in a clinical workplace learning environment.
Previous research looking into the prevalence and type of
bullying and harassment that health professional students
face were from a variety of instruments lacking validity evi-
dence to support the intended purpose.
This study has the potential to contribute to the litera-
ture on measuring exposure to bullying and harassment of
health professional students by providing: empirical evi-
dence through supporting the validation of a new instru-
ment; an instrument that is specific to a clinical workplace
learning environment; and an instrument that contains
more relevant factors associated with literature on bullying
and harassment of health professional students.
Using an instrument (NAQ-R) that was developed for a
different context (workplace bullying among employees),
and with evidence supporting its validity, we modified it to
Table 3. Confirmatory factor analysis of alternative models.
Model v
2
df RMSEA CFI TLI WRMR
Alternative one-factor (unidimensional) 1995.244434 0.082 0.845 0.834 2.136
Alternative three-factor 906.221431 0.045 0.953 0.949 1.266
Hypothesized five-factor 878.706424 0.045 0.955 0.951 1.232
Alternative second-order 906.289429 0.045 0.953 0.949 1.280
Note. RMSEA: Root Mean Square Error of Approximation; CFI: Comparative Fit Index; TLI: Tucker-Lewis Index; WRMR: Weighted Root Mean Square Residual.
p<.01
Table 4. Clinical workplace learning NAQ-R scale factor correlations and reliabilities.
WLRB PRB PIB SH EH
Workplace Learning-Related Bullying (WLRB) –
Person-Related Bullying (PRB) 0.95 –
Physically Intimidating Bullying (PIB) 0.80 0.89 –
Sexual Harassment (SH) 0.59 0.54 0.52 –
Ethnic Harassment (EH) 0.48 0.58 0.49 0.48 –
Reliability (x) 0.89 0.93 0.79 0.90 0.94
Table 5. Standardized factor loadings of the Clinical Workplace Learning
NAQ-R.
Item Standardized estimate
WLRB 1 0.65
WLRB 2 0.63
WLRB 3 0.72
WLRB 4 0.75
WLRB 5 0.69
WLRB 6 0.71
WLRB 7 0.80
WLRB 8 0.70
PRB 1 0.75
PRB 2 0.66
PRB 3 0.71
PRB 4 0.69
PRB 5 0.73
PRB 6 0.73
PRB 7 0.83
PRB 8 0.65
PRB 9 0.82
PRB 10 0.83
PRB 11 0.80
PIB 1 0.79
PIB 2 0.76
PIB 3 0.76
SH 1 0.91
SH 2 0.92
SH 3 0.65
SH 4 0.66
SH 5 0.93
EH 1 0.93
EH 2 0.96
EH 3 0.91
EH 4 0.91
818 K. SMITH-HAN ET AL.
investigate if it would fit in our context of (clinical work-
place learning environments for students, not employees).
The original NAQ-R contained three factors related to bully-
ing in the workplace: workplace learning-related bullying,
person-related bullying and physical related bullying. We
modified these three factors and also added two new fac-
tors not in the original NAQ-R related to sexual harassment
and ethnic harassment, because of the reported prevalence
of these types of experiences by students learning in a
CWE. Our analyses provide evidence for the validity of
these two new factors.
We suggest naming this instrument the Clinical
Workplace Learning NAQ-R scale, as this acknowledges the
significant body of work by Einarsen et al. (2009) in devel-
oping and analysing the psychometric properties of the ori-
ginal NAQ-R designed for workplace bullying of employees.
The results indicated that the original three factors of
the NAQ-R that we modified to fit a clinical workplace
learning (workplace learning-related bullying, person-
related bullying and physical related bullying) had relatively
high factor correlations. This indicates the modifications we
made to the original NAQ-R (Table 1) to reflect the new
context of measuring exposure to clinical workplace learn-
ing bullying, did not change the strength of the associa-
tions among factors we borrowed from the original NAQ-R.
The magnitude of the factor loadings indicated that all
items were strong indicators of the factors they were
related to. The factor loadings of the items of two new fac-
tors of sexual harassment and ethnic harassment were
even higher than the original NAQ-R items which provided
further convergent validity evidence. The correlations
between the two novel factors and the original ones were
moderate which revealed discriminant validity evidence.
Both convergent and discriminant validity are important
components of construct validity. The CFA analysis sug-
gests that adding these two new factors support that med-
ical and nursing students differentiate between the five
factors that are also inter-related to each other for the
overall construct of clinical workplace learning bullying
and harassment.
Implications
Having a comprehensive five factor model that includes
sexual and ethnic harassment among the bullying behav-
iours experienced provides a more comprehensive instru-
ment that aligns with the definition of bullying offered
earlier, and more accurately reflects the varied negative
experiences of health professional students learning in the
clinical workplace described in the literature (Dineen et al.
1997; Richardson et al. 1997; White 2000; Rautio et al.
2005; Witte et al. 2006; Wilkinson et al. 2006; Garbin et al.
2010; Premadasa et al. 2011; Rees and Monrouxe 2012;
Bruce et al. 2015; Przedworski et al. 2015; Rees et al. 2015).
During the development of modifying the original NAQ-
R and adding the two factors of sexual harassment and
ethnic/racial harassment we were mindful of trying to keep
the instrument short. The final questionnaire is a 31 item
instrument, which is only nine items longer the original
NAQ-R questionnaire. The length of a questionnaire is
important to consider, as practically implementing long
questionnaire in large organisations, or student groups can
be difficult to administer, and run the risk of larger attrition
of participant responses. When implemented, the 31 item
questionnaire takes participants approximately 5 minutes
to complete.
The Clinical Workplace Learning NAQ-R scale, as a five
factor model, would be useful for health professional edu-
cation institutions who would like to measure their stu-
dents’exposure to bullying and harassment in clinical
workplace learning environments. Developing a scale spe-
cifically for student learners in the workplace and extend-
ing the original NAQ-R to a five factor model could assist
institutions in identifying particular problematic areas (if
any) that their students maybe experiencing. For example,
are students experiencing bullying behaviours that reflect
the workplace learning aspects of their student learning
role (e.g. being asked to do something above their level of
competence); or are they experiencing more person-related
bullying (e.g. Being ignored or excluded from the clinical
team); or experiencing sexual harassment (e.g. inappropri-
ate physical contact); or ethnic harassment (e.g. made
derogatory comments about your racial or ethnic group).
Identifying these specific areas of bullying and harassment
would significantly benefit institutions in planning any
interventions to address the negative behaviours experi-
enced by health professional students.
Limitations
The questionnaire was delivered to only two health profes-
sional groups (medical students and nursing students) yet
these two groups would represent the largest health pro-
fessional groups in New Zealand. Although these two
groups occupy a variety of settings that include and repre-
sent various primary and secondary and community clinical
environments, we acknowledge they may not exactly mir-
ror all health professional CWEs. Additionally, using these
groups in our study could also be viewed as a strength,
given many validation studies only include more homoge-
neous populations.
Moreover, even though we view the statements used in
the Clinical Workplace Learning NAQ-R scale are generic
enough to apply to many clinical workplace learning set-
tings, further testing to look at how the instrument works
with other health professional student groups to confirm
this would be worthwhile.
We also understand that using this instrument design of
measuring only behaviours and their frequency does not
provide answers to other specific questions institutions
may be wanting. For example, this method does not exam-
ine what Einarsen et al. (2009, p. 40) describes as ‘who did
what to whom.’However, this issue could be addressed by
adding a self-labelling method. For example, after adminis-
tering the Clinical Workplace Learning NAQ-R scale, a defin-
ition of bullying and harassment is offered to participants
and then asked if they view themselves as victims accord-
ing to this definition and to describe what happened to
them and by who (Einarsen et al. 2009).
Although the five-factor model yielded the best fit, hav-
ing acceptable fit for three-factor and second-order models
suggests that there is still room for improvement on the
psychometric properties of the NAQ-R scale. We agree with
Einarsen et al. (2009) that even though the original three
MEDICAL TEACHER 819
dimensions of reported workplace bullying can be distin-
guished, ‘yet they do not discriminate well between differ-
ent types of bullying behaviours, suggesting co-occurrence
of these different types of bullying (p. 31).’Also, results
from the second-order model may support that idea that
NAQ-R constructs are correlated reflecting the presence of
a more general construct at a higher conceptual level and
can be considered together to create a composite score. In
this research, we wanted to maintain the integrity and
structure of the original NAQ-R as much as possible but
future research may consider improving the constructs by
refining the item wording or shortening the scale especially
for the WLRB and PRB factors.
The Clinical Workplace Learning NAQ-R was developed
based in a New Zealand cultural context which shares
some similarities with Einarsen’s et al. (2009) Anglo-
American context, yet also maintains its own cultural con-
text. The literature reports many behaviours that are similar
between many cultural contexts in relation to bullying and
harassment behaviours at medical and nursing schools.
However, it would be pertinent to assume that there would
be different beliefs, values and practices specific to certain
cultures that may inform the concepts of bullying and har-
assment. This would influence the wording of the state-
ments used in the instrument along with the meaning that
is attributed to the statements as well. Therefore, further
work needs to be conducted in making the instrument
applicable in different cultural contexts. Finally, future val-
idity research could explore how sensitive the scores are to
change over time, for example following an intervention.
Conclusion
In this paper we have shown the development of the
Clinical Workplace Learning NAQ-R scale and analysis of its
factor structure and provided supporting validity evidence
for its use. The Clinical Workplace Learning NAQ-R scale is
a quickly administered instrument in order to measure
exposure to bullying and harassment experienced by
health professional students in a clinical workplace learning
environment. Its structure may assist health professional
leadership to obtain vital information into the negative
experiences students may be facing, including what spe-
cific experiences may be occurring more frequently than
others in regards to the bullying and harassment of
their students. In turn, this information may assist in devel-
oping specific interventions to target the particular experi-
ences faced by health professional students learning in
a CWE.
Acknowledgments
The authors would like to thank all of the Associate Deans of Medical
Education for Otago Medical School who assisted with the data collec-
tion for this questionnaire, as well as Faculty in The School of Nursing
at Otago Polytechnic. We would also like to thank Dr Ella Iosua and
Michel de Lange for their initial advice on the analysis of this paper.
We would also like to thank the students for their time and effort in
completing this questionnaire.
Disclosure statement
The authors report no conflicts of interest. The authors alone are
responsible for the content and writing of the article.
Glossary
Confirmatory factor analysis (CFA): Is a type of structural
equation modeling (SEM) that deals specifically with measure-
ment models that is, the relationships between observed meas-
ures or indicators (e.g., test items, test scores, behavioral
observation ratings) and latent variables or factors”(Brown &
Little, 2015, p.1).
Funding
The present research was financially supported by Division of Health
Sciences, University of Otago.
Notes on contributors
Kelby Smith-Han, MHealSc, PhD, is a Postdoctoral and Teaching
Fellow, Department of Anatomy and Otago Medical School, University
of Otago, Dunedin, New Zealand.
Emma Collins, MN, is Principal Lecturer, School of Nursing, Otago
Polytechnic, Dunedin, New Zealand.
Mustafa Asil, MA, PhD, is a Senior Research Fellow, Educational
Assessment Research Unit (EARU), College of Education, University of
Otago, Dunedin, New Zealand.
Althea Gamble Blakey, MHealSc, PhD, is a Research Fellow, Otago
Medical School, University of Otago, Dunedin, New Zealand.
Lynley Anderson, MHealSc, PhD, is an Associate Professor, Bioethics
Centre, University of Otago, Dunedin, New Zealand.
Elizabeth Berryman, MHSc, MBChB, is a Resident Medical Officer,
Waitemata District Health Board, Auckland, New Zealand.
Tim J. Wilkinson, MBChB, MClinEd, PhD, MD, is Professor, Dean’s
Department and Department of Medicine, Otago Medical School,
University of Otago, Christchurch, New Zealand.
ORCID
Kelby Smith-Han http://orcid.org/0000-0003-2105-5062
Mustafa Asil http://orcid.org/0000-0002-7827-6686
Althea Gamble Blakey http://orcid.org/0000-0002-8373-5816
Lynley Anderson http://orcid.org/0000-0002-7329-7058
Tim J. Wilkinson http://orcid.org/0000-0002-4080-4164
References
Arvey RD, Cavanaugh MA. 1995. Using surveys to assess the preva-
lence of sexual harassment: some methodological problems. J Soc
Issues. 51(1):39–52.
Association of American Medical Colleges (AAMC). 2018. Medical
School Graduation Questionnaire; [accessed 2018 Nov 23]. https://
www.aamc.org/download/490454/data/2018gqallschoolssummaryre-
port.pdf.
Baldwin DC, Jr, Daugherty SR, Eckenfel EJ. 1991. Student perceptions
of mistreatment and harassment during medical school: a survey of
ten United States schools. West J Med. 155(2):140–145.
Brown T. 2006. Confirmatory factor analysis for applied research. New
York: Guildford press.
Browne MW, Cudeck R. 1992. Alternative ways of assessing model fit.
Sociol Methods Res. 21(2):230–258.
Brown, T. and Little, T. 2015. Confirmatory factor analysis for applied
research. 2nd ed. New York: Guilford Press.
820 K. SMITH-HAN ET AL.
Bruce AN, Battista A, Plankey MW, Johnson LB, Marshall MB. 2015.
Perceptions of gender-based discrimination during surgical training
and practice. Med Educ Online. 20:25923.
Cheung GW, Rensvold RB. 2002. Evaluating goodness-of-fit indexes for
testing measurement invariance. Struct Equ Modeling. 9(2):233–255.
Clarke C, Kane D, Rajacich D, Lafreniere K. 2012. Bullying in under-
graduate clinical nursing education. J Nurs Educ. 51(5):269–276.
Crebbin W, Campbell G, Hillis DA, Watters DA. 2015. Prevalence of bul-
lying, discrimination and sexual harassment in surgery in
Australasia. ANZ J Surg. 85(12):905–909.
Dineen M, Cole M, Dally RJ, Dinned S, Gaughran F. 1997. Stress in
medical students. Ir Med J. 90(5):184–185.
Gaughran F, Dineen S, Dineen M, Cole M, Daly RJ. 1997. Stress in med-
ical students. Ir Med J. 90(5):184–185.
Einarsen S, Hoel H, Notelaers G. 2009. Measuring exposure to bullying
and harassment at work: validity, factor structure and psychometric
properties of the Negative Acts Questionnaire-Revised. Work Stress.
23(1):24–44.
Einarsen S, Hoel H, Zapf D, Cooper C, ed. 2011. Bullying and harass-
ment in the workplace: developments in theory, research, and prac-
tice (2nd ed.). Boca Raton: CRC Press Taylor & Francis Group.
Fan X, Sivo SA. 2005. Sensitivity of fit indexes to misspecified structural
or measurement model components: rationale of two-index strat-
egy revisited. Struct Equ Modeling. 12(3):343–367.
Fan X, Sivo SA. 2007. Sensitivity of fit indices to model misspecification
and model types. Multivariate Behav Res. 42(3):509–529.
Ferns T, Meerabeau L. 2008. Verbal abuse experienced by nursing stu-
dents. J Adv Nurs. 61(4):436–444.
Fitzgerald LF, Gelfand MJ, Drasgow F. 1995. Measuring sexual harass-
ment: theoretical and psychometric advances. Basic Appl Soc Psych.
17(4):425–445.
Flora DB, Curran PJ. 2004. An empirical evaluation of alternative meth-
ods of estimation for confirmatory factor analysis with ordinal data.
Psychol Methods. 9(4):466–491.
Fnais N, Soobiah C, Chen M, Lillie E, Perrier L, Tashkhandi M, Straus S,
Mamdani M, Al-Omran M, Tricco A. 2014. Harassment and discrimin-
ation in medical training: a systematic review and meta-analysis.
Acad Med. 89(5):817–827.
Foster B, Mackie B, Barnett N. 2004. Bullying in the health sector: a
study of bullying of nursing students. N Z J Employ Relat. 29:67–83.
Gamble Blakey A, Smith-Han K, Anderson L, Collins E, Berryman E,
Wilkinson T. 2019a. Interventions addressing student bullying in the
clinical workplace: a narrative review. BMC Med Educ. 19(1):220.
Gamble Blakey A, Smith-Han K, Anderson L, Collins E, Berryman E,
Wilkinson T. 2019b. “‘They cared about us students:’learning from
exemplar clinical teaching environments.”BMC Med Educ. 19(1):119.
Gamble Blakey A, Smith-Han K, Anderson L, Collins E, Berryman E,
Wilkinson T. 2019c. It’s‘probably the teacher!’A strategic frame-
work for clinical staff engagement in clinical student bullying inter-
vention. BMC Med Educ. 19(1):116.
Garbin C, Zina L, Garbin A, Moimaz S. 2010. Sexual harassment in
Dentistry: prevalence in dental school. J Appl Oral Sci. 18(5):
447–452.
Hair JF, Black WC, Babin BJ, Anderson RE. 2010. Multivariate data ana-
lysis (7th ed.). Englewood Cliffs: Prentice Hall.
Hu L, Bentler PM. 1998. Fit indices in covariance structure modeling:
sensitivity to underparameterized model misspecification. Psychol
Methods. 3(4):424–453.
Keashly L. 1997. Emotional abuse in the workplace. J Aggress Maltreat
Trauma. 1(1):85–117.
Knapp K, Shane P, Sasaki-Hill D, Yoshizuka K, Chan P, Vo T. 2014.
Bullying in the clinical training of pharmacy students. Am J Pharm
Educ. 78(6):117.
Lubke GH, Muth
en BO. 2004. Applying multigroup confirmatory factor
models for continuous outcomes to Likert scale data complicates
meaningful group comparisons. Struct Equ Modeling. 11(4):
514–534.
MacCallum RC, Browne MW, Sugawara HM. 1996. Power analysis and
determination of sample size for covariance structure modeling.
Psychol Methods. 1(2):130–149.
McDonald R. 1999. Test theory: a unified treatment. Mahwah, NJ:
Lawrence Erlbaum Associates.
Mikkelsen EG, Einarsen S. 2001. Bullying in danish work-life: prevalence
and health correlates. Eur J Work Organ Psychol. 10(4):393–413.
Muth
en LK, Muth
en BO. 2012. Mplus: statistical analysis with latent
variables: user’s guide. Los Angeles, CA: Muth
en & Muth
en.
Noar SM. 2003. The role of structural equation modeling in scale
development. Struct Equ Modeling. 10(4):622–647.
Owoaje ET, Uchendu OC, Ige OK. 2012. Experiences of mistreatment
among medical students in a university in south west Nigeria. Niger
J Clin Pract. 15(2):214–219.
Premadasa IG, Wanigasooriya NC, Thalib L, Ellepola A. 2011.
Harassment of newly admitted undergraduates by senior students
in a Faculty of Dentistry in Sri Lanka. Med Teach. 33:556–563.
Przedworski JM, Dovidio JF, Hardeman RR, Phelan SM, Burke SE, Ruben
MA, Perry S, Burgess DJ, Nelson DB, Yeazel MW, et al. 2015. A com-
parison of the mental health and well-being of sexual minority and
heterosexual first-year medical students: a report from the medical
student change study. Acad Med. 90(5):652–659.
Rautio A, Sunnari V, Nuutinen M, Laitala M. 2005. Mistreatment of uni-
versity students most common during medical studies. BMC Med
Educ. 5:36.
Rees CE, Monrouxe LV. 2012. A morning since eight of just pure grill”:
a multischool qualitative study of student abuse. Acad Med. 86:
1374–1382.
Rees CE, Monrouxe LV, Ternan E, Endacott R. 2015. Workplace abuse
narratives from dentistry, nursing, pharmacy and physiotherapy stu-
dents: a multi-school qualitative study. Eur J Dent Educ. 19(2):
95–106.
Richardson DA, Becker M, Frank RR, Sokol RJ. 1997. Assessing medical
students’perceptions of mistreatment in their second and third
years. Acad Med. 72(8):728–730.
Rowland ML, Naidoo S, AbdulKadir R, Moraru R, Huang B, Pau A. 2010.
Perceptions of intimidation and bullying in dental schools: a multi-
national study. Int Dent J. 60(2):106–112.
Schneider KT, Hitlan RT, Radhakrishnan P. 2000. An examination of the
nature and correlates of ethnic harassment experiences in multiple
contexts. J Appl Psychol. 85(1):3–12.
Sheehan KH, Sheehan DV, White K, Leibowitz A, Baldwin DC. 1990. A
pilot study of medical student abuse: student perceptions of mis-
treatment and misconduct in medical school. JAMA. 263(4):
533–537.
Sijtsma K. 2009. On the use, the misuse, and the very limited useful-
ness of Cronbach’s Alpha. Psychometrika. 74(1):107–120.
Strauss ME, Smith GT. 2009. Construct validity: advances in theory and
methodology. Annu Rev Clin Psychol. 5:1–25.
Stubbs B, Soundy A. 2013. Physiotherapy students’experiences of bul-
lying on clinical internships: an exploratory study. Physiotherapy.
99(2):178–180.
Teo T, Fan X. 2013. Coefficient alpha and beyond: issues and
alternatives for educational research. Asia-Pacific Edu Res. 22(2):
209–213.
Till F. 1980. Sexual harassment: a report on the sexual harassment of
students. Washington, DC: National Advisory Council on Women’s
Educational Programs.
Uhari M, Kokkonen J, Nuutinen M, Vainionpaa L, Rantala H, Lautala P,
V€
ayrynen M. 1994. Medical student abuse: an international
phenomenon. JAMA. 271(13):1049–1051.
White GE. 2000. Sexual harassment during medical training: the per-
ceptions of medical students at a university medical school in
Australia. Med Educ. 34(12):980–986.
Whiteside D, Stubbs B, Soundy A. 2014. Physiotherapy students’expe-
riences of bullying on clinical internships: a qualitative study.
Physiotherapy. 100(1):41–46.
Wilkinson T, Gill DJ, Fitzjohn J, Palmer CL, Mulder RT. 2006. The impact
on students of adverse experiences during medical school. Med
Teach. 28(2):129–135.
Witte FM, Stratton TD, Nora LM. 2006. Stories from the field: students’
descriptions of gender discrimination and sexual harassment during
medical school. Acad Med. 81(7):648–654.
Wolf T, Scurria PL, Webster MG. 1998. A four-year study of anxiety,
depression, loneliness, social support, and perceived mistreatment
in medical students. J Health Psychol. 3(1):125–136.
Woolley DC, Paolo AM, Bonaminio GA, Moser SE. 2006. Student treat-
ment on clerkships based on their specialty interests. Teach Learn
Med. 18(3):237–243.
Yu CY. 2002. Evaluating cut-off criteria of model fit indices for latent
variable models with binary and continuous outcomes [Doctoral
dissertation]. University of California Los Angeles.
MEDICAL TEACHER 821
