Content uploaded by Andrew J Scally
Author content
All content in this area was uploaded by Andrew J Scally on Oct 10, 2015
Content may be subject to copyright.
Measurement and Evaluation in
Counseling and Development
2014, Vol 47(1) 79 –86
© The Author(s) 2013
Reprints and permissions:
sagepub.com/journalsPermissions.nav
DOI: 10.1177/0748175613513808
mecd.sagepub.com
Methods Plainly Speaking
Introduction
Content validation refers to a process that
aims to provide assurance that an instrument
(checklist, questionnaire, or scale) measures
the content area it is expected to measure
(Frank-Stromberg & Olsen, 2004). One way
of achieving content validity involves a panel
of subject matter experts considering the
importance of individual items within an
instrument. Lawshe’s method, initially pro-
posed in a seminal paper in 1975 (Lawshe,
1975), has been widely used to establish and
quantify content validity in diverse fields
including health care, education, organiza-
tional development, personnel psychology,
and market research (Wilson, Pan, & Schum-
sky, 2012). It involves a panel of subject mat-
ter “experts” rating items into one of three
categories: “essential,” “useful, but not
essential,” or “not necessary.” Items deemed
“essential” by a critical number of panel
members are then included within the final
instrument, with items failing to achieve this
critical level discarded. Lawshe (1975) sug-
gested that based on “established psychophys-
ical principles,” a level of 50% agreement
gives some assurance of content validity.
The CVR (content validity ratio) proposed
by Lawshe (1975) is a linear transformation of
a proportional level of agreement on how
many “experts” within a panel rate an item
“essential” calculated in the following way:
CVR
nN
N
e
=−
()
/
/
,
2
2
where CVR is the content validity ratio, ne is
the number of panel members indicating an
item “essential,” and N is the number of panel
members.
Lawshe (1975) suggested the transforma-
tion (from proportion to CVR) was of worth
as it could readily be seen whether the level of
agreement among panel members was greater
than 50%. CVR values range between −1
(perfect disagreement) and +1 (perfect agree-
ment) with CVR values above zero indicating
that over half of panel members agree an item
essential. However, when interpreting a CVR
for any given item, it may be important to
513808MECXXX10.1177/0748175613513808Measurement and Evaluation in Counseling and Development 47(1)Ayre and Scally
research-article2013
1University of Bradford, Bradford, UK
Corresponding Author:
Colin Ayre, University of Bradford, School of Health
Studies, Bradford BD7 1DP, UK.
Email: c.a.ayre1@bradford.ac.uk
Critical Values for Lawshe’s
Content Validity Ratio: Revisiting
the Original Methods of Calculation
Colin Ayre1 and Andrew John Scally1
Abstract
The content validity ratio originally proposed by Lawshe is widely used to quantify content
validity and yet methods used to calculate the original critical values were never reported.
Methods for original calculation of critical values are suggested along with tables of exact
binomial probabilities.
Keywords
measurement, content validity, validation design, content validity ratio
by guest on July 6, 2015mec.sagepub.comDownloaded from
80 Measurement and Evaluation in Counseling and Development 47(1)
consider whether the level of agreement is
also above that which may have occurred by
chance. As a result, Lawshe reported a table
of critical CVR (CVRcritical) values computed
by his colleague Lowell Schipper, where
CVRcritical is the lowest level of CVR such that
the level of agreement exceeds that of chance
for a given item, for a given alpha (Type I
error probability, suggested to be .05 using a
one-tailed test). CVRcritical values can be used
to determine how many panel members need
to agree an item essential and thus which
items should be included or discarded from
the final instrument. To include or discard
items from a given instrument appropriately,
it is imperative that the CVRcritical values are
accurate. Recently, concern has been raised
that the original methods used for calculating
CVRcritical were not reported in Lawshe’s arti-
cle on content validity, and as both Lawshe
and Schipper have since passed away, it is
now not possible to gain clarification (Wilson
et al., 2012). Furthermore, an apparent anom-
aly exists in the table of critical values between
panel sizes of 8 and 9, where CVRcritical unex-
pectedly rises to 0.78 from 0.75 before mono-
tonically decreasing with increasing panel
size up to the calculated maximum panel size
of 40. This led Wilson et al. (2012) to try and
identify the method used by Schipper to calcu-
late the original CVRcritical values in Lawshe
(1975) in the hope of providing corrected
values.
Despite their attempts, Wilson et al.
(2012) fell short of their aims. They sug-
gested that Schipper had used the normal
approximation to the binomial distribution
for panel sizes of 10 or more, yet these claims
were theoretical as they were unable to
reproduce the values of CVRcritical reported in
Lawshe (1975). As values for CVRcritical cal-
culated by Wilson and colleagues differed
significantly from those reported in Lawshe
(1975), it was suggested that instead of the
one-tailed test reported Schipper had, in fact,
used a two-tailed test as this more closely
resembled their results. In addition, for panel
sizes below 10 no satisfactory explanation
was provided of how CVRcritical may have
originally been calculated. Furthermore, they
were unable to provide a satisfactory expla-
nation for the apparent anomaly between
panel sizes of 8 and 9.
In their article, Wilson et al. (2012) pro-
duced a new table of CVRcritical values using
the normal approximation to the binomial dis-
tribution. This method we believe to be inferior
to calculation of exact binomial probabilities
as, by definition, it is ultimately just an approx-
imation and may not be valid for small sample
sizes and for proportions approaching 0 or 1
(Armitage, Berry, & Matthews, 2002). It is
understandable that a normal approximation
was used for larger panel sizes when Schipper
calculated the original CVRcritical values in
1975, but as statistical programs can now read-
ily calculate exact binomial probabilities, it
would seem more appropriate to do so in the
present day. We had further concerns regard-
ing the methods used by Wilson et al. (2012)
to calculate the normal approximation, as it
appeared a continuity correction had not been
employed. In cases where the continuous nor-
mal distribution has been used to approximate
the discrete binomial distribution more accu-
rate results are obtained through use of a con-
tinuity correction (Gallin & Ognibene, 2007;
Rumsey, 2006).
Based on the wide discrepancy between
CVRcritical reported by Wilson et al. (2012) and
Lawshe (1975), we intended to answer the
following questions:
1. Did Wilson et al. (2012) correctly
employ a method for calculating bino-
mial probabilities?
2. What method was employed by Schip-
per to calculate CVRcritical in Lawshe
(1975) for all panel sizes?
3. Are there anomalies in Schipper’s
table of critical values in Lawshe
(1975)?
4. Did Lawshe report CVRcritical for a one-
tailed test or a two-tailed test?
As a result of our belief that exact binomial
probabilities were more appropriate than nor-
mal approximations, we also intended to cal-
culate exact binomial probabilities for all
panel sizes between 5 and 40.
by guest on July 6, 2015mec.sagepub.comDownloaded from
Ayre and Scally 81
Method
We calculated the minimum number of
experts required to agree an item “essential”
for a given panel size, such that the level of
agreement exceed that of chance. In keeping
with previous work, we assumed the outcome
as dichotomous (i.e., “essential” or “not
essential”) although we acknowledge it could
be considered trichotomous as there are three
possible outcomes when rating any given item
(“essential,” “important, but not essential,”
and “not necessary”). As the CVR is designed
to show a level of agreement above that of
chance, we are only concerned with testing in
one direction. Thus, in this case a one-tailed
hypothesis test is appropriate.
Hypothesis:
H0: ne = N/2
Significance (α) was set at .05.
Using a one-tailed test, we would reject H0
if P(ne ≥ ncritical) ≤ .05, where ncritical is the low-
est number of experts required to agree an
item “essential” for agreement to be above
that of chance and ne = the number of experts
rating an item as “essential.”
We calculated exact CVRcritical values for
panel sizes between 5 and 40, based on the
discrete binomial distribution, computed
using Stata Statistical Software: Release 12
(StataCorp, College Station, TX). The follow-
ing command was used:
bitesti N ne p
where N is the total number of panel mem-
bers, ne is the number of experts agreeing
“essential,” and p is the hypothesized proba-
bility of success (agreeing the item as essen-
tial) = ½.
Using this method we produced a table of
the minimum number of experts (ne) required
to agree an item essential such that we could
reject H0 (i.e., the minimum number of experts
such that p ≤ .05). Values for CVRcritical were
then calculated on the basis of the minimum
number of experts required using the formula
for calculating CVR given previously in the
article. Exact one-sided p values are reported.
To allow direct comparison, we calculated
the exact binomial probabilities according to
the method used by Wilson et al. (2012),
described in their article, using the Microsoft
Excel function:
ncritical = CRITBINOM (n,p,1-α)
where ncritical is the minimum number of
experts required to agree an item essential, n
is the panel size, p is the probability of suc-
cess = ½, and α = .05.
Normal approximation to the binomial was
calculated using the following formula incor-
porating a continuity correction (Armitage et al.,
2002). This subtracts 0.5 from the number of
panel experts required to agree an item essen-
tial to account for using the continuous nor-
mal distribution for approximation of the
discrete binomial distribution.
znNp
Np pN
e
=−−
√−
()
(.)
[]
~(
,)
05
101
Therefore, as p = ½:
nzNN
e=
+
+
22
05
.,
where z is normal approximation of the bino-
mial, N is the total number of panel members,
ne is the number of experts agreeing “essen-
tial,” p is the probability of agreeing each
item essential = ½, and 0.5 is the continuity
correction.
CVR based on the normal approximation
was calculated in the following way:
CVR =
()
+
()
[/
].
/
zN
N
20
5
2
Therefore,
CVR=
()
+
[]
.
zN
N
1
Normal approximations for CVR critical were
calculated using this method for all panel
sizes to allow comparison with previous work.
Results
The calculations for CVRcritical based on exact
binomial probabilities for panel sizes of 5 to
by guest on July 6, 2015mec.sagepub.comDownloaded from
82 Measurement and Evaluation in Counseling and Development 47(1)
40 are shown in Table 1. Calculations using
the CRITBINOM function returned values for
the critical number of experts 1 fewer for all
panel sizes compared with our calculations
(Table 1).
Figure 1 shows a comparison of CVRcritical
values from our exact binomial and normal
approximation to the binomial calculations
and those reported by Lawshe (1975) and Wil-
son et al. (2012). Normal approximation using
the continuity correction returned values equal
to those reported in Lawshe (1975) for all
given panel sizes of 10 and above other than a
minor difference of 0.01 for a panel size of 13.
Table 1. CVRcritical One-Tailed Test (α = .05) Based on Exact Binomial Probabilities.
N (Panel Size)
Proportion
Agreeing
Essential
CVRCritical
Exact
Values
One-Sided
p Value
Ncritical (Minimum Number
of Experts Required to
Agree Item Essential)—
Ayre and Scally, This Article
Ncritical Calculated
From CRITBINOM
Function—Wilson
et al. (2012)
5 1 1.00 .031 5 4
6 1 1.00 .016 6 5
7 1 1.00 .008 7 6
8 .875 .750 .035 7 6
9 .889 .778 .020 8 7
10 .900 .800 .011 9 8
11 .818 .636 .033 9 8
12 .833 .667 .019 10 9
13 .769 .538 .046 10 9
14 .786 .571 .029 11 10
15 .800 .600 .018 12 11
16 .750 .500 .038 12 11
17 .765 .529 .025 13 12
18 .722 .444 .048 13 12
19 .737 .474 .032 14 13
20 .750 .500 .021 15 14
21 .714 .429 .039 15 14
22 .727 .455 .026 16 15
23 .696 .391 .047 16 15
24 .708 .417 .032 17 16
25 .720 .440 .022 18 17
26 .692 .385 .038 18 17
27 .704 .407 .026 19 18
28 .679 .357 .044 19 18
29 .690 .379 .031 20 19
30 .667 .333 .049 20 19
31 .677 .355 .035 21 20
32 .688 .375 .025 22 21
33 .667 .333 .040 22 21
34 .676 .353 .029 23 22
35 .657 .314 .045 23 22
36 .667 .333 .033 24 23
37 .649 .297 .049 24 23
38 .658 .316 .036 25 24
39 .667 .333 .027 26 25
40 .650 .300 .040 26 25
by guest on July 6, 2015mec.sagepub.comDownloaded from
Ayre and Scally 83
Discussion
We have produced a table of exact values for
CVRcritical including the minimum number of
panel members required such that agreement
is above that of chance. We believe we are the
first to produce a table of values for CVRcritical
from exact binomial probabilities. In contrast
to previous work, all of the values for CVR-
critical are calculated based on an achievable
CVR, given the discrete nature of the vari-
ables under investigation.
Comparison with previous work is given
below.
Comparison to Lawshe (1975)
The exact critical values for CVR we have
produced are equal to those given in Lawshe
(1975) for panel sizes below 10, allowing for
adjustments and rounding (see Figure 1). We
therefore believe that Lawshe (1975) calcu-
lated exact binomial probabilities for panel
sizes below 10. This approach is reasonable as
the use of a normal approximation for a bino-
mial distribution is only justifiable when:
Np > 5 and N(1-p) > 5 (Rumsey, 2006). Where
N = the number of panel members and p = the
probability of success in any trial.
This would be satisfied for panel sizes above
10 assuming p = ½.
We do not believe that there is an anomaly
for panel sizes between 8 and 9 in CVRcritical
reported in Lawshe (1975). It can be seen in
Figure 1 that CVRcritical does increase between
panel size of 8 and 9, related to the discrete
nature of both the panel size and number of
experts who can agree any item is essential. It
can be seen from our calculations that,
although the overall pattern is for CVRcritical to
fall with increasing panel size, there are a
number of instances where CVRcritical increases.
This is an important consideration when deter-
mining panel size for those using the CVR
method to gain content validity.
For panel sizes of 10 and above the nor-
mal approximation to the binomial has been
calculated and we have been able to repro-
duce the same values reported in Lawshe
(1975) notwithstanding a minor discrepancy
for a panel size of 13 (see Figure 1). As the
normal distribution is based on a continuous
distribution and it is being used to approxi-
mate a discrete distribution, Schipper has
correctly used a continuity correction which
will more likely result in more accurate
approximations. It would appear that Schip-
per and Wilson et al. used identical methods
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
510152025303540
Content validity rao (CVR)
Panel size
CVR crical exact
CVR crical normal
approximaon (Ayre and
Scally, this paper)
CVR crical normal
approximaon (Wilson et
al, 2012)
CVR crical (Lawshe, 1975)
Figure 1. Chart showing comparison between critical values for content validity ratio.
by guest on July 6, 2015mec.sagepub.comDownloaded from
84 Measurement and Evaluation in Counseling and Development 47(1)
for calculating CVRcritical with the exception
of the continuity correction.
As the values we have calculated are the
same as those of Lawshe (1975) it is apparent
they have also used a one-tailed test at p = .05
as they originally reported, and not a two-
tailed test as suggested by Wilson et al. (2012).
Can the critical CVR values given by
Lawshe (1975) be used to accurately deter-
mine panel size?
In general, use of the originally calculated
CVR values from Lawshe (1975) yields an
equal value for the critical number of experts
required as shown in our exact calculations.
The only discrepancy occurs for a panel size
of 13 where the exact CVRcritical is marginally
under that reported by Lawshe. Importantly,
our findings would suggest that question-
naires and checklists developed using the
CVRcritical values originally reported by Law-
she (1975) remain valid.
Comparison to Wilson et al. (2012)
CVRcritical Based on Exact Binomial Probabili-
ties. The exact CVRcritical based on binomial
probabilities we have calculated using Stata
differ from those given by the CRITBINOM
function in Microsoft Excel employed by
Wilson et al. (2012) as a result of the discrep-
ancy in the critical number of experts required
to agree an item “essential” produced by each
method (see Table 1). We believe that Wilson
et al. (2012) have incorrectly interpreted the
result returned from the CRITBINOM func-
tion and therefore the CVRcritical based on the
exact binomial probabilities shown in Figure 1
of their article are incorrect. The method used
by Wilson et al. (2012) returns one fewer than
the true critical number of experts required to
ensure agreement above that of chance for a
given value of α (Table 1) yet no mention of
this can be seen in their article. This can be
illustrated through an example using a panel
size of 15.
Example: Considering a panel size (n) of 15,
probability of success (p) of .5 and α = .05.
CRITBINOM (15, 0.5, 0.95) = 11
As this utilizes the cumulative binomial prob-
ability the interpretation of this result is “there
is at least a probability of 0.95 of getting 11 or
fewer successes.” Thus, there is at most a
probability of .05 of getting 12 or more suc-
cesses. This is the critical number we are
interested in to assure a level of agreement
above that of chance at α set at .05.
The error in calculating exact binomial
probabilities may explain why Wilson et al.
(2012) failed to realize that Shipper had calcu-
lated exact binomial probabilities up to a
panel size of 10.
CVRcritical Based on the Normal Approximation to
the Binomial. CVRcritical values reported by
Wilson et al. (2012) based on a normal approx-
imation to the binomial distribution are mark-
edly lower than those we have calculated using
a continuity correction for all given panel sizes
(see Figure 1). It is clear from their formula for
calculating the critical value for CVR that a
continuity correction was not used by Wilson
et al. Conversely, the values given in Lawshe
are consistent with the normal approximation
using the continuity correction and are there-
fore closer to the exact binomial probabilities
we have reported in this article. On this basis,
we believe that the recalculated values for
CVRcritical reported in Wilson et al. (2012) are
inaccurate and therefore should not be used.
Wilson et al. (2012) and Lawshe (1975)
have both calculated CVRcritical values for
panel sizes of 10 or more (Wilson et al., 2012,
used the normal approximation for all panel
sizes) based on a normal approximation of the
binomial distribution. We believe this is an
inferior method to the exact calculations we
have reported for the following reasons:
1. If the normal approximation value for
CVRcritical is higher than that produced
from exact calculations of binomial
probability the panel size deemed nec-
essary will be higher than required.
2. If the normal approximation value for
CVRcritical is lower than that produced
from exact calculations of binomial
probability the panel size deemed nec-
essary may be lower than required.
by guest on July 6, 2015mec.sagepub.comDownloaded from
Ayre and Scally 85
Table 2. Simplified Table of CVRcritical Including the Number of Experts Required to Agree an Item
Essential.
Panel Size
Ncritical (Minimum Number of
Experts Required to Agree an
Item Essential for Inclusion)
Proportion
Agreeing
Essential CVRcritical
5 5 1 1.00
6 6 1 1.00
7 7 1 1.00
8 7 .875 .750
9 8 .889 .778
10 9 .900 .800
11 9 .818 .636
12 10 .833 .667
13 10 .769 .538
14 11 .786 .571
15 12 .800 .600
16 12 .750 .500
17 13 .765 .529
18 13 .722 .444
19 14 .737 .474
20 15 .750 .500
21 15 .714 .429
22 16 .727 .455
23 16 .696 .391
24 17 .708 .417
25 18 .720 .440
26 18 .692 .385
27 19 .704 .407
28 19 .679 .357
29 20 .690 .379
30 20 .667 .333
31 21 .677 .355
32 22 .688 .375
33 22 .667 .333
34 23 .676 .353
35 23 .657 .314
36 24 .667 .333
37 24 .649 .297
38 25 .658 .316
39 26 .667 .333
40 26 .650 .300
Presented above is a simplified table of CVR-
critical values, calculated using exact binomial
probabilities, which includes the number of
experts required to agree any given item is
essential (Table 2).
It can be seen from Table 2 that preferred
panel sizes exist, when the addition of a
further panel member leads to a significant
reduction in the required proportion level of
agreement that an item is “essential” for it to
be included (e.g., between panel sizes of 12
and 13). In addition, it is also immediately
apparent that increasing the panel size by 1
will actually increase the required proportion
level of agreement on occasions (e.g.,
between panel sizes of 13 and 14). We believe
by guest on July 6, 2015mec.sagepub.comDownloaded from
86 Measurement and Evaluation in Counseling and Development 47(1)
this table is of most use to researchers’ wish-
ing to quantify content validity using the
CVR method, both to decide the most appro-
priate panel size and when determining
whether a critical level of agreement has
been reached.
Conclusions
The method used by Schipper to calculate
the original critical values reported in Law-
she’s article has been suggested and we have
been able to successfully reproduce the val-
ues using discrete calculation for panel sizes
below 10 and normal approximation to the
binomial for panel sizes of 10 and above.
We have identified problems with both the
discrete calculations and normal approxima-
tion to the binomial suggested by Wilson et al.
Consequently, we do not believe that values
for CVRcritical reported in Wilson et al. should
be used to determine whether a critical level
of agreement has been reached and therefore
whether items should be included or
excluded from a given instrument. Although
it is safe to use the values for CVRcritical pro-
posed by Lawshe to determine whether
items should be included on an instrument,
we believe that exact CVRcritical based
on discrete binomial calculations is most
appropriate.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of
interest with respect to the research, authorship,
and/or publication of this article.
Funding
The author(s) received no financial support for the
research, authorship, and/or publication of this article.
References
Armitage, P., Berry, G., & Matthews, J. N. S.
(2002). Statistical methods in medical research
(4th ed.). Oxford, England: Blackwell.
Frank-Stromberg, M., & Olsen, S. J. (2004).
Instruments for clinical health-care research.
London, England: Jones & Bartlett.
Gallin, J. I., & Ognibene, F. P. (2007). Principles
and practice of clinical research (2nd ed.).
Boston, MA: Elsevier.
Lawshe, C. H. (1975). A quantitative approach to con-
tent validity. Personnel psychology, 28, 563–575.
Rumsey, D. (2006). Probability for dummies.
Indianapolis, IN: Wiley.
Wilson, F. R., Pan, W., & Schumsky, D. A. (2012).
Recalculation of the critical values for Lawshe’s
content validity ratio. Measurement and
Evaluation in Counseling and Development,
45, 197–210. doi:10.1177/0748175612440286
Author Biographies
Colin Ayre is a PhD student and honorary lecturer
in the School of Health Studies, University of
Bradford. He is also a qualified physiotherapist
working in Bradford Teaching Hospitals NHS
Foundation Trust.
Andrew John Scally is a senior lecturer in the
School of Health Studies, University of Bradford. He
is a qualified radiographer, a graduate physicist and
has Master’s degrees in computational physics and
medical statistics. His current professional role com-
bines work in both radiation physics/radiological
protection and the application of medical statistics in
a wide range of medical and health disciplines.
by guest on July 6, 2015mec.sagepub.comDownloaded from
