Content uploaded by Andrew J Scally

Author content

All content in this area was uploaded by Andrew J Scally on Oct 10, 2015

Content may be subject to copyright.

Measurement and Evaluation in

Counseling and Development

2014, Vol 47(1) 79 –86

© The Author(s) 2013

Reprints and permissions:

sagepub.com/journalsPermissions.nav

DOI: 10.1177/0748175613513808

mecd.sagepub.com

Methods Plainly Speaking

Introduction

Content validation refers to a process that

aims to provide assurance that an instrument

(checklist, questionnaire, or scale) measures

the content area it is expected to measure

(Frank-Stromberg & Olsen, 2004). One way

of achieving content validity involves a panel

of subject matter experts considering the

importance of individual items within an

instrument. Lawshe’s method, initially pro-

posed in a seminal paper in 1975 (Lawshe,

1975), has been widely used to establish and

quantify content validity in diverse fields

including health care, education, organiza-

tional development, personnel psychology,

and market research (Wilson, Pan, & Schum-

sky, 2012). It involves a panel of subject mat-

ter “experts” rating items into one of three

categories: “essential,” “useful, but not

essential,” or “not necessary.” Items deemed

“essential” by a critical number of panel

members are then included within the final

instrument, with items failing to achieve this

critical level discarded. Lawshe (1975) sug-

gested that based on “established psychophys-

ical principles,” a level of 50% agreement

gives some assurance of content validity.

The CVR (content validity ratio) proposed

by Lawshe (1975) is a linear transformation of

a proportional level of agreement on how

many “experts” within a panel rate an item

“essential” calculated in the following way:

CVR

nN

N

e

=−

()

/

/

,

2

2

where CVR is the content validity ratio, ne is

the number of panel members indicating an

item “essential,” and N is the number of panel

members.

Lawshe (1975) suggested the transforma-

tion (from proportion to CVR) was of worth

as it could readily be seen whether the level of

agreement among panel members was greater

than 50%. CVR values range between −1

(perfect disagreement) and +1 (perfect agree-

ment) with CVR values above zero indicating

that over half of panel members agree an item

essential. However, when interpreting a CVR

for any given item, it may be important to

513808MECXXX10.1177/0748175613513808Measurement and Evaluation in Counseling and Development 47(1)Ayre and Scally

research-article2013

1University of Bradford, Bradford, UK

Corresponding Author:

Colin Ayre, University of Bradford, School of Health

Studies, Bradford BD7 1DP, UK.

Email: c.a.ayre1@bradford.ac.uk

Critical Values for Lawshe’s

Content Validity Ratio: Revisiting

the Original Methods of Calculation

Colin Ayre1 and Andrew John Scally1

Abstract

The content validity ratio originally proposed by Lawshe is widely used to quantify content

validity and yet methods used to calculate the original critical values were never reported.

Methods for original calculation of critical values are suggested along with tables of exact

binomial probabilities.

Keywords

measurement, content validity, validation design, content validity ratio

by guest on July 6, 2015mec.sagepub.comDownloaded from

80 Measurement and Evaluation in Counseling and Development 47(1)

consider whether the level of agreement is

also above that which may have occurred by

chance. As a result, Lawshe reported a table

of critical CVR (CVRcritical) values computed

by his colleague Lowell Schipper, where

CVRcritical is the lowest level of CVR such that

the level of agreement exceeds that of chance

for a given item, for a given alpha (Type I

error probability, suggested to be .05 using a

one-tailed test). CVRcritical values can be used

to determine how many panel members need

to agree an item essential and thus which

items should be included or discarded from

the final instrument. To include or discard

items from a given instrument appropriately,

it is imperative that the CVRcritical values are

accurate. Recently, concern has been raised

that the original methods used for calculating

CVRcritical were not reported in Lawshe’s arti-

cle on content validity, and as both Lawshe

and Schipper have since passed away, it is

now not possible to gain clarification (Wilson

et al., 2012). Furthermore, an apparent anom-

aly exists in the table of critical values between

panel sizes of 8 and 9, where CVRcritical unex-

pectedly rises to 0.78 from 0.75 before mono-

tonically decreasing with increasing panel

size up to the calculated maximum panel size

of 40. This led Wilson et al. (2012) to try and

identify the method used by Schipper to calcu-

late the original CVRcritical values in Lawshe

(1975) in the hope of providing corrected

values.

Despite their attempts, Wilson et al.

(2012) fell short of their aims. They sug-

gested that Schipper had used the normal

approximation to the binomial distribution

for panel sizes of 10 or more, yet these claims

were theoretical as they were unable to

reproduce the values of CVRcritical reported in

Lawshe (1975). As values for CVRcritical cal-

culated by Wilson and colleagues differed

significantly from those reported in Lawshe

(1975), it was suggested that instead of the

one-tailed test reported Schipper had, in fact,

used a two-tailed test as this more closely

resembled their results. In addition, for panel

sizes below 10 no satisfactory explanation

was provided of how CVRcritical may have

originally been calculated. Furthermore, they

were unable to provide a satisfactory expla-

nation for the apparent anomaly between

panel sizes of 8 and 9.

In their article, Wilson et al. (2012) pro-

duced a new table of CVRcritical values using

the normal approximation to the binomial dis-

tribution. This method we believe to be inferior

to calculation of exact binomial probabilities

as, by definition, it is ultimately just an approx-

imation and may not be valid for small sample

sizes and for proportions approaching 0 or 1

(Armitage, Berry, & Matthews, 2002). It is

understandable that a normal approximation

was used for larger panel sizes when Schipper

calculated the original CVRcritical values in

1975, but as statistical programs can now read-

ily calculate exact binomial probabilities, it

would seem more appropriate to do so in the

present day. We had further concerns regard-

ing the methods used by Wilson et al. (2012)

to calculate the normal approximation, as it

appeared a continuity correction had not been

employed. In cases where the continuous nor-

mal distribution has been used to approximate

the discrete binomial distribution more accu-

rate results are obtained through use of a con-

tinuity correction (Gallin & Ognibene, 2007;

Rumsey, 2006).

Based on the wide discrepancy between

CVRcritical reported by Wilson et al. (2012) and

Lawshe (1975), we intended to answer the

following questions:

1. Did Wilson et al. (2012) correctly

employ a method for calculating bino-

mial probabilities?

2. What method was employed by Schip-

per to calculate CVRcritical in Lawshe

(1975) for all panel sizes?

3. Are there anomalies in Schipper’s

table of critical values in Lawshe

(1975)?

4. Did Lawshe report CVRcritical for a one-

tailed test or a two-tailed test?

As a result of our belief that exact binomial

probabilities were more appropriate than nor-

mal approximations, we also intended to cal-

culate exact binomial probabilities for all

panel sizes between 5 and 40.

by guest on July 6, 2015mec.sagepub.comDownloaded from

Ayre and Scally 81

Method

We calculated the minimum number of

experts required to agree an item “essential”

for a given panel size, such that the level of

agreement exceed that of chance. In keeping

with previous work, we assumed the outcome

as dichotomous (i.e., “essential” or “not

essential”) although we acknowledge it could

be considered trichotomous as there are three

possible outcomes when rating any given item

(“essential,” “important, but not essential,”

and “not necessary”). As the CVR is designed

to show a level of agreement above that of

chance, we are only concerned with testing in

one direction. Thus, in this case a one-tailed

hypothesis test is appropriate.

Hypothesis:

H0: ne = N/2

Significance (α) was set at .05.

Using a one-tailed test, we would reject H0

if P(ne ≥ ncritical) ≤ .05, where ncritical is the low-

est number of experts required to agree an

item “essential” for agreement to be above

that of chance and ne = the number of experts

rating an item as “essential.”

We calculated exact CVRcritical values for

panel sizes between 5 and 40, based on the

discrete binomial distribution, computed

using Stata Statistical Software: Release 12

(StataCorp, College Station, TX). The follow-

ing command was used:

bitesti N ne p

where N is the total number of panel mem-

bers, ne is the number of experts agreeing

“essential,” and p is the hypothesized proba-

bility of success (agreeing the item as essen-

tial) = ½.

Using this method we produced a table of

the minimum number of experts (ne) required

to agree an item essential such that we could

reject H0 (i.e., the minimum number of experts

such that p ≤ .05). Values for CVRcritical were

then calculated on the basis of the minimum

number of experts required using the formula

for calculating CVR given previously in the

article. Exact one-sided p values are reported.

To allow direct comparison, we calculated

the exact binomial probabilities according to

the method used by Wilson et al. (2012),

described in their article, using the Microsoft

Excel function:

ncritical = CRITBINOM (n,p,1-α)

where ncritical is the minimum number of

experts required to agree an item essential, n

is the panel size, p is the probability of suc-

cess = ½, and α = .05.

Normal approximation to the binomial was

calculated using the following formula incor-

porating a continuity correction (Armitage et al.,

2002). This subtracts 0.5 from the number of

panel experts required to agree an item essen-

tial to account for using the continuous nor-

mal distribution for approximation of the

discrete binomial distribution.

znNp

Np pN

e

=−−

√−

()

(.)

[]

~(

,)

05

101

Therefore, as p = ½:

nzNN

e=

+

+

22

05

.,

where z is normal approximation of the bino-

mial, N is the total number of panel members,

ne is the number of experts agreeing “essen-

tial,” p is the probability of agreeing each

item essential = ½, and 0.5 is the continuity

correction.

CVR based on the normal approximation

was calculated in the following way:

CVR =

()

+

()

[/

].

/

zN

N

20

5

2

Therefore,

CVR=

()

+

[]

.

zN

N

1

Normal approximations for CVR critical were

calculated using this method for all panel

sizes to allow comparison with previous work.

Results

The calculations for CVRcritical based on exact

binomial probabilities for panel sizes of 5 to

by guest on July 6, 2015mec.sagepub.comDownloaded from

82 Measurement and Evaluation in Counseling and Development 47(1)

40 are shown in Table 1. Calculations using

the CRITBINOM function returned values for

the critical number of experts 1 fewer for all

panel sizes compared with our calculations

(Table 1).

Figure 1 shows a comparison of CVRcritical

values from our exact binomial and normal

approximation to the binomial calculations

and those reported by Lawshe (1975) and Wil-

son et al. (2012). Normal approximation using

the continuity correction returned values equal

to those reported in Lawshe (1975) for all

given panel sizes of 10 and above other than a

minor difference of 0.01 for a panel size of 13.

Table 1. CVRcritical One-Tailed Test (α = .05) Based on Exact Binomial Probabilities.

N (Panel Size)

Proportion

Agreeing

Essential

CVRCritical

Exact

Values

One-Sided

p Value

Ncritical (Minimum Number

of Experts Required to

Agree Item Essential)—

Ayre and Scally, This Article

Ncritical Calculated

From CRITBINOM

Function—Wilson

et al. (2012)

5 1 1.00 .031 5 4

6 1 1.00 .016 6 5

7 1 1.00 .008 7 6

8 .875 .750 .035 7 6

9 .889 .778 .020 8 7

10 .900 .800 .011 9 8

11 .818 .636 .033 9 8

12 .833 .667 .019 10 9

13 .769 .538 .046 10 9

14 .786 .571 .029 11 10

15 .800 .600 .018 12 11

16 .750 .500 .038 12 11

17 .765 .529 .025 13 12

18 .722 .444 .048 13 12

19 .737 .474 .032 14 13

20 .750 .500 .021 15 14

21 .714 .429 .039 15 14

22 .727 .455 .026 16 15

23 .696 .391 .047 16 15

24 .708 .417 .032 17 16

25 .720 .440 .022 18 17

26 .692 .385 .038 18 17

27 .704 .407 .026 19 18

28 .679 .357 .044 19 18

29 .690 .379 .031 20 19

30 .667 .333 .049 20 19

31 .677 .355 .035 21 20

32 .688 .375 .025 22 21

33 .667 .333 .040 22 21

34 .676 .353 .029 23 22

35 .657 .314 .045 23 22

36 .667 .333 .033 24 23

37 .649 .297 .049 24 23

38 .658 .316 .036 25 24

39 .667 .333 .027 26 25

40 .650 .300 .040 26 25

by guest on July 6, 2015mec.sagepub.comDownloaded from

Ayre and Scally 83

Discussion

We have produced a table of exact values for

CVRcritical including the minimum number of

panel members required such that agreement

is above that of chance. We believe we are the

first to produce a table of values for CVRcritical

from exact binomial probabilities. In contrast

to previous work, all of the values for CVR-

critical are calculated based on an achievable

CVR, given the discrete nature of the vari-

ables under investigation.

Comparison with previous work is given

below.

Comparison to Lawshe (1975)

The exact critical values for CVR we have

produced are equal to those given in Lawshe

(1975) for panel sizes below 10, allowing for

adjustments and rounding (see Figure 1). We

therefore believe that Lawshe (1975) calcu-

lated exact binomial probabilities for panel

sizes below 10. This approach is reasonable as

the use of a normal approximation for a bino-

mial distribution is only justifiable when:

Np > 5 and N(1-p) > 5 (Rumsey, 2006). Where

N = the number of panel members and p = the

probability of success in any trial.

This would be satisfied for panel sizes above

10 assuming p = ½.

We do not believe that there is an anomaly

for panel sizes between 8 and 9 in CVRcritical

reported in Lawshe (1975). It can be seen in

Figure 1 that CVRcritical does increase between

panel size of 8 and 9, related to the discrete

nature of both the panel size and number of

experts who can agree any item is essential. It

can be seen from our calculations that,

although the overall pattern is for CVRcritical to

fall with increasing panel size, there are a

number of instances where CVRcritical increases.

This is an important consideration when deter-

mining panel size for those using the CVR

method to gain content validity.

For panel sizes of 10 and above the nor-

mal approximation to the binomial has been

calculated and we have been able to repro-

duce the same values reported in Lawshe

(1975) notwithstanding a minor discrepancy

for a panel size of 13 (see Figure 1). As the

normal distribution is based on a continuous

distribution and it is being used to approxi-

mate a discrete distribution, Schipper has

correctly used a continuity correction which

will more likely result in more accurate

approximations. It would appear that Schip-

per and Wilson et al. used identical methods

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

510152025303540

Content validity rao (CVR)

Panel size

CVR crical exact

CVR crical normal

approximaon (Ayre and

Scally, this paper)

CVR crical normal

approximaon (Wilson et

al, 2012)

CVR crical (Lawshe, 1975)

Figure 1. Chart showing comparison between critical values for content validity ratio.

by guest on July 6, 2015mec.sagepub.comDownloaded from

84 Measurement and Evaluation in Counseling and Development 47(1)

for calculating CVRcritical with the exception

of the continuity correction.

As the values we have calculated are the

same as those of Lawshe (1975) it is apparent

they have also used a one-tailed test at p = .05

as they originally reported, and not a two-

tailed test as suggested by Wilson et al. (2012).

Can the critical CVR values given by

Lawshe (1975) be used to accurately deter-

mine panel size?

In general, use of the originally calculated

CVR values from Lawshe (1975) yields an

equal value for the critical number of experts

required as shown in our exact calculations.

The only discrepancy occurs for a panel size

of 13 where the exact CVRcritical is marginally

under that reported by Lawshe. Importantly,

our findings would suggest that question-

naires and checklists developed using the

CVRcritical values originally reported by Law-

she (1975) remain valid.

Comparison to Wilson et al. (2012)

CVRcritical Based on Exact Binomial Probabili-

ties. The exact CVRcritical based on binomial

probabilities we have calculated using Stata

differ from those given by the CRITBINOM

function in Microsoft Excel employed by

Wilson et al. (2012) as a result of the discrep-

ancy in the critical number of experts required

to agree an item “essential” produced by each

method (see Table 1). We believe that Wilson

et al. (2012) have incorrectly interpreted the

result returned from the CRITBINOM func-

tion and therefore the CVRcritical based on the

exact binomial probabilities shown in Figure 1

of their article are incorrect. The method used

by Wilson et al. (2012) returns one fewer than

the true critical number of experts required to

ensure agreement above that of chance for a

given value of α (Table 1) yet no mention of

this can be seen in their article. This can be

illustrated through an example using a panel

size of 15.

Example: Considering a panel size (n) of 15,

probability of success (p) of .5 and α = .05.

CRITBINOM (15, 0.5, 0.95) = 11

As this utilizes the cumulative binomial prob-

ability the interpretation of this result is “there

is at least a probability of 0.95 of getting 11 or

fewer successes.” Thus, there is at most a

probability of .05 of getting 12 or more suc-

cesses. This is the critical number we are

interested in to assure a level of agreement

above that of chance at α set at .05.

The error in calculating exact binomial

probabilities may explain why Wilson et al.

(2012) failed to realize that Shipper had calcu-

lated exact binomial probabilities up to a

panel size of 10.

CVRcritical Based on the Normal Approximation to

the Binomial. CVRcritical values reported by

Wilson et al. (2012) based on a normal approx-

imation to the binomial distribution are mark-

edly lower than those we have calculated using

a continuity correction for all given panel sizes

(see Figure 1). It is clear from their formula for

calculating the critical value for CVR that a

continuity correction was not used by Wilson

et al. Conversely, the values given in Lawshe

are consistent with the normal approximation

using the continuity correction and are there-

fore closer to the exact binomial probabilities

we have reported in this article. On this basis,

we believe that the recalculated values for

CVRcritical reported in Wilson et al. (2012) are

inaccurate and therefore should not be used.

Wilson et al. (2012) and Lawshe (1975)

have both calculated CVRcritical values for

panel sizes of 10 or more (Wilson et al., 2012,

used the normal approximation for all panel

sizes) based on a normal approximation of the

binomial distribution. We believe this is an

inferior method to the exact calculations we

have reported for the following reasons:

1. If the normal approximation value for

CVRcritical is higher than that produced

from exact calculations of binomial

probability the panel size deemed nec-

essary will be higher than required.

2. If the normal approximation value for

CVRcritical is lower than that produced

from exact calculations of binomial

probability the panel size deemed nec-

essary may be lower than required.

by guest on July 6, 2015mec.sagepub.comDownloaded from

Ayre and Scally 85

Table 2. Simplified Table of CVRcritical Including the Number of Experts Required to Agree an Item

Essential.

Panel Size

Ncritical (Minimum Number of

Experts Required to Agree an

Item Essential for Inclusion)

Proportion

Agreeing

Essential CVRcritical

5 5 1 1.00

6 6 1 1.00

7 7 1 1.00

8 7 .875 .750

9 8 .889 .778

10 9 .900 .800

11 9 .818 .636

12 10 .833 .667

13 10 .769 .538

14 11 .786 .571

15 12 .800 .600

16 12 .750 .500

17 13 .765 .529

18 13 .722 .444

19 14 .737 .474

20 15 .750 .500

21 15 .714 .429

22 16 .727 .455

23 16 .696 .391

24 17 .708 .417

25 18 .720 .440

26 18 .692 .385

27 19 .704 .407

28 19 .679 .357

29 20 .690 .379

30 20 .667 .333

31 21 .677 .355

32 22 .688 .375

33 22 .667 .333

34 23 .676 .353

35 23 .657 .314

36 24 .667 .333

37 24 .649 .297

38 25 .658 .316

39 26 .667 .333

40 26 .650 .300

Presented above is a simplified table of CVR-

critical values, calculated using exact binomial

probabilities, which includes the number of

experts required to agree any given item is

essential (Table 2).

It can be seen from Table 2 that preferred

panel sizes exist, when the addition of a

further panel member leads to a significant

reduction in the required proportion level of

agreement that an item is “essential” for it to

be included (e.g., between panel sizes of 12

and 13). In addition, it is also immediately

apparent that increasing the panel size by 1

will actually increase the required proportion

level of agreement on occasions (e.g.,

between panel sizes of 13 and 14). We believe

by guest on July 6, 2015mec.sagepub.comDownloaded from

86 Measurement and Evaluation in Counseling and Development 47(1)

this table is of most use to researchers’ wish-

ing to quantify content validity using the

CVR method, both to decide the most appro-

priate panel size and when determining

whether a critical level of agreement has

been reached.

Conclusions

The method used by Schipper to calculate

the original critical values reported in Law-

she’s article has been suggested and we have

been able to successfully reproduce the val-

ues using discrete calculation for panel sizes

below 10 and normal approximation to the

binomial for panel sizes of 10 and above.

We have identified problems with both the

discrete calculations and normal approxima-

tion to the binomial suggested by Wilson et al.

Consequently, we do not believe that values

for CVRcritical reported in Wilson et al. should

be used to determine whether a critical level

of agreement has been reached and therefore

whether items should be included or

excluded from a given instrument. Although

it is safe to use the values for CVRcritical pro-

posed by Lawshe to determine whether

items should be included on an instrument,

we believe that exact CVRcritical based

on discrete binomial calculations is most

appropriate.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of

interest with respect to the research, authorship,

and/or publication of this article.

Funding

The author(s) received no financial support for the

research, authorship, and/or publication of this article.

References

Armitage, P., Berry, G., & Matthews, J. N. S.

(2002). Statistical methods in medical research

(4th ed.). Oxford, England: Blackwell.

Frank-Stromberg, M., & Olsen, S. J. (2004).

Instruments for clinical health-care research.

London, England: Jones & Bartlett.

Gallin, J. I., & Ognibene, F. P. (2007). Principles

and practice of clinical research (2nd ed.).

Boston, MA: Elsevier.

Lawshe, C. H. (1975). A quantitative approach to con-

tent validity. Personnel psychology, 28, 563–575.

Rumsey, D. (2006). Probability for dummies.

Indianapolis, IN: Wiley.

Wilson, F. R., Pan, W., & Schumsky, D. A. (2012).

Recalculation of the critical values for Lawshe’s

content validity ratio. Measurement and

Evaluation in Counseling and Development,

45, 197–210. doi:10.1177/0748175612440286

Author Biographies

Colin Ayre is a PhD student and honorary lecturer

in the School of Health Studies, University of

Bradford. He is also a qualified physiotherapist

working in Bradford Teaching Hospitals NHS

Foundation Trust.

Andrew John Scally is a senior lecturer in the

School of Health Studies, University of Bradford. He

is a qualified radiographer, a graduate physicist and

has Master’s degrees in computational physics and

medical statistics. His current professional role com-

bines work in both radiation physics/radiological

protection and the application of medical statistics in

a wide range of medical and health disciplines.

by guest on July 6, 2015mec.sagepub.comDownloaded from