Page 1

SPECIAL GUEST EDITOR SECTION

Measurement Uncertainty in Microbiology

LYNNE I. FORSTER

Lynne I. Forster Training & Consulting Services, PO Box 15847, New Lynn, Auckland, New Zealand

Testing laboratories wishing to comply with the re-

quirements of ISO/IEC 17025:1999 need to estimate

uncertainty of measurement for their quantitative

methods. Many microbiological laboratories have

had procedures available for monitoring variability

in duplicate results generated by laboratory ana-

lysts for some time. These procedures, however,

do not necessarily include all possible contribu-

tions to uncertainty in the calculations. Procedures

for estimating microbiological method uncertainty,

based on the Poisson distribution, have been pub-

lished but, at times, the procedures can either un-

derestimate uncertainty or require laboratories to

undertake considerable experimental studies and

more complex statistical calculations. This paper

proposes procedures for estimating uncertainty of

measurement in microbiology, whereby routine

laboratory quality control data can be analyzed

with simple statistical equations. The approaches

used in these procedures are also applied to pub-

lished data and examples, demonstrating that es-

sentially equivalent results can be obtained with

these procedures.

A

rameter associated with the result of a measurement that char-

acterizes the dispersion of the values that could reasonably be

attributedtothemeasurand”(theparticularquantitysubjectto

measurement; 1).

In1992,atitsconferenceheldinOttawa,Canada,theInter-

national Laboratory Accreditation Conference (ILAC; now

International Laboratory Accreditation Cooperation) sug-

gested that a series of working groups be established to con-

siderproceduresforestimatinguncertaintyofmeasurementin

different testing disciplines, one of which was microbiology.

At the following ILAC conference held in Hong Kong in

1994, the working group concerned summarized its findings

asfollows:“Itisvirtuallyimpossibletoknowtheexactmicro-

bial concentration of any sample, natural or artificial. In cer-

tain circumstances, assigned values based on consensus may

s is well understood, no measurement is perfect. It has

an associated uncertainty arising from many factors.

Measurement uncertainty is defined as being “a pa-

be used but certified reference materials for running as con-

trols alongside tests are not generally available and where

these are available, it will be unlikely that they will be matrix

matched ...” (2).

During the past few years, considerable attention has been

paidtotheestimationofmeasurementuncertaintyinmicrobi-

ology and how this can be undertaken. ISO/IEC 17025:1999

requires that a “laboratory shall at least attempt to identify all

the components of uncertainty and make a reasonable estima-

tion ...” (3).

Few suggestions have yet been made on how bias may be

determined for microbiological analyses. The ILAC working

group suggested that comparison with a reference procedure

maynotbesufficienttodeterminebias,asthereferencemethod

may have its own bias, which may not easily be determined.

The working group also stated that interlaboratory comparison

programs and proficiency testing schemes may not give infor-

mationaboutbiasinabsolutetermsbutcouldbeusefulindeter-

mining the most likely number of organisms in a sample using

particular methodologies. When uncertainties are estimated,

those that cannot be evaluated statistically can be estimated

only through a thorough knowledge of all the steps in the mea-

surement process. The working group believed that quantifica-

tion of at least some of these might well be possible, but the

workload involved may well be prohibitive (2).

Formanyyears,variousmicrobiologicaltextsandpublica-

tions have included procedures for determining the precision

criterion in performing microbiological analyses (4, 5). This

involves laboratory analysts performing sample analyses in

duplicate. The sets of duplicate results are collected and

logarithmicallytransformed.Therangebetweeneachofthese

transformed duplicate results is calculated and the average

range determined and multiplied by 3.27 to give the precision

criterion.Itcanbeusedasthebenchmarkforassessingranges

in subsequent transformed results. This has been described as

either measuring analyst precision (4) or the precision of

quantitative methods (5).

A number of more recent approaches for estimating uncer-

tainty are based on the concept of the Poisson distribu-

tion (6–8). A Poisson distribution is defined as being “a fully

randomdistributionofparticlenumberswhensamplingaper-

fectly mixed suspension” (6). It is unique in that the standard

deviation is equal to the square root of the mean of the counts

obtained,i.e.,thevarianceisnumericallyequaltothemean.In

other words, the precision of a colony count method is gov-

erned by the magnitude of the count itself. Equations for cal-

culating the confidence interval associated with a count at a

FORSTER: JOURNAL OF AOAC INTERNATIONAL VOL. 86, NO. 5, 2003

1089

Guest edited as a special report on “Uncertainty of Measurement in

Chemical and Microbiological Testing” by John L. Love.

Corresponding author’s e-mail: lforster@clear.net.nz.

Page 2

95% confidence level, based on the Poisson distribution, are

included in the publications described above.

In simple equations, the uncertainty associated with a count

depends primarily on the total colony count, dilutions, and the

number of replicate plates. All contributions to uncertainty are

therefore not necessarily included in the estimation. Various

publications have recognized that when samples are analyzed

inreplicate,variabilityisgreaterthanfullyrandom(inthePois-

son sense), i.e., overdispersion may be observed (6, 7). Pure

cultures of bacteria can be expected to follow a Poisson series,

butmixedculturesmaydeviatefromPoisson,especiallywhen

sublethal cell damage has occurred (9).

ISO/TR 13843 includes a worked example in which the

overdispersion factor is estimated (6; Annex B). In this exam-

ple, results from 12 laboratories are used to demonstrate the

calculations involved. Each laboratory analyzed a sample of

thesametype,butofitsownchoice,andperformedquadrupli-

cate parallel counts on its homogenized sample suspensions.

All laboratories used the same method. The results obtained

were used to calculate the overdispersion factor by a number

of statistical techniques including linear regression.

ThedatafromISO/TR13843canbedevelopedfurtherand

used as an example of the additional calculations needed to

determine the confidence interval for microbiological counts

onaparticulartypeofsampleata95%confidencelevelusing

an overdispersion model. In this calculation:

SC u C

=+

22

where S is the standard deviation, C is the colony count, and u

is the overdispersion factor, calculated from the slope of the

line relating the variance-to-mean ratio to concentration. In

the ISO/TR 13843 example, u = 0.088.

At a 95% confidence interval, the count

???

2

22

C C + u C

If C = 105, the 95% confidence interval

????

1052105 + (0.00766 1052)

= 105 ????? i.e., 77 – 133

Other approaches are available for estimating u (7).

A laboratory may use the ISO/TR 13843 procedure to esti-

mate the uncertainty or the confidence interval for each of its

methods and sample types by undertaking a series of replicate

determinations. ISO/TR 13843 suggests replicates be in ex-

cess of quadruplicate for each sample and that more than 12

samples may be desirable for reliable results.

Manylaboratorieslacktheresourcesofstaffnumbersand

the time required to accumulate sufficient data for the

ISO/TR 13843 approach. They will, however, be conversant

with the concept of precision criterion, and a similar approach

for estimating uncertainty from duplicate data could therefore

be more readily adopted.

1090

FORSTER:JOURNAL OF AOAC INTERNATIONAL VOL. 86, NO. 5, 2003

Table 1.Calculation of intermediate precision for Heterotrophic Plate Counts of clean watersa

No. Result 1Result 2

LogR1

LogR2

(LogR1– LogR2) (LogR1? LogR2)2

1 1121272.0492 2.10380.05460.002981

2 37391.56821.59110.02290.022900

3 26231.41501.36170.05330.002841

4 35 371.54411.5682 0.02410.000581

5 75 59 1.87511.77080.10430.010878

6 21231.3222 1.36170.03950.001560

7 2292202.3598 2.34240.0174 0.000303

8161147 2.20682.1673 0.03950.001560

9 102 892.00861.94940.0592 0.003505

10 98 1071.9912 2.02940.0382 0.001459

1153 491.7243 1.69020.0341 0.001163

12 217223 2.3364 2.34830.0119 0.000142

1372 48 1.85731.6812 0.17610.031011

1430271.4771 1.43140.04570.002088

15 217199 2.33642.29880.0376 0.001414

161302102.1139 2.3222 0.20830.043389

Sum 0.127775

aData provided by Palmerston North City Council (New Zealand).

Page 3

The EURACHEM/CITAC Approach

In 1995, the 1st Ed. of the EURACHEM/CITAC publica-

tion Quantifying Uncertainty in Analytical Measurement was

published; the 2nd Ed. was published in 2000. This protocol

establishes general rules for the evaluation and expression of

uncertaintyinquantitativechemicalanalysis,basedontheap-

proach laid down in the ISO Guide to the Expression of Un-

certainty in Measurement. In the evaluation of the measure-

ment uncertainty of a method, the EURACHEM guide

requires the analyst to look closely at all the possible sources

of uncertainty within a method and states that “in practice, a

preliminary study will quickly identify the most significant

sources of uncertainty” which will be the dominating influ-

ences in the total uncertainty of the method.

Many of the following concepts and procedures in this

EURACHEM guide apply equally well to microbiological

testing:

(a) Specifying clearly what is being measured or specify-

ing the measurand

(b) Identifying contributions to uncertainty in the method

concerned

(c) Estimating the size of each identified contribution to

uncertainty as a standard deviation

(d) If necessary, combining the values obtained for uncer-

tainties

(e) Calculating the expanded uncertainty

An important point is made in the EURACHEM guide re-

gardingempiricalmethods.Insuchmethods,theanalyticalre-

sults obtained are dependent on the procedures used in the

analysis. The method accordingly defines the measurand or,

inotherwords,the“right”answerisnotapropertyofthesam-

pleorofthetargetorganisms,butofthemethod.Wheresucha

methodisinusewithinitsdefinedfieldofapplication,thebias

associated with the method is defined as being zero. That is, it

is not meaningful to consider correction for bias intrinsic to

these methods (10).

It may well be that the majority of quantitative microbio-

logical methods can be considered to be empirical methods,

where results generated are dependent on the media used,

times and temperatures of incubation, and inclusion or exclu-

sionofresuscitativestepsinthemethods.Variationsinthere-

covery of organisms resulting from the above factors have

been well documented over the years (5, 9).

Each of the above steps specified in the EURACHEM

guide for estimating uncertainty can be considered in turn.

Specification

In this step, what is being measured is clearly defined. The

equation used to calculate the value of the measurand at the

endofthemethodprocessisagoodstartingpoint.Inmicrobi-

ology,verysimpleequationsareusuallyinvolvedinthecalcu-

lation of colony-forming units (CFUs) or specific organisms

in a sample. These equations normally take into account the

average of duplicate results, the dilution used, and the volume

of the inoculum.

Specification can also include an overview or flowchart of

the steps undertaken in the performance of the method. Ini-

tially, a laboratory may wish to consider contributions to un-

certainty from the subsampling stage, which is normally a

characteristic of a test method.

Identification of Sources of Uncertainty

On the whole, general quantitative microbiological analy-

ses are very straightforward, most being based on the same

generalprinciples,i.e.,subsampling,dilution,plating,incuba-

tion, and counting (with, on occasion, confirmation of the

identity of organisms).

The EURACHEM guide recommends the use of “cause

and effect” diagrams for identifying contributions to uncer-

tainty.Agoodstartingpointintheconstructionofacauseand

FORSTER: JOURNAL OF AOAC INTERNATIONAL VOL. 86, NO. 5, 2003

1091

Table 2.Transformation of ISO/TR 13843 data into logarithms10and estimation of confidence intervals

No. Log R1

Log R2

Log R3

Log R4

MeanVariance Si2

12.29672.36742.3384 2.4048 2.3518 0.002089333

2 2.19032.16142.1761 2.11732.16130.000998667

3 1.76341.72431.8062 1.81951.7784 0.001871333

41.56821.62321.57981.49141.5656 0.003011000

52.09342.0253 1.9638 2.06822.0377 0.003215667

61.4472 1.23041.04141.30101.2550 0.028429000

7 2.22272.37662.3284 2.31392.31040.004136333

8 1.0000 1.0792 1.11390.9031 1.02400.008774000

9 1.81951.9243 1.97311.8512 1.89200.004847000

100.90311.1139 0.84510.69900.89030.029600000

112.30962.2695 2.35222.3344 2.31640.001284000

12 2.2095 2.14922.22012.2988 2.21940.003776667

Total 0.092033000

Page 4

effectdiagramistheequationusedtocalculatethemeasurand

(i.e., CFUs or organisms being analyzed for) and consider-

ation of each step of the analysis depicted in the flowchart. In

this way, a clear picture of all potentially significant contribu-

tions to uncertainty is obtained.

Quantification of Contributions to Uncertainty

The contributions to uncertainty that have been identified

above are usually examined to see which are accounted for by

data already available in the laboratory. These contributions

canthenbegroupedtogetherinthecauseandeffectdiagram.

Not all identified contributions to uncertainty will make a

significant contribution to the total uncertainty. Unless there

isalargenumberofthem,contributionsthatarelessthanone

third of the largest need not be quantified in detail. In micro-

biology, we can expect that the precision of the method itself

formsthedominantcontributiontotheuncertaintyestimate.

In microbiology, it is usually possible to quantify the com-

bined effect of most (if not all) sources of uncertainty, reduc-

ingtheoveralleffortinvolved.Microbiologicaltestinglabora-

tories generally have a program whereby a certain number of

samples are analyzed at least in duplicate. Duplicate data (for

a particular test and for particular types of samples) collected

over a period of time can be analyzed to determine the stan-

dard deviation. In ISO 5725, this is called the intermediate

precisionofthetestmethod,ifallpossiblesourcesofvariation

in the method are taken into account when duplicate analyses

areperformed(11).Thesesourcesofvariationincludestorage

effects, laboratory environmental effects, operator effects, ef-

fects of using different items of equipment, different batches

of media, etc.

At this stage, publications suggest that estimations of un-

certaintycommencefromthesubsamplingstageofananaly-

sis, i.e., from the time samples are received in the laboratory

for testing. Subsampling procedures are normally included

in a test method, whereas external sampling is not.

The standard deviation or intermediate precision of a series

of duplicate results, for a particular sample type, is calculated

by:

??

S

t

=

12

= 1

yy

ii

i

?

t

?

?

??

?

?

?

?

?

??

?

?

?

?

2

1 2

/

2

where(yi1–yi2)isthedifferencebetweenindividualduplicate

results; ?(yi1– yi2)2is the sum of the squared differences be-

tween each set of duplicate results; and t is the number of test

samplesanalyzed(notethatotherpublicationsmayusediffer-

ent symbols from those in ISO 5725).

Microbial distributions are not necessarily symmetrical, as

countsareoftencharacterizedashavingaskeweddistribution

becauseofmanylowvaluesandafewhighones.Theapplica-

tion of parametric statistical techniques generally includes the

assumptionthatthedatabeinganalyzedisfromasymmetrical

or normal distribution. In microbiology, a common practice is

to first transform data into logarithms10before equations for

calculatingstandarddeviation,etc.,areapplied(4,5,9,12–14).

An example of the calculation of intermediate precision from

a series of duplicate Heterotrophic Plate Count results for

clean water is included in Table 1. In Table 2, the data in-

cludedinISO/TR13843arereproducedandconfidenceinter-

valsrecalculatedafterthequadruplicatedatafromthe12labo-

ratories are transformed into logarithms10. The result that the

95% confidence interval is 70–157 for a mean result of 105

canbecomparedwiththeconfidenceintervalof77–133calcu-

latedwiththesesamedata,butbytheISO/TR13843procedure.

Using the data in Table 1:

??

S

t

2

2

=

LogRLogR

2

= 0.127775/ 32

12

?

?

S = 0.0632

2S = 0.1264

For sample No. 1, the mean of the results = (2.0492 +

2.1038)/2 = 2.0765. At a 95% confidence level, the confidence

interval = 2.0765 ± 0.1264 = 1.9501–2.2029. On antilogging,

for a result of 120, the confidence interval is 90–160.

Combination of Uncertainty Values

It can be expected that all major contributions to uncer-

tainty are accounted for in calculating intermediate precision.

Onoccasion,minorcontributionstouncertaintymaybequan-

tified separately as standard deviations and, if relevant, com-

bined with the intermediate precision value.

1092

FORSTER:JOURNAL OF AOAC INTERNATIONAL VOL. 86, NO. 5, 2003

Table 3.Fifteen replicate results for fecal coliforms (MPN) and estimation of the confidence intervala

Operator 1Log resultOperator 2 Log resultOperator 3 Log result

80000 4.90311700005.230480000 4.9031

500004.699050000 4.699080000 4.9031

90000 4.95421700005.2304 2200005.3424

80000 4.9031 1300005.113980000 4.9031

1100005.041480000 4.903180000 4.9031

aData supplied by Watercare Laboratory Services (Auckland, New Zealand).

Page 5

Calculation of the Expanded Uncertainty

The final stage is to multiply the (combined) standard un-

certainty by a chosen coverage factor k, in order to obtain an

expanded uncertainty.

TheexpandeduncertaintyUisrequiredtoprovideaninter-

val which may be expected to encompass a large fraction of

thedistributionofvalueswhichcouldreasonablybeattributed

tothemeasurand,i.e.,anintervalwithinwhichthevalueofthe

measurand is believed to lie, with a high level of confidence.

For most purposes, a coverage factor of 2 is chosen (confi-

dence level of approximately 95%). Note that the value ob-

tainedisstillexpressedasalogarithm10andmustbeconverted

to give the actual range.

Using the data in Table 2:

S

Nk

S

2=

1

?

?

??

i

()

nii

k

1

2

1

S2= (3 ? 0.092033)/(48 – 12), S = 0.0876, 2S = 0.1752

whereS=overallstandarddeviationforthemethod;Si=sam-

ple standard deviation of the replicates for the ith sample, us-

ing (n – 1) as the denominator; ni= number of replicates for

the ith sample; N = total number of analyses (number of sam-

ples ? number of replicates); k = total number of samples.

For a mean sample count of 105, the confidence interval is

2.0212 ? 0.1752, which is 1.8460–2.1964, which is 70–157.

Most Probable Number (MPN) Determinations

It is traditional in MPN analyses to refer to MPN tables to

obtain a test result and the associated 95% confidence limits.

These have been established statistically, assuming that mi-

croorganisms are distributed in accordance with the Poisson

distribution. That is, complete randomness of particle distri-

bution in a liquid medium is assumed. MPN tables, however,

may not necessarily include all contributions to uncertainty.

Laboratories should therefore establish if the confidence

limits quoted in MPN tables are reasonable estimates of un-

certainty for their circumstances. One way of doing this is to

establish if replicate determinations indicate a similar level of

uncertaintyasindicatedintherelevantMPNtable.Aseriesof

duplicate results can be analyzed and the intermediate preci-

sion calculated as described previously or another approach

can be used, as below.

InTable3,theresultsfrom3operatorsinalaboratory,each

analyzing, in quintuplicate, an effluent sample for fecal

coliforms, are analyzed statistically. A plot of the results after

transformation on normal probability paper (Figure 1) sug-

gests that these transformed data are consistent with a normal

distribution (15). The similarity of this calculated uncertainty

to that in the relevant MPN table would indicate that no other

significant sources of uncertainty are present and that, with

subsequent analyses, a laboratory can use the confidence lim-

its specified in the MPN tables.

UsingthedatainTable3:mean=4.9755(ca95 000);stan-

darddeviation=0.1848.Ata95%confidencelevel,theresult

is 4.9755 ? 2 ? 0.1848 = 4.6059–5.3451; i.e., 40 000–220 000

(taking antilogs).

Foraresultof90 000,the95%confidenceintervalinMPN

tables is approximately 30 000–290 000.

Reporting Uncertainty of Measurement

Traditionally, results from microbiological analyses are

presented unaccompanied by any form of uncertainty estima-

tion. This situation, however, may change in the future. The

information could be reported as a confidence interval or as

confidence limits, once the expanded uncertainty data have

beenantilogged(9).Forexample:Result:x(units)withacon-

fidenceintervalofytoz,or,x(units)withconfidencelimitsof

y and z.

Conclusions

Two procedures for estimating uncertainty of measure-

ment in microbiology are presented in this paper. In these ap-

proaches, laboratory quality control results for both

Heterotrophic Plate Count and MPN determinations are ana-

lyzed statistically, using simple standard deviation and inter-

mediate precision equations.

Inthese2procedures,resultsarefirsttransformedintolog-

arithms10. Figure 1 shows a plot of MPN results from Table 3

on normal probability paper, suggesting that these trans-

formed data are consistent with the assumption of normality.

With general quantitative methods, intermediate precision

can be calculated from a series of duplicate results (not less

than 15) when all possible method variations are taken into ac-

countinperformingtheduplicates.InthecaseofMPNdetermi-

nations, if a laboratory can demonstrate that its estimated mea-

surement uncertainty is within the confidence limits published

in MPN tables, the appropriate tabulated values can be quoted

with subsequent analytical results, as this observation suggests

no other significant sources of uncertainty are present.

FORSTER: JOURNAL OF AOAC INTERNATIONAL VOL. 86, NO. 5, 2003

1093

Figure 1.

fecal coliforms (MPN).

Log normal plot of 15 replicate results for