Measurement uncertainty in microbiology.
ABSTRACT Testing laboratories wishing to comply with the requirements of ISO/IEC 17025:1999 need to estimate uncertainty of measurement for their quantitative methods. Many microbiological laboratories have had procedures available for monitoring variability in duplicate results generated by laboratory analysts for some time. These procedures, however, do not necessarily include all possible contributions to uncertainty in the calculations. Procedures for estimating microbiological method uncertainty, based on the Poisson distribution, have been published but, at times, the procedures can either underestimate uncertainty or require laboratories to undertake considerable experimental studies and more complex statistical calculations. This paper proposes procedures for estimating uncertainty of measurement in microbiology, whereby routine laboratory quality control data can be analyzed with simple statistical equations. The approaches used in these procedures are also applied to published data and examples, demonstrating that essentially equivalent results can be obtained with these procedures.
- [show abstract] [hide abstract]
ABSTRACT: The development of effective vaccines against zoonotic pathogens represents a priority in public health protection programs. The design of clinical trials and appropriate data analysis of the experiments results are crucial for the assessment of vaccine effectiveness. This manuscript reviews important issues related to the assessment of the effectiveness of vaccines designed to obtain a quantitative reduction of the pathogen in animals or animal products. An effective vaccine will reduce the risk of human infections and therefore the number of human cases. Important considerations will be illustrated using a vaccination trial of a new campylobacter vaccine candidate developed to reduce the numbers of campylobacter in chickens and consequently the numbers of human campylobacteriosis cases. The design of the author's vaccination trial was based on the use of isolators, a parallel group design and several rotations. The effect of clustering or design effect was considered in the sample size calculations. Chickens were randomly assigned to different isolators (treatments) and challenged with Campylobacter jejuni. Samples were obtained at different intervals and processed in the laboratory. C. jejuni counts were determined as colony-forming unit-per-gram of chicken cecum or fecal mass in order to assess vaccine effectiveness. A desired vaccine effect of 2 logs reduction on the numbers of C. jejuni recovered from vaccinated chickens was selected. Sample-size calculations, desired vaccine effect, biological and epidemiological aspects, experimental design and appropriate statistical analysis of data considering group or clustering effects will be the focus of this manuscript.Expert Review of Vaccines 10/2012; 11(10):1179-88. · 4.22 Impact Factor
- [show abstract] [hide abstract]
ABSTRACT: Quantitative real-time PCR is one of the newer methods for measurement of the amount of nucleic material in biological systems. However, reliable measurement requires an appropriate estimation of uncertainty and this paper has developed the uncertainty budget associated with this procedure using as an example, data from a quantitative real-time PCR method for the enumeration of Campylobacter jejuni. This uncertainty is relatively large and for instance, a measured result of 151 units of DNA would have a 95% confidence interval of +/-84 units of DNA with the main sources of uncertainty being the measurement of the threshold cycle (Ct) value, the predicted DNA content of the unknown sample from the calibration line and the molar absorbance value for DNA.Journal of Microbiological Methods 12/2006; 67(2):349-56. · 2.16 Impact Factor
SPECIAL GUEST EDITOR SECTION
Measurement Uncertainty in Microbiology
LYNNE I. FORSTER
Lynne I. Forster Training & Consulting Services, PO Box 15847, New Lynn, Auckland, New Zealand
Testing laboratories wishing to comply with the re-
quirements of ISO/IEC 17025:1999 need to estimate
uncertainty of measurement for their quantitative
methods. Many microbiological laboratories have
had procedures available for monitoring variability
in duplicate results generated by laboratory ana-
lysts for some time. These procedures, however,
do not necessarily include all possible contribu-
tions to uncertainty in the calculations. Procedures
for estimating microbiological method uncertainty,
based on the Poisson distribution, have been pub-
lished but, at times, the procedures can either un-
derestimate uncertainty or require laboratories to
undertake considerable experimental studies and
more complex statistical calculations. This paper
proposes procedures for estimating uncertainty of
measurement in microbiology, whereby routine
laboratory quality control data can be analyzed
with simple statistical equations. The approaches
used in these procedures are also applied to pub-
lished data and examples, demonstrating that es-
sentially equivalent results can be obtained with
rameter associated with the result of a measurement that char-
acterizes the dispersion of the values that could reasonably be
national Laboratory Accreditation Conference (ILAC; now
International Laboratory Accreditation Cooperation) sug-
gested that a series of working groups be established to con-
different testing disciplines, one of which was microbiology.
At the following ILAC conference held in Hong Kong in
1994, the working group concerned summarized its findings
bial concentration of any sample, natural or artificial. In cer-
tain circumstances, assigned values based on consensus may
s is well understood, no measurement is perfect. It has
an associated uncertainty arising from many factors.
Measurement uncertainty is defined as being “a pa-
be used but certified reference materials for running as con-
trols alongside tests are not generally available and where
these are available, it will be unlikely that they will be matrix
matched ...” (2).
During the past few years, considerable attention has been
ology and how this can be undertaken. ISO/IEC 17025:1999
requires that a “laboratory shall at least attempt to identify all
the components of uncertainty and make a reasonable estima-
tion ...” (3).
Few suggestions have yet been made on how bias may be
determined for microbiological analyses. The ILAC working
group suggested that comparison with a reference procedure
may have its own bias, which may not easily be determined.
The working group also stated that interlaboratory comparison
programs and proficiency testing schemes may not give infor-
mining the most likely number of organisms in a sample using
particular methodologies. When uncertainties are estimated,
those that cannot be evaluated statistically can be estimated
only through a thorough knowledge of all the steps in the mea-
surement process. The working group believed that quantifica-
tion of at least some of these might well be possible, but the
workload involved may well be prohibitive (2).
tions have included procedures for determining the precision
criterion in performing microbiological analyses (4, 5). This
involves laboratory analysts performing sample analyses in
duplicate. The sets of duplicate results are collected and
transformed duplicate results is calculated and the average
range determined and multiplied by 3.27 to give the precision
in subsequent transformed results. This has been described as
either measuring analyst precision (4) or the precision of
quantitative methods (5).
A number of more recent approaches for estimating uncer-
tainty are based on the concept of the Poisson distribu-
tion (6–8). A Poisson distribution is defined as being “a fully
fectly mixed suspension” (6). It is unique in that the standard
deviation is equal to the square root of the mean of the counts
other words, the precision of a colony count method is gov-
erned by the magnitude of the count itself. Equations for cal-
culating the confidence interval associated with a count at a
FORSTER: JOURNAL OF AOAC INTERNATIONAL VOL. 86, NO. 5, 2003
Guest edited as a special report on “Uncertainty of Measurement in
Chemical and Microbiological Testing” by John L. Love.
Corresponding author’s e-mail: firstname.lastname@example.org.
95% confidence level, based on the Poisson distribution, are
included in the publications described above.
In simple equations, the uncertainty associated with a count
depends primarily on the total colony count, dilutions, and the
number of replicate plates. All contributions to uncertainty are
therefore not necessarily included in the estimation. Various
publications have recognized that when samples are analyzed
son sense), i.e., overdispersion may be observed (6, 7). Pure
cultures of bacteria can be expected to follow a Poisson series,
sublethal cell damage has occurred (9).
ISO/TR 13843 includes a worked example in which the
overdispersion factor is estimated (6; Annex B). In this exam-
ple, results from 12 laboratories are used to demonstrate the
calculations involved. Each laboratory analyzed a sample of
cate parallel counts on its homogenized sample suspensions.
All laboratories used the same method. The results obtained
were used to calculate the overdispersion factor by a number
of statistical techniques including linear regression.
used as an example of the additional calculations needed to
determine the confidence interval for microbiological counts
an overdispersion model. In this calculation:
SC u C
where S is the standard deviation, C is the colony count, and u
is the overdispersion factor, calculated from the slope of the
line relating the variance-to-mean ratio to concentration. In
the ISO/TR 13843 example, u = 0.088.
At a 95% confidence interval, the count
C C + u C
If C = 105, the 95% confidence interval
1052105 + (0.00766 1052)
= 105 ????? i.e., 77 – 133
Other approaches are available for estimating u (7).
A laboratory may use the ISO/TR 13843 procedure to esti-
mate the uncertainty or the confidence interval for each of its
methods and sample types by undertaking a series of replicate
determinations. ISO/TR 13843 suggests replicates be in ex-
cess of quadruplicate for each sample and that more than 12
samples may be desirable for reliable results.
the time required to accumulate sufficient data for the
ISO/TR 13843 approach. They will, however, be conversant
with the concept of precision criterion, and a similar approach
for estimating uncertainty from duplicate data could therefore
be more readily adopted.
FORSTER:JOURNAL OF AOAC INTERNATIONAL VOL. 86, NO. 5, 2003
Table 1.Calculation of intermediate precision for Heterotrophic Plate Counts of clean watersa
No. Result 1Result 2
(LogR1– LogR2) (LogR1? LogR2)2
1 1121272.0492 2.10380.05460.002981
4 35 371.54411.5682 0.02410.000581
5 75 59 1.87511.77080.10430.010878
6 21231.3222 1.36170.03950.001560
7 2292202.3598 2.34240.0174 0.000303
8161147 2.20682.1673 0.03950.001560
9 102 892.00861.94940.0592 0.003505
10 98 1071.9912 2.02940.0382 0.001459
1153 491.7243 1.69020.0341 0.001163
12 217223 2.3364 2.34830.0119 0.000142
1372 48 1.85731.6812 0.17610.031011
15 217199 2.33642.29880.0376 0.001414
161302102.1139 2.3222 0.20830.043389
aData provided by Palmerston North City Council (New Zealand).
The EURACHEM/CITAC Approach
In 1995, the 1st Ed. of the EURACHEM/CITAC publica-
tion Quantifying Uncertainty in Analytical Measurement was
published; the 2nd Ed. was published in 2000. This protocol
establishes general rules for the evaluation and expression of
proach laid down in the ISO Guide to the Expression of Un-
certainty in Measurement. In the evaluation of the measure-
ment uncertainty of a method, the EURACHEM guide
requires the analyst to look closely at all the possible sources
of uncertainty within a method and states that “in practice, a
preliminary study will quickly identify the most significant
sources of uncertainty” which will be the dominating influ-
ences in the total uncertainty of the method.
Many of the following concepts and procedures in this
EURACHEM guide apply equally well to microbiological
(a) Specifying clearly what is being measured or specify-
ing the measurand
(b) Identifying contributions to uncertainty in the method
(c) Estimating the size of each identified contribution to
uncertainty as a standard deviation
(d) If necessary, combining the values obtained for uncer-
(e) Calculating the expanded uncertainty
An important point is made in the EURACHEM guide re-
sults obtained are dependent on the procedures used in the
analysis. The method accordingly defines the measurand or,
associated with the method is defined as being zero. That is, it
is not meaningful to consider correction for bias intrinsic to
these methods (10).
It may well be that the majority of quantitative microbio-
logical methods can be considered to be empirical methods,
where results generated are dependent on the media used,
times and temperatures of incubation, and inclusion or exclu-
covery of organisms resulting from the above factors have
been well documented over the years (5, 9).
Each of the above steps specified in the EURACHEM
guide for estimating uncertainty can be considered in turn.
In this step, what is being measured is clearly defined. The
equation used to calculate the value of the measurand at the
lation of colony-forming units (CFUs) or specific organisms
in a sample. These equations normally take into account the
average of duplicate results, the dilution used, and the volume
of the inoculum.
Specification can also include an overview or flowchart of
the steps undertaken in the performance of the method. Ini-
tially, a laboratory may wish to consider contributions to un-
certainty from the subsampling stage, which is normally a
characteristic of a test method.
Identification of Sources of Uncertainty
On the whole, general quantitative microbiological analy-
ses are very straightforward, most being based on the same
tion, and counting (with, on occasion, confirmation of the
identity of organisms).
The EURACHEM guide recommends the use of “cause
and effect” diagrams for identifying contributions to uncer-
FORSTER: JOURNAL OF AOAC INTERNATIONAL VOL. 86, NO. 5, 2003
Table 2.Transformation of ISO/TR 13843 data into logarithms10and estimation of confidence intervals
No. Log R1
12.29672.36742.3384 2.4048 2.3518 0.002089333
2 2.19032.16142.1761 2.11732.16130.000998667
3 1.76341.72431.8062 1.81951.7784 0.001871333
52.09342.0253 1.9638 2.06822.0377 0.003215667
61.4472 1.23041.04141.30101.2550 0.028429000
7 2.22272.37662.3284 2.31392.31040.004136333
8 1.0000 1.0792 1.11390.9031 1.02400.008774000
9 1.81951.9243 1.97311.8512 1.89200.004847000
112.30962.2695 2.35222.3344 2.31640.001284000
12 2.2095 2.14922.22012.2988 2.21940.003776667
(i.e., CFUs or organisms being analyzed for) and consider-
ation of each step of the analysis depicted in the flowchart. In
this way, a clear picture of all potentially significant contribu-
tions to uncertainty is obtained.
Quantification of Contributions to Uncertainty
The contributions to uncertainty that have been identified
above are usually examined to see which are accounted for by
data already available in the laboratory. These contributions
Not all identified contributions to uncertainty will make a
significant contribution to the total uncertainty. Unless there
third of the largest need not be quantified in detail. In micro-
biology, we can expect that the precision of the method itself
In microbiology, it is usually possible to quantify the com-
bined effect of most (if not all) sources of uncertainty, reduc-
tories generally have a program whereby a certain number of
samples are analyzed at least in duplicate. Duplicate data (for
a particular test and for particular types of samples) collected
over a period of time can be analyzed to determine the stan-
dard deviation. In ISO 5725, this is called the intermediate
in the method are taken into account when duplicate analyses
effects, laboratory environmental effects, operator effects, ef-
fects of using different items of equipment, different batches
of media, etc.
At this stage, publications suggest that estimations of un-
sis, i.e., from the time samples are received in the laboratory
for testing. Subsampling procedures are normally included
in a test method, whereas external sampling is not.
The standard deviation or intermediate precision of a series
of duplicate results, for a particular sample type, is calculated
results; ?(yi1– yi2)2is the sum of the squared differences be-
tween each set of duplicate results; and t is the number of test
ent symbols from those in ISO 5725).
Microbial distributions are not necessarily symmetrical, as
tion of parametric statistical techniques generally includes the
or normal distribution. In microbiology, a common practice is
to first transform data into logarithms10before equations for
An example of the calculation of intermediate precision from
a series of duplicate Heterotrophic Plate Count results for
clean water is included in Table 1. In Table 2, the data in-
ratories are transformed into logarithms10. The result that the
95% confidence interval is 70–157 for a mean result of 105
Using the data in Table 1:
= 0.127775/ 32
S = 0.0632
2S = 0.1264
For sample No. 1, the mean of the results = (2.0492 +
2.1038)/2 = 2.0765. At a 95% confidence level, the confidence
interval = 2.0765 ± 0.1264 = 1.9501–2.2029. On antilogging,
for a result of 120, the confidence interval is 90–160.
Combination of Uncertainty Values
It can be expected that all major contributions to uncer-
tainty are accounted for in calculating intermediate precision.
tified separately as standard deviations and, if relevant, com-
bined with the intermediate precision value.
FORSTER:JOURNAL OF AOAC INTERNATIONAL VOL. 86, NO. 5, 2003
Table 3.Fifteen replicate results for fecal coliforms (MPN) and estimation of the confidence intervala
Operator 1Log resultOperator 2 Log resultOperator 3 Log result
80000 4.90311700005.230480000 4.9031
500004.699050000 4.699080000 4.9031
90000 4.95421700005.2304 2200005.3424
80000 4.9031 1300005.113980000 4.9031
1100005.041480000 4.903180000 4.9031
aData supplied by Watercare Laboratory Services (Auckland, New Zealand).
Calculation of the Expanded Uncertainty
The final stage is to multiply the (combined) standard un-
certainty by a chosen coverage factor k, in order to obtain an
val which may be expected to encompass a large fraction of
measurand is believed to lie, with a high level of confidence.
For most purposes, a coverage factor of 2 is chosen (confi-
dence level of approximately 95%). Note that the value ob-
to give the actual range.
Using the data in Table 2:
S2= (3 ? 0.092033)/(48 – 12), S = 0.0876, 2S = 0.1752
ple standard deviation of the replicates for the ith sample, us-
ing (n – 1) as the denominator; ni= number of replicates for
the ith sample; N = total number of analyses (number of sam-
ples ? number of replicates); k = total number of samples.
For a mean sample count of 105, the confidence interval is
2.0212 ? 0.1752, which is 1.8460–2.1964, which is 70–157.
Most Probable Number (MPN) Determinations
It is traditional in MPN analyses to refer to MPN tables to
obtain a test result and the associated 95% confidence limits.
These have been established statistically, assuming that mi-
croorganisms are distributed in accordance with the Poisson
distribution. That is, complete randomness of particle distri-
bution in a liquid medium is assumed. MPN tables, however,
may not necessarily include all contributions to uncertainty.
Laboratories should therefore establish if the confidence
limits quoted in MPN tables are reasonable estimates of un-
certainty for their circumstances. One way of doing this is to
establish if replicate determinations indicate a similar level of
duplicate results can be analyzed and the intermediate preci-
sion calculated as described previously or another approach
can be used, as below.
analyzing, in quintuplicate, an effluent sample for fecal
coliforms, are analyzed statistically. A plot of the results after
transformation on normal probability paper (Figure 1) sug-
gests that these transformed data are consistent with a normal
distribution (15). The similarity of this calculated uncertainty
to that in the relevant MPN table would indicate that no other
significant sources of uncertainty are present and that, with
subsequent analyses, a laboratory can use the confidence lim-
its specified in the MPN tables.
is 4.9755 ? 2 ? 0.1848 = 4.6059–5.3451; i.e., 40 000–220 000
tables is approximately 30 000–290 000.
Reporting Uncertainty of Measurement
Traditionally, results from microbiological analyses are
presented unaccompanied by any form of uncertainty estima-
tion. This situation, however, may change in the future. The
information could be reported as a confidence interval or as
confidence limits, once the expanded uncertainty data have
y and z.
Two procedures for estimating uncertainty of measure-
ment in microbiology are presented in this paper. In these ap-
proaches, laboratory quality control results for both
Heterotrophic Plate Count and MPN determinations are ana-
lyzed statistically, using simple standard deviation and inter-
mediate precision equations.
arithms10. Figure 1 shows a plot of MPN results from Table 3
on normal probability paper, suggesting that these trans-
formed data are consistent with the assumption of normality.
With general quantitative methods, intermediate precision
can be calculated from a series of duplicate results (not less
than 15) when all possible method variations are taken into ac-
nations, if a laboratory can demonstrate that its estimated mea-
surement uncertainty is within the confidence limits published
in MPN tables, the appropriate tabulated values can be quoted
with subsequent analytical results, as this observation suggests
no other significant sources of uncertainty are present.
FORSTER: JOURNAL OF AOAC INTERNATIONAL VOL. 86, NO. 5, 2003
fecal coliforms (MPN).
Log normal plot of 15 replicate results for