Accuracy of magnetic resonance imaging for the diagnosis of multiple sclerosis: systematic review.

Abstract

To determine the accuracy of magnetic resonance imaging criteria for the early diagnosis of multiple sclerosis in patients with suspected disease.
Systematic review.
12 electronic databases, citation searches, and reference lists of included studies. Review methods Studies on accuracy of diagnosis that compared magnetic resonance imaging, or diagnostic criteria incorporating such imaging, to a reference standard for the diagnosis of multiple sclerosis.
29 studies (18 cohort studies, 11 other designs) were included. On average, studies of other designs (mainly diagnostic case-control studies) produced higher estimated diagnostic odds ratios than did cohort studies. Among 15 studies of higher methodological quality (cohort design, clinical follow-up as reference standard), those with longer follow-up produced higher estimates of specificity and lower estimates of sensitivity. Only two such studies followed patients for more than 10 years. Even in the presence of many lesions (> 10 or > 8), magnetic resonance imaging could not accurately rule multiple sclerosis in (likelihood ratio of a positive test result 3.0 and 2.0, respectively). Similarly, the absence of lesions was of limited utility in ruling out a diagnosis of multiple sclerosis (likelihood ratio of a negative test result 0.1 and 0.5).
Many evaluations of the accuracy of magnetic resonance imaging for the early detection of multiple sclerosis have produced inflated estimates of test performance owing to methodological weaknesses. Use of magnetic resonance imaging to confirm multiple sclerosis on the basis of a single attack of neurological dysfunction may lead to over-diagnosis and over-treatment.

Full text

Page 1

Research
Accuracy of magnetic resonance imaging for the diagnosis of
multiple sclerosis: systematic review
Penny Whiting, Roger Harbord, Caroline Main, Jonathan J Deeks, Graziella Filippini, Matthias Egger, Jonathan A C
Sterne
Abstract
Objective To determine the accuracy of magnetic resonance
imaging criteria for the early diagnosis of multiple sclerosis in
patients with suspected disease.
Design Systematic review.
Data sources 12 electronic databases, citation searches, and
reference lists of included studies.
Review methods Studies on accuracy of diagnosis that
compared magnetic resonance imaging, or diagnostic criteria
incorporating such imaging, to a reference standard for the
diagnosis of multiple sclerosis.
Results 29 studies (18 cohort studies, 11 other designs) were
included. On average, studies of other designs (mainly
diagnostic case-control studies) produced higher estimated
diagnostic odds ratios than did cohort studies. Among 15
studies of higher methodological quality (cohort design, clinical
follow-up as reference standard), those with longer follow-up
produced higher estimates of specificity and lower estimates of
sensitivity. Only two such studies followed patients for more
than10 years. Even in the presence of many lesions ( > 10 or
> 8), magnetic resonance imaging could not accurately rule
multiple sclerosis in (likelihood ratio of a positive test result 3.0
and 2.0, respectively). Similarly, the absence of lesions was of
limited utility in ruling out a diagnosis of multiple sclerosis
(likelihood ratio of a negative test result 0.1 and 0.5).
Conclusions Many evaluations of the accuracy of magnetic
resonance imaging for the early detection of multiple sclerosis
have produced inflated estimates of test performance owing to
methodological weaknesses. Use of magnetic resonance
imaging to confirm multiple sclerosis on the basis of a single
attack of neurological dysfunction may lead to over-diagnosis
and over-treatment.
Introduction
Diagnosis of multiple sclerosis is based on the principle of
dissemination in both time and space. Recent criteria state that
patients shouldexperience
dysfunction, such as optic neuritis, transverse myelitis, double
vision,or numbness and tingling of the leg,occurring at different
points in time and affecting different parts of the central nervous
system—that is, signs or symptoms that cannot be attributable to
a single lesion.1Many years may elapse between first and second
attacks, and not all patients who experience a first attack develop
multiple sclerosis. In a study of patients with optic neuritis, a
common presenting symptom of multiple sclerosis, 38%
developed the disease by 10 years; of these, 50% received their
twoattacksof neurological
diagnosis more than three years after presentation and 28%
more than five years after presentation.2In a study of patients
presenting with clinically isolated syndromes (optic, spinal cord,
or brain symptoms) 68% of patients had developed multiple
sclerosis by 14 years, the proportions being similar for the differ-
ent presenting symptoms.3
Magnetic resonance imaging may assist in earlier diagnosis
of multiple sclerosis by enabling visualisation of lesions in the
brain that are clinically silent. The McDonald 2001 criteria for
the diagnosis of multiple sclerosis4allow an early diagnosis of
multiple sclerosis to be made after one clinical attack if the
patient also meets criteria for a positive result on a magnetic
resonance imaging scan. The McDonald criteria have been
adopted in England and Wales by the National Institute for
Health and Clinical Excellence (NICE),5but they are not univer-
sally accepted.1Evidence shows that patients’ wellbeing is
affected by early diagnosis,6–9usually in a beneficial way but also
occasionally in a negative way—for example, through increased
insurance premiums and discrimination in the workplace.10Ear-
lier diagnosis of multiple sclerosis could mean the availability of
earlier treatment, such as the disease modifying therapies
interferon beta and glatiramer acetate, provided under the “risk
sharing scheme” in the United Kingdom (www.dh.gov.uk/
assetRoot/04/01/22/14/04012214.pdf).
We carried out a systematic review to estimate the accuracy of
different magnetic resonance imaging criteria for the early diag-
nosis of multiple sclerosis in patients presenting with suspected
disease, to investigate whether magnetic resonance imaging has
the potential to alter diagnoses and patient management.
Methods
We identified studies, published and unpublished, by searching
12 databases from inception until September or November
2004. Search terms were “multiple sclerosis” combined with
“magneticresonanceimaging”
restrictions were applied. We undertook a citation search on the
articlereportingtheMcDonald
reference lists of included studies, and assessed studies included
in the NICE multiple sclerosis guidelines.5
Studies were eligible that compared magnetic resonance
imaging (or diagnostic criteria incorporating such imaging) to a
reference standard for the diagnosis of multiple sclerosis and
reported sufficient data to enable a 2×2 table of test
or“MRI”.Nolanguage
2001criteria,4
screened
Additional information and references w1-w43 are on bmj.com
Cite this article as: BMJ, doi:10.1136/bmj.38771.583796.7C (published 24 March 2006)
BMJ Online First bmj.com
page 1 of 10

Page 2

performance to be constructed. If studies were reported more
than once,we included the publication that provided data for the
longest follow-up. We also included separate publications that
reported on different criteria for magnetic resonance imaging or
separate results for relevant patient subgroups.
Two reviewers independently screened titles and abstracts for
relevance. Screening for inclusion, data extraction, and quality
assessment were carried out by one reviewer and checked by a
second. Studies were assessed for methodological quality against
the QUADAS (quality assessment of diagnostic accuracy studies)
criteria.11(See bmj.com for a summary of how items were
scored.) One item,the avoidance of disease progression bias,was
omitted as it was not relevant to this topic. We grouped studies
according to patient spectrum: prospective cohort studies that
enrolled patients with suspected multiple sclerosis,and studies of
other designs.
Data analysis
From each 2×2 table we computed sensitivity, specificity, and
likelihood ratios, which combine data on sensitivity and
specificity to give an indication of a test’s ability to rule in or rule
out a condition.12
We plotted all results from all included studies on a receiver
operating characteristic plot of sensitivity against specificity, with
the specificity axis reversed. To compare accuracy of cohort and
other studies we selected the result with the median diagnostic
odds ratio (defined as the odds of positivity among people with
the disease, divided by the odds of positivity among people with-
out the disease) for each study. We used random effects
meta-analysis to obtain summary diagnostic odds ratios in each
group,and we carried out a permutation test13to obtain a P value
for their comparison. We restricted all further analyses to cohort
studies that used a reference standard diagnosis of clinically defi-
nite multiple sclerosis, arrived at solely by clinical data.
As a final diagnosis of multiple sclerosis may be reached
many years after a patient first presents with possible disease, we
investigated the effect of duration of follow-up on estimates of
diagnostic accuracy. We used the hierarchical summary receiver
operating characteristic method proposed by Rutter and
Gatsonis14to assess the effect of duration of follow-up on overall
accuracy and threshold. An association with threshold would
indicate that sensitivity increased as specificity decreased, or vice
versa. We drew separate receiver operating characteristic plots
for studies that evaluated commonly reported magnetic
resonance imaging criteria, the Barkhof, Paty, and Fazekas crite-
ria, and the McDonald 2001 criteria, which combine clinical
information with findings on magnetic resonance imaging.
Further analysis was restricted to cohort studies with at least
10 years’ clinical follow-up. We produced separate receiver oper-
ating characteristic plots for each of these studies and compared
areas under the curves. The statistical software package Stata
release 9 was used for all analyses, except the hierarchical
summary receiver operating characteristic model, which was fit-
ted in SAS.15
Results
Figure 1 shows the flow of studies through the review. Sixty one
publications met the inclusion criteria, 21 of which were earlier
reports of included studies and were not extracted.w1-w43Forty
publications reporting the results of 29 studies (some reported
results for different magnetic resonance imaging criteria, for
imaging of the spine rather than the brain, or for patient
subgroups) were included. Sample sizes were generally small
(median 70), ranging from 15 to 1500 patients. The proportions
of dropouts ranged from 0 to 58% (median 4%), increasing with
length of follow-up. Table 1 provides details of the 29
publications reporting the results of 18 cohort studies. Most of
these studies used clinical follow-up as the reference standard.
Most used the Poser criteria,16although some used the McDon-
ald 1977 criteria.17The McDonald 1977 criteria,based on clinical
information alone, are not the same as the McDonald 2001 cri-
teria, which incorporate magnetic resonance imaging.4Table 2
provides details of the 11 studies of other designs. The studies
differed according to population, quality, magnetic resonance
imaging protocol,and criteria used to define a positive test result.
Cohort studies varied in their inclusion criteria; some included
only patients presenting with a particular clinically isolated syn-
drome (for example, optic neuritis or a spinal cord syndrome),
whereas others included all patients being evaluated for possible
multiple sclerosis. Publication dates ranged from 1986 to 2003.
Over this time improvements occurred in magnetic resonance
imaging technology; this is reflected in differences in scanning
protocols (see table A on bmj.com).
Figure 2 summarises the results of the quality assessment (see
table B on bmj.com for results of individual studies). Study qual-
ity was generally poor: only four QUADAS items were met by
over 70% of studies (avoidance of partial and differential
verification bias and reporting of uninterpretable results and
withdrawals). Studies scored badly on three items: blinding, the
use of an appropriate reference standard, and the availability of
clinical information. Four publications, reporting results from
three cohort studies, were susceptible to incorporation bias as
magnetic resonanceimaging
diagnosis.18–21Three of these used a combination of clinical
follow-up and paraclinical tests as the reference standard,19–21the
other relied on paraclinical tests alone.18All other cohort studies
used clinical follow-up alone as the reference standard.
Figure 3 shows that cohort studies produced lower estimated
sensitivity and specificity than studies of other designs. The
pooled diagnostic odds ratio was 9 (95% confidence interval 5 to
16) for cohort studies and 213 (85 to 535) for studies of other
designs (P < 0.001, permutation test). Further analysis was
restricted to the 15 cohort studies that used a diagnosis of clini-
cally definite multiple sclerosis, arrived at by clinical information
alone, as the reference standard.
The average duration of follow-up ranged from seven
months to 14 years. The only criteria for which sufficient data
were available to investigate the effects of duration of follow-up
were presence of one or more lesions and presence of one or
contributedto the final
Titles and abstracts screened (n=6474)
Full articles screened (n=188)
Met inclusion criteria (29 studies, 61 publications)
Excluded studies (n=127):
Not a primary diagnostic accuracy study (n=118)
No reference standard for diagnosis of multiple sclerosis (n=6)
Magnetic resonance imaging not used as index test (n=2)
Insufficient data to construct 2x2 table of test performance (n=1)
Not relevant (n=6286)
Not included in review: publications
with duplicate data (n=21)
Included in review: publications
with unique data (n=40)
Fig 1 Flow of studies through review process
Research
page 2 of 10
BMJ Online First bmj.com

Page 3

Table 1 Study details and results of cohort studies recruiting patients with suspected multiple sclerosis (MS)
Author (year),
country
Lee (1991)w1,
Canada 1
No of
patients* (%
female)
184 (67)
Mean (range)
age (years)
43 (12-79)
Presenting
symptoms Criteria for MS
Schumacherw2
Mean follow-up
(range) in
months
25 (NR)
% with MS
30
MRI criteria
≥1 lesions
Sensitivity,
specificity (%)
95, 49
Positive, negative
likelihood ratios
1.8, 0.1NR
Paty
≥1 non-clinical lesions
84, 63
91, 50
2.2, 0.3
1.8, 0.2
Ford (1992)w3,
Canada 2
Reese (1986)w5,
Canada 3
15 (53)33 (22-44)Spinal cord lesion
(n=15)
NR
Poserw4
38.5 (NR)80
47 (NR)NR Diagnosed by
referring
neurologist (no
details)
NR
68
≥1 T2 lesions 88, 875.5, 0.2
Frederiksen
(1989)w6,
Denmark
Brex (2002)w7,
England 1
60 (70)31 (12-53) Optic lesion (n=60)Median 11
(1-28)
20
≥1 non-clinical lesions 100, 441.7, 0.1
71 (62)32 (13-49) Optic lesion
(n=36), spinal cord
lesion (n=21),
brainstem lesion
(n=14)
Poserw4
168 (150-202) 68
≥1 non-clinical T2
lesions
92, 743.4, 0.1
≥4 non-clinical T2
lesions
>10 non-clinical T2
lesions
Initial lesion load
>1.23 cm3
58, 833.1, 0.5
31, 91 3.0, 0.8
Filippi (1994)w8,
England 1
84 (70)31 (13-50)Optic lesion
(n=40), spinal cord
lesion (n=28),
brainstem lesion
(n=16)
Optic lesion (n=69)
63 (43-84)40
53, 947.7, 0.5
Miller (1988)w9,
England 1
Miller (1989)w10,
England 1
53 (62)33 (16-48)12 (5-30) 28
≥1 non-clinical T2
lesions
≥1 non-clinical T2
lesions
≥1 non-clinical T2
lesions
Paty
80, 421.4, 0.5
56 (63)32 (13-49) Spinal cord lesion
(n=33)
Brainstem lesion
(n=23)
Spinal cord lesion
(n=25), brainstem
lesion (n=20)
14.7 (6-31)3090, 61 2.2, 0.2
35 100, 401.6, 0.1
Sharief
(1991)w11,
England 1
45 (62)37.3 (<50)
Poserw4†,
paraclinical data
contributed to
diagnosis
Poserw4†
18 (NR)49 77, 70 2.4, 0.3
Brex (2001)w12
(brain MRI),
England 2
68 (57)31 (16-50) Optic lesion
(n=46), spinal cord
lesion (n=6),
brainstem lesion
(n=16)
>12 26
≥1 non-clinical T2
lesions
89, 36 1.4, 0.4
≥4 non-clinical T2
lesions
≥9 non-clinical T2
lesions
≥1 non-clinical
enhancing lesions
≥1 enhancing lesions
at baseline and one
new lesion at three
months
≥1 non-clinical T2
lesions at baseline and
one new lesion at
three months
≥1 non-clinical T2
lesions
78, 541.7, 0.4
61, 722.1, 0.6
61, 80 2.9, 0.5
39, 945.8, 0.6
83, 763.3, 0.2
Brex (1999)w13
(spinal MRI),
England 2
50 (57) 30 (16-49) Optic lesion
(n=46), spinal cord
lesion (n=6),
brainstem lesion
(n=16)
Poserw4
12 (12-19) 2677, 70 2.5, 0.4
≥1 non-clinical
enhancing lesions
New T2 lesion at three
months
15, 92 1.9, 0.9
Dalton
(2003)w14,
England 2
56 (54) 32 (17-50)Optic lesion
(n=37), spinal cord
lesion (n=5),
brainstem lesion
(n=14)
Poserw4
NR (>36)34 84, 897.0, 0.2
McDonald 2001
criteria at three
months
McDonald 2001
criteria or new T2
lesions at three
months
58, 958.7, 0.5
74, 927.9, 0.3
Research
BMJ Online First bmj.com
page 3 of 10

Page 4

Author (year),
country
No of
patients* (%
female)
50 (59)
Mean (range)
age (years)
Presenting
symptoms
Criteria for MSMean follow-up
(range) in
months
37 (29-67)
% with MSMRI criteriaSensitivity,
specificity (%)
Positive, negative
likelihood ratios
Dalton
(2002)w15,
England 2
31 (16-50)Optic lesion
(n=90), spinal cord
lesion (n=10),
brainstem lesion
(n=19)
Poserw4
38
≥1 non-clinical lesions
(brain or spinal cord)
100, 351.5, 0.1
Barkhof (brain or
spinal cord MRI)
Barkhof (brain MRI
only)
McDonald 2001 MRI
at three months
McDonald 2001 MRI
at 12 months
McDonald 2001
criteria at three
months
McDonald 2001
criteria at 12 months
≥1 T2 lesions
79, 773.3, 0.3
63, 77 2.7, 0.5
59, 93 6.8, 0.5
83, 83 4.4, 0.2
65, 937.4, 0.4
94, 83 5.0, 0.1
Barkhof
(1997)w16,
Holland 1
74 (NR)NROptic lesion
(n=40), spinal cord
lesion (n=22),
brainstem lesion
(n=12)
Poserw4
39 (23-69)45 97, 291.4, 0.1
Barkhof
Fazekas
Paty
≥1 white matter
lesions
82, 78
88, 54
88, 54
100, 48
3.6, 0.2
1.9, 0.2
1.9, 0.2
1.9, 0
Ghezzi
(1999)w17,
Italy 1
102 (71) 29 (NR) Optic lesion
(n=143)
Poserw4
76 (>48) 36
Paty
McDonald 2001
criteria at three
months
≥3 multifocal lesions
86, 55
89, 68
1.9, 0.3
2.7, 0.2
Di Legge
(2002)w18,
Italy 2
Paolino
(1996)w19,
Italy 3
53 (66)30.5 (NR) NRNR >1836
44 (70)30 (<50) Spinal cord lesion
(n=22), brainstem
lesion (n=22)
McDonald 1977w20
Non-MS (80
months), MS
(26 month),
ranges NR
35 (NR)
6860, 712.0, 0.6
Filippini
(1994)w21,
Italy 4
82 (65)28 (14-51)Optic lesion (n=21)McDonald 1977w20
34
≥1 MS-like
abnormalities
96, 441.7, 0.1
Paty
Barkhof
68, 69
63, 70
2.1, 0.5
2.1, 0.5
Sastre-Garriga
(2004)w22,
Spain 1
153 (NR)32 (14-50) Optic lesion
(n=56), spinal cord
lesion (n=46),
brainstem lesion
(n=51);
Optic lesion
(n=56), spinal cord
lesion (n=46)
Brainstem lesion
(n=51)
Brainstem lesion
(n=51)
Poserw4
34-4026, 22, 35
Barkhof52, 73 1.9, 0.7
Barkhof78, 611.9, 0.4
Sastre-Garriga
(2003)w23,
Spain 1
51 (63) 29 (14-49) 37 (22-49)‡35
≥1 lesions94, 12 1.1, 0.6
≥1 non-clinical lesions
Fazekas
Paty
Fazekas
94, 42
89, 48
89, 52
77, 51
1.6, 0.2
1.7, 0.3
1.8, 0.3
1.6, 0.5
Tintore
(2001)w24,
Spain 1
112 (66)28 (13-49)Optic lesion
(n=36), spinal cord
lesion (n=41),
brainstem lesion
(n=23), other
(n=12)
31 (12-60) 23
Paty
McDonald 2001
criteria at 12 months
77, 51
74, 85
1.6, 0.5
4.8, 0.3
Tintore
(2003)w25,
Spain 1
86 (73)30 (13-49) Optic lesion
(n=58), spinal cord
lesion (n=39),
brainstem lesion
(n=34), other (n=8)
Optic lesion (n=35)
39 (12-77)44
Rio (1997)w26,
Spain 2
Soderstrom
(1998)w27,
Sweden
35 (80) 31 (17-47)NR29.4 (12-66) 20
≥1 T2 lesions100, 67 2.7, 0.1
147 (80)34 (12-57) Optic lesion
(n=147)
Poserw4
25 (0-71) 41
≥1 lesions85, 652.4, 0.2
Research
page 4 of 10
BMJ Online First bmj.com

Page 5

more non-clinical lesions. Figure 4 is a receiver operating
characteristic plot for these criteria, with numbers showing the
duration of follow-up in years. Evidence shows (P = 0.074 from
hierarchical summary receiver operating characteristic analysis)
that studies with longer follow-up produced higher estimated
specificity and lower estimated sensitivity.
The longest average duration of follow-up was three years in
studies that assessed the Barkhof, Fazekas, and McDonald 2001
criteria, and six years for studies that assessed the Paty criteria. It
is therefore possible to draw conclusions regarding the ability of
these criteria to predict the development of multiple sclerosis
only over these relatively short periods. Figure 5 shows the
receiver operating characteristic plots for these criteria. The
study that developed the Barkhof criteria22showed higher
estimated sensitivity and specificity than did the other studies of
this criterion. The negative likelihood ratios for the Barkhof,
Fazekas, and Paty criteria ranged from 0.2 to 0.5, suggesting that
a negative result on magnetic resonance imaging on the basis of
these criteria is of limited utility for ruling out the development
of multiple sclerosis within three to six years. Positive likelihood
ratios were < 5: thus these criteria are also of limited utility in
predicting the development of multiple sclerosis within three to
six years. Positive likelihood ratios for the McDonald 2001 crite-
ria ranged from 2.7 to 8.7, suggesting that they have more
potential for predicting the development of multiple sclerosis
within three years than any of the criteria based on magnetic
resonance imaging alone.23–26Negative likelihood ratios were 0.1
in one study and 0.2 to 0.5 in three studies, suggesting that the
McDonald 2001 criteria are of limited utility for ruling out the
development of multiple sclerosis within three years.
Only two studies, one from the United States2and one from
England,3followed patients for more than 10 years, long enough
to be reasonably confident that almost all patients had been
diagnosed as having multiple sclerosis who ever would be. Both
studies fulfilled all but one QUADAS criterion (the availability of
clinical information), and in the US study it was unclear whether
review bias had been avoided (see bmj.com). The US study
included 351 patients with optic neuritis; follow-up of more than
10 years was available for 302 (86%) of these. The study used
survival analysis to estimate the cumulative proportions of
patients diagnosed, with patients who did not receive a diagnosis
of multiple sclerosis censored at the time of their last clinical
follow-up. The English study included 135 patients with a range
of presenting symptoms, of whom 71 (53%) were included in the
final evaluation. Both studies evaluated thresholds based on the
number of non-clinical T2 lesions present on magnetic
resonance imaging of the brain.
Figure 6 shows the estimates of sensitivity and specificity,with
confidence intervals, for each of the thresholds evaluated in
these two studies. Sensitivity and specificity varied according to
the number of lesions used to define a positive result on
magnetic resonance imaging: sensitivity was higher with fewer
lesions but specificity was lower. Estimates of specificity were
similar for the two studies, but the English study tended to pro-
duce higher estimates of sensitivity. Comparison of areas under
the curves suggested better accuracy in the English study than in
the US study (P = 0.045). Estimates of the positive likelihood
ratios for the presence of various numbers of lesions ranged
from 2.0 to 3.4. Assuming a pretest probability of multiple scle-
rosis of 60% this is equivalent to a post-test probability of
75%-84%, suggesting that magnetic resonance imaging is of lim-
ited utility for ruling in multiple sclerosis at any threshold.
Estimates of the negative likelihood ratio ranged from 0.1 to 0.9
but were greater than 0.5 for all but one of the thresholds in the
English study. This is equivalent to modifying a pretest probabil-
ity of 60% to give a post-test probability of multiple sclerosis of
43%-57%, suggesting that magnetic resonance imaging is also of
limited utility in ruling out a diagnosis of multiple sclerosis.
Author (year),
country
No of
patients* (%
female)
189 (57)
Mean (range)
age (years)
Presenting
symptoms
Criteria for MS Mean follow-up
(range) in
months
No follow-up:
diagnosis after
testing
% with MSMRI criteria Sensitivity,
specificity (%)
Positive, negative
likelihood ratios
Beer (1995)w28,
Switzerland
38 (16-67)NR Poserw4§, also
incorporated
paraclinical tests
75
≥1 non-clinical lesions 84, 62 2.2, 0.3
Fazekas
≥1 non-clinical T2
lesions
≥2 T2 lesions
≥5 T2 lesions
≥9 T2 lesions
≥2 lesions, at least
one periventricular or
ovoid
≥1 lesions
60, 87
68, 68
4.4, 0.5
2.1, 0.5
Beck (2003)w29,
USA 1
388 (77) 32 (18-46) Optic lesion
(n=388)
Poserw4
Estimates for
MS at 10 years
37
51, 77
32, 89
12, 94
59, 73
2.2, 0.6
2.8, 0.8
2.0, 0.9
2.0, 0.6
Tumani
(1998)w30,
USA 1
Jacobs
(1997)w31
USA 2
Mushlin
(1993)w32,
USA and
Canada
28 (NR)32 (18-46) Optic lesion (n=36)Poserw4†>48 61
74 (69)34 (12-61) Optic lesion (n=74)McDonald 1977w20
67 (4-228)28 76, 511.5, 0.5
303 (73)37 (14-75) NR Committee decision
based on clinical
and paraclinical
data
7-8 (>6) 54Paty58, 916.1, 0.5
Three lesions or two
with one
periventricular
Multiple white matter
and periventricular
lesions
75, 773.3, 0.3
36, 9920.1, 0.7
Several publications refer to same study, as indicated by numbers after country.
MRI=magnetic resonance imaging; NR=not reported.
*Total number who entered trial (includes withdrawals); this may differ from sum of number of patients with each of presenting symptoms as for most studies these were reported only for
patients who completed the study.
†Patients with clinically probable MS were classified as having the disease.
‡Interquartile range.
§Patients with clinically probable or possible MS were classified as having the disease.
Research
BMJ Online First bmj.com
page 5 of 10

Page 6

Table 2 Study details and results of case-control studies and studies of other designs
Author
(year),
countryStudy design
Consecutive
scans done for
multiple
purposes (9%
query MS)
No of
patients*
(% female)
1500 (52)
Mean age
(range) in
years
46 (12-93)
No of patients
with confirmed
MS
0
No with other
conditions
1251
No of healthy
volunteers
115
Criteria for MS
Poserw4‡
MRI
criteria†
Fazekas
Sensitivity,
specificity (%)
81, 96
Positive,
negative
likelihood
ratios
20, 0.2
Offenbacher
(1993)w33,
Austria 1
≥3 areas of
high signal
intensity
Paty A
Paty B
Fazekas
90, 713.3, 0.1
87, 74
87, 92
88, 100
3.3, 0.2
10.2, 0.1
73.2, 0.1
Fazekas
(1988)w34,
Austria 2
Case-control 91 (NR)46 (14-77) 500 49NR†
≥1 lesion >6
mm
diameter
≥1
infratentorial
lesion
≥1
periventricular
≥3 lesions
≥3 mm
Three
lesions >3
mm
≥1
periventricular
lesion
92, 9515.3, 0.1
66, 9818.4, 0.4
80, 9822.2, 0.2
96, 59
98, 56
96, 59
2.3, 0.1
2.2, 0.1
2.3, 0.1
Ravnborg
(1992)w35,
Denmark
2
Cross-sectional
study of
patients with ≥1
neurological
attacks
Nested
case-control
68 (56)40 (18-63)0 680 Poserw4
95, 652.7, 0.1
Van der
Eerden
(1990)w36,
Holland 2
Bot
(2002)w37,
Holland 3
49 (50)45 (21-77)8000Schumacherw2
Presence of
periventricular
white matter
lesions
Barkhof
93, 907.7, 0.1
Case-control91 (73) 43 (NR)25660 Poserw4
76, 929.1, 0.3
Barkhof plus
abnormal
spinal cord
Fazekas
Fazekas plus
abnormal
spinal cord
Paty
Paty plus
abnormal
spinal MRI
≥1 lesions
Any
abnormality
in spinal
cord
Barkhof
76, 100 101, 0.3
92, 79
84, 97
4.2, 0.1
22.2, 0.2
100, 50
92, 95
2.0, 0.04
17.3, 0.1
100, 36
92, 94
1.5, 0.1
13.5, 0.1
Rovaris
(2002)w38,
Italy 5
Case-control 123 (67)40 (NR) 64 590 Poserw4
80, 9719, 0.2
Barkhof or
>1 spinal
cord lesion
≥1 lesions
Any
abnormality
in spinal
cord
≥1 lesions
>6 mm
diameter
≥1
infratentorial
lesion >3
mm
diameter
95, 100114, 0.1
100, 53
90, 100
2.1, 0.01
77.7, 0.1
Rovaris
(2000)w39,
Italy 6
Case-control54 (69) 40 (20-66) 10440NR
Kuroda
(1995)w40,
Japan 1
Case-control72 (71)44 (13-72)36 360 Schumacherw2‡
56, 948.2, 0.5
39, 100 29, 0.6
Research
page 6 of 10
BMJ Online First bmj.com

Page 7

Discussion
Use of magnetic resonance imaging to confirm multiple sclero-
sis on the basis of a single attack of neurological dysfunction may
lead to over-diagnosis and over-treatment. Many studies in our
systematic review produced inflated estimates of test perform-
ance owing to methodological weaknesses.
Only two cohort studies on the accuracy of magnetic
resonance imaging for the diagnosis of multiple sclerosis
included at least 10 years’ follow-up. These suggested that the
role of magnetic resonance imaging either in ruling in or ruling
out a diagnosis of multiple sclerosis is limited. Studies that did
not include an appropriate patient spectrum tended to overesti-
mate both sensitivity and specificity.Studies that included shorter
clinical follow-up tended to overestimate sensitivity and underes-
timate specificity. Specific criteria developed for the interpreta-
tion of magnetic resonance imaging scans as indicating multiple
sclerosis, the Fazekas, Barkhof, and Paty criteria, have poor accu-
racy for predicting the development of multiple sclerosis within
three to six years. The limited data on the McDonald 2001 crite-
ria suggest that these have some potential to rule in the develop-
ment of multiple sclerosis within three years. Neither the specific
magnetic resonance imaging criteria nor McDonald 2001 were
evaluated in studies with long term follow-up. It is therefore not
possible to determine their accuracy for the diagnosis of multiple
sclerosis.
Author
(year),
country
Study designNo of
patients* (%
female)
Mean age
(range) in years
No of patients
with confirmed
MS
No with other
conditions
No of healthy
volunteers
Criteria for MSMRI criteria†Sensitivity,
specificity (%)
Positive,
negative
likelihood ratios
10.6, 0.3
≥1
periventricular
lesion >3
mm
Three
lesions >3
mm
Subcallosal
striations
72, 94
86, 9412.6, 0.2
Palmer
(1999)w41,
USA 3
Gean-Marton
(1991)w42,
USA 4
Case-control 50 (76)40 (18-70)6 250 NR100, 84 5.8, 0.05
Case-control169 (46) 41 (22-87) 471220Poserw4
Focal
abnormalities
on
callosal-septal
interface
Abnormal
MRI scan:
equivocal
findings
classed as
normal
Abnormal
MRI scan:
equivocal
findings
classed as
normal
93, 9834, 0.1
Yetkin
(1991)w43,
USA 5
Case-control260 (NR) NR (31-83)92 68100Schumacherw2
76, 9617.1, 0.3
80, 9618.1, 0.2
MS=multiple sclerosis; MRI=magnetic resonance imaging; NR=not reported.
*Total number of patients who entered trial (includes withdrawals); in some cases this may differ from sum of number of patients with confirmed MS, other conditions, or healthy volunteers as
for most studies these were only reported for patients who completed the study.
†Patients with clinically probable MS were classified as having the disease.
‡Patients with clinically probable or possible MS were classified as having the disease.
Percentage
Patient spectrum
Selection criteria
Reference standard
Partial verification
Differential verification
Incorporation
Test details
Reference standard details
Test bias
Review bias
Clinical data
Uninterpretable results
Withdrawals
0
20
40
60
80
100
YesNo Unclear
Fig 2 Results of quality assessment for appropriate patient spectrum studies
Specificity
Sensitivity
1.0 0.8 0.6 0.40.20
0
0.2
0.4
0.6
0.8
1.0
Cohort studies
Other designs
Fig 3 Receiver operating characteristic plots for cohort studies and for studies of
other designs
Research
BMJ Online First bmj.com
page 7 of 10

Page 8

Strengths and weaknesses of the study
We carried out extensive literature searches, assessed study qual-
ity, and used recently developed statistical methods. Consider-
able weaknesses existed in the primary studies included in the
review. The only reference standard for the diagnosis of multiple
sclerosis is long term clinical follow-up. Most studies followed
patients for relatively short periods and so will have classified
some patients as not having multiple sclerosis who had a second
clinical attack after follow-up ended. Most studies included an
inappropriate patient spectrum,which we found to be associated
with considerably higher estimated diagnostic accuracy. Most of
such studies used a case-control design—they selected people
with clinically definite multiple sclerosis and a control group of
people known not to have the disease, either healthy controls or
patients with conditions that may present with similar symptoms
to multiple sclerosis. That such studies tend to exaggerate the
accuracy27of magnetic resonance imaging in the diagnosis of
multiple sclerosis is to be expected; people with more advanced
multiple sclerosis are more likely to have lesions on their
magnetic resonance imaging scans than those presenting in the
early stages of multiple sclerosis.
Strengths and weaknesses in relation to other studies
Although several reviews have assessed the accuracy of magnetic
resonance imaging in the diagnosis of multiple sclerosis,4 28 29we
are unaware of any systematic reviews. The McDonald 2001 cri-
teria incorporate the Barkhof criteria to define a positive MRI
scan.4The article reporting the McDonald 2001 criteria4refers to
a small number of studies to justify its selection of the Barkhof
criteria for this purpose. All these had methodological
weaknesses: they either used a case-control design or had an
average of less than three years clinical follow-up.This paper was
published before the two long term cohort studies from England
and the United States.2 3
A recently published detailed (but not systematic) review is
the report of the Therapeutics and Technology Assessment Sub-
committee of the American Academy of Neurology.28This was
limited to cohort studies and discussed in detail the problems
associated with the lack of a true reference standard, an accurate
method of determining whether or not a patient has multiple
sclerosis that can be applied at the same time as the index test,for
the diagnosis of the disease. It did not carry out any statistical
synthesis and instead presents a narrative overview of the results
of the English study and several other studies, also included in
our review, which had relatively short clinical follow-up, and was
published before the US study. It concluded, in contrast with our
findings, that the presence of at least three lesions on a magnetic
resonance imaging scan is a sensitive predictor of the
development of multiple sclerosis in the next 7-10 years,and that
normal results suggest that future development of multiple scle-
rosis is less likely. A more recent review article focused on the
McDonald 2001 criteria but also draws on the results of the
report oftheTherapeutics
Subcommittee of the American Academy of Neurology.29It
highlights the limitations of the evidence base for the McDonald
2001 criteria and draws on the results of the US2and English
studies3to conclude, consistent with the results presented here,
that presence of brain lesions does not guarantee development
of multiple sclerosis over 10-14 years.
and TechnologyAssessment
Unanswered questions and future research
The main clinical question is whether magnetic resonance imag-
ing should be included in the work-up of patients with multiple
Specificity
Sensitivity
1.00.8 0.60.4 0.20
0
0.2
0.4
0.6
0.8
1.0
≥1 lesion
≥1 non-clinical lesion
Fitted receiver operating characteristic curve
6
6
3
3
3
2
2
2
3
1
14
12
Fig 4 Receiver operating characteristic plots for studies included in hierarchical
summary receiver operating characteristic analysis. Numbers are duration of
follow-up in years
Specificity
Sensitivity
1.00.80.60.40.20
0
0.2
0.4
0.6
0.8
1.0Paty
Specificity
1.0
0.80.60.4 0.20
McDonald 2001
Sensitivity
0
0.2
0.4
0.6
0.8
1.0Barkhof
Fazekas
Fig 5 Receiver operating characteristic plots (95% confidence intervals) for
Barkhof, Fazekas, Paty, and McDonald 2001 criteria. In Barkhof plot, red is study
that proposed Barkhof criteria.22In McDonald 2001 plot, red indicates studies
where McDonald 2001 criterion was applied after three months rather than 12
months
Specificity
Sensitivity
1.0 0.80.60.40.20
0
0.2
0.4
0.6
0.8
1.0English study
Specificity
1.0
0.8 0.60.4 0.20
US study
≥1
≥1
≥4
≥11
≥2
≥5
≥9
Fig 6 Sensitivity plotted against specificity (95% confidence intervals) for
different thresholds (number of lesions shown next to plots) reported in English
and US studies2 3
Research
page 8 of 10
BMJ Online First bmj.com

Page 9

sclerosis. Several factors need to be considered, in particular the
reasons why magnetic resonance imaging is ordered. This is not
simply to increase the certainty of the diagnosis: other possible
reasons include ruling out differential diagnoses such as brain
tumours, providing a baseline for monitoring disease progres-
sion, patient request, and patient reassurance. If magnetic
resonance imaging scans are ordered to inform the diagnosis of
multiple sclerosis, and if the McDonald 2001 criteria that incor-
porate such imaging are to be used in practice, then further
research, based on long term cohort studies, is required to evalu-
ate these criteria. A limitation consequent on the need for long
term clinical follow-up in studies that evaluate the accuracy of
magnetic resonance imaging is that such studies inevitably use
older technology.Studies with more advanced,and hence recent,
technology inevitably had much shorter periods of follow-up.
Differences in estimates of sensitivity and specificity according to
magnetic resonanceimaging
confounded by differences in duration of follow-up.
The two studies that included follow-up of longer than 10
years produced differing results, with the US study reporting
lower estimates of sensitivity than the English study for similar
thresholds for magnetic resonance imaging. It is possible that
these differences reflect the smaller sample size of the English
study or that the large proportion of dropouts from this study
biased results. An alternative explanation is that magnetic
resonance imaging may be more accurate in patients presenting
with brainstem or spinal cord symptoms than in patients with
optic neuritis. Future studies should assess whether the accuracy
of magnetic resonance imaging varies according to presenting
symptoms.
Rather than the accuracy of magnetic resonance imaging
alone in diagnosing multiple sclerosis, the issue of clinical
relevance is, arguably, the added value of such imaging in
diagnosing the disease compared with the patient’s history and
clinical examination alone.30None of the identified studies
addressed this issue. A further limitation of published studies is
that they tend to dichotomise the results of magnetic resonance
imaging into positive or negative scans. The use of a scale based
on features present on a scan, ranging from no lesions (in which
case the probability of disease is low), to specific lesions (which
may imply a greatly increased probability of disease), should be
considered as an alternative to dichotomisation. This is probably
consistent with how the results of magnetic resonance imaging
are interpreted in practice.
technology weretherefore
Implications
In patients with clinically suspected multiple sclerosis, magnetic
resonance imaging currently allows a diagnosis of the disease
according to the McDonald 2001 criteria. Our results suggest
that magnetic resonance imaging is a relatively poor test for both
ruling in and ruling out multiple sclerosis. In clinical practice a
false positive diagnosis of multiple sclerosis is potentially more
dangerous than a false negative one because it implies unneces-
sary successive tests and treatments, or needless anxiety and psy-
chological distress for the patient.Wrongly ruling out a diagnosis
of multiple sclerosis after a first attack seems less dangerous: not
all patients who experience a first attack will develop the disease
and currently no treatment has been shown to delay conversion
to clinically definite multiple sclerosis or impacts on long term
disability. Neurologists should discuss with their patients the
potential diagnosis, treatment, and ultimate effect of potential
errors of false positive and false negative magnetic resonance
imaging results. High quality clinical research based on
improved magnetic resonance imaging techniques and meas-
ures in combination with a complete description of participants
and long term clinical follow-up are needed for quantitative
assessment of the clinical efficacy of magnetic resonance
imaging in the diagnosis of multiple sclerosis. The disease
remains a predominantly clinical diagnosis.
Contributors: PW, JACS, JJD, CM, and RH designed the study. PW carried
out the literature searches. PW and CM screened the results of the searches
for relevance, assessed inclusion, and carried out the data extraction and
quality assessment. GF provided expert advice on magnetic resonance
imaging and multiple sclerosis. RH, JJD, JACS, and PW developed the plan
of analysis and RH carried out the analysis. PW, JACS, RH, JJD, and GF
drafted the paper. All authors commented on drafts of the paper and
approved the final manuscript. PW is guarantor for the paper.
Funding:This work was supported by the Medical Research Council Health
Services Research Collaboration.The authors’work was independent of the
funders. JJD is funded by a senior research fellowship in evidence synthesis
from the UK Department of Health.
Competing interests: None declared.
Ethical approval: Not required.
1Poser CM,Brinar VV.Diagnostic criteria for multiple sclerosis:an historical review.Clin
Neurol Neurosurg 2004;106:147-58.
Beck RW, Trobe JD, Moke PS, Gal RL, Xing D, Bhatti MT, et al. High- and low-risk pro-
files for the development of multiple sclerosis within 10 years after optic neuritis:expe-
rience of the optic neuritis treatment trial. Arch Ophthalmol 2003;121:944-9.
Brex PA, Ciccarelli O, O’Riordan JI, Sailer M, Thompson AJ, Miller DH. A longitudinal
study of abnormalities on MRI and disability from multiple sclerosis. New Engl J Med
2002;346:158-64.
McDonald WI, Compston A, Edan G, Goodkin D, Hartung HP, Lublin FD, et al.
Recommended diagnostic criteria for multiple sclerosis: guidelines from the
international panel on the diagnosis of multiple sclerosis. Ann Neurol 2001;50:121-7.
National Collaborating Centre for Chronic Conditions. Multiple sclerosis. National clini-
cal guideline for diagnosis and management in primary and secondary care. London: Royal
College of Physicians, 2004.
Mushlin AI, Mooney C, Grow V, Phelps CE. The value of diagnostic information to
patients with suspected multiple sclerosis. Rochester-Toronto MRI Study Group. Arch
Neurol 1994;51:67-72.
Koopman W, Schweitzer A. The journey to multiple sclerosis: a qualitative study. J Neu-
rosci Nurs 1999;31:17-26.
Heesen C, Kolbeck J, Gold SM, Schulz H, Schulz KH. Delivering the diagnosis of
MS—results of a survey among patients and neurologists. Acta Neurol Scand
2005;107:363-8.
O’Connor P, Detsky AS, Tansey C, Kucharczyk W. Effect of diagnostic testing for multi-
ple sclerosis on patient health perceptions. Rochester-Toronto MRI Study Group. Arch
Neurol 1994;51:46-51.
10 O’Connor P, Canadian Multiple Sclerosis Working Group. Key issues in the diagnosis
and treatment of multiple sclerosis. An overview. Neurology 2002;59:S1-33.
11 Whiting P, Rutjes AW, Dinnes J, Reitsma J, Bossuyt PM, Kleijnen J. Development and
validation of methods for assessing the quality of diagnostic accuracy studies. Health
Technol Assess 2004;8:1-234.
12 Deeks JJ, Altman DG. Diagnostic tests 4: likelihood ratios. BMJ 2004;329:168-9.
13 Higgins JP, Thompson SG. Controlling the risk of spurious findings from
meta-regression. Stat Med 2004;23:1663-82.
14 Rutter CA, Gatsonis CA. A hierarchical regression approach to meta-analysis of diag-
nostic test accuracy evaluations. Stat Med 2001;20:2865-84.
15 Macaskill P. Empirical Bayes estimates generated in a hierarchical summary ROC
analysis agreed closely with those of a full Bayesian analysis. J Clin Epidemiol
2004;57:925-32.
16 Poser CM, Paty DW, Scheinberg L, McDonald WI, Davis FA, Ebers GC, et al. New diag-
nostic criteria for multiple sclerosis: guidelines for research protocols. Ann Neurol
1983;13:227-31.
2
3
4
5
6
7
8
9
What is already known on this topic
Magnetic resonance imaging has been recommended in the
diagnosis of multiple sclerosis
The diagnostic accuracy of such imaging has been assessed
but a systematic review has not previously been carried out
What this study adds
Magnetic resonance imaging is of limited utility for both
ruling in and ruling out multiple sclerosis
Studies with shorter follow-up tended to produce higher
estimates of sensitivity and lower estimates of specificity
compared with longer term studies
Research
BMJ Online First bmj.com
page 9 of 10

Page 10

17 McDonald WI, Halliday AM. Diagnosis and classification of multiple sclerosis. Br Med
Bull 1977;33:4-9.
18 Beer S, Rosler KM, Hess CW. Diagnostic value of paraclinical tests in multiple sclerosis:
relative sensitivities and specificities for reclassification according to the Poser commit-
tee criteria. J Neurol Neurosurg Psychiatry 1995;59:152-9.
19 Miller DH, Ormerod IE, McDonald WI, MacManus DG, Kendall BE, Kingsley DP, et al.
The early risk of multiple sclerosis after optic neuritis. J Neurol Neurosurg Psychiatry
1988;51:1569-71.
20 Sharief MK, Thompson EJ. The predictive value of intrathecal immunoglobulin
synthesis and magnetic resonance imaging in acute isolated syndromes for subsequent
development of multiple sclerosis. Ann Neurol 1991;29:147-51.
21 Mushlin AI, Detsky AS, Phelps CE, O’Connor PW, Kido DK, Kucharczyk W, et al. The
accuracy of magnetic resonance imaging in patients with suspected multiple sclerosis.
TheRochester-TorontoMagneticResonance
1993;269:3146-51.
22 Barkhof F, Filippi M, Miller DH, Scheltens P, Campi A, Polman CH, et al. Comparison
of MRI criteria at first presentation to predict conversion to clinically definite multiple
sclerosis. Brain 1997;120:2059-69.
23 Dalton CM,Brex PA,Miszkiel KA,Hickman SJ,MacManus DG,Plant GT,et al.Applica-
tion of the new McDonald criteria to patients with clinically isolated syndromes
suggestive of multiple sclerosis. Ann Neurol 2002;52:47-53.
24 Dalton CM, Brex PA, Miszkiel KA, Fernando K, MacManus DG, Plant GT, et al. New T2
lesions enable an earlier diagnosis of multiple sclerosis in clinically isolated syndromes.
Ann Neurol 2003;53:673-6.
25 Di Legge S, Piattella MC, Pantano P, Pestalozza IF, Nucciarelli W, Bozzao L, et al. The
impact of revised McDonald criteria in predicting multiple sclerosis. Neurology
2002;58:A173.
26 Tintore M, Rovira A, Rio J, Nos C, Grive E, Sastre-Garriga J, et al. New diagnostic crite-
ria for multiple sclerosis: application in first demyelinating episode. Neurology
2003;60:27-30.
27 Whiting P,Rutjes AW,Reitsma JB,Glas AS,Bossuyt PM,Kleijnen J.Sources of variation
and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med
2004;140:189-202.
ImagingStudy Group.
JAMA
28 Frohman EM, Goodin DS, Calabresi PA, Corboy JR, Coyle PK, Filippi M, et al. The util-
ity of MRI in suspected MS: report of the Therapeutics and Technology Assessment
Subcommittee of the American Academy of Neurology. Neurology 2003;61:602-11.
29 Miller DH, Filippi M, Fazekas F, Frederiksen JL, Matthews PM, Montalban X, et al. Role
of magnetic resonance imaging within diagnostic criteria for multiple sclerosis. Ann
Neurol 2004;56:273-8.
30 Moons KG, Biesheuvel CJ, Grobbee DE. Test research versus diagnostic research. Clin
Chem 2004;50:473-6.
(Accepted 2 February 2006)
doi 10.1136/bmj.38771.583796.7C
MRC Health Services Research Collaboration, Department of Social Medicine,
Bristol BS8 2PR
Penny Whiting research fellow
Roger Harbord research associate
Jonathan A C Sterne reader in medical statistics and epidemiology
Centre for Reviews and Dissemination, University of York
Caroline Main research fellow
Centre for Statistics in Medicine, Wolfson College, Oxford
Jonathan J Deeks senior medical statistician
Unit of Neuroepidemiology, Istituto Nazionale Neurologico “Carlo Besta,” Milan,
Italy
Graziella Filippini head
Department of Social and Preventive Medicine, University of Bern, Switzerland
Matthias Egger professor of epidemiology and public health
Correspondence to: P Whiting penny.whiting@bristol.ac.uk
Research
page 10 of 10
BMJ Online First bmj.com