![Page on the internet showing, in the left and right columns, MR images of the temporomandibular joint (TMJ) in the closed mouth position [proton density (PD) and T 2 ], open mouth position (PD) and coronal view (PD). Images can be shown in larger magnification in the middle fields. In the row in the bottom, alternative diagnoses can be chosen. In the corner to the left, it is possible to click on the different buttons and the written criteria will be shown](https://www.researchgate.net/profile/Kristina-Hellen-Halme/publication/224896335/figure/fig1/AS:302785804423173@1449201169931/Page-on-the-internet-showing-in-the-left-and-right-columns-MR-images-of-the_Q320.jpg)
Content uploaded by Kristina Hellén-Halme
Author content
All content in this area was uploaded by Kristina Hellén-Halme on Nov 24, 2014
Content may be subject to copyright.
RESEARCH
Web-based calibration of observers using MRI of the
temporomandibular joint
K Helle´n-Halme*
,1
, L Hollender
2
, M Janda
3
and A Petersson
1
1
Department of Oral and Maxillofacial Radiology, Faculty of Odontology, Malmo¨ University, Malmo¨ , Sweden;
2
Department of Oral
Medicine, School of Dentistry, University of Washington, Seattle, WA, USA;
3
Public Dental Health Service of Ska˚ne, Lund, Sweden
Objectives: Research diagnostic criteria for temporomandibular disorders (RDC/TMDs)
were proposed in 1992 with the aim of standardizing and testing methods for diagnosing
TMDs. RDC/TMDs have so far been lacking standardized methods for imaging and criteria
for imaging diagnosis of disorders of the temporomandibular joint (TMJ). Criteria for
disorders of the TMJ have recently been proposed for MRI. The aim of this study was to
publish MR images of the TMJ on the web and to test the agreement of five observers using
the criteria.
Methods: 20 cases of MRI of the TMJ were published on the web together with the criteria.
The MR images were taken in closed and open mouth positions, and sagittal and coronal
views. Five observers diagnosed disc position, disc shape, joint effusion and loose calcified
bodies in the TMJ.
Results: In all cases except one, three or more observers agreed upon the diagnosis. All
agreed on whether a loose calcified body was present or not. The second best agreement was
obtained for disc position in the sagittal view, where all observers agreed in 16 of the 20 cases.
For disc position in the coronal view and the evaluation of the disc shape, observer
agreement was lower.
Conclusion: Criteria were useful in order to standardize and simplify evaluation and thereby
probably increase the diagnostic outcome among different observers for MRI of the TMJ.
We recommend that the criteria be used internationally to facilitate comparisons between
different studies.
Dentomaxillofacial Radiology (2012) 41, 656–661. doi: 10.1259/dmfr/82622690
Keywords: magnetic resonance imaging; observer variation; temporomandibular disorders;
temporomandibular joint
Introduction
Evaluation of research depends on complete and accurate
reporting. The Standards for Reporting of Diagnostic
Accuracy (STARD) steering committee has taken an
initiative to improve reporting of studies of diagnostic
accuracy, and a checklist for reporting diagnostic
accuracy studies has been published.
1
The list includes
methods to test the reproducibility of observers.
Research diagnostic criteria for temporomandibular
disorders (RDC/TMDs) were proposed by Dworkin
and LeResche
2
in 1992 with the aim of standardizing
and testing methods for diagnosing TMDs. Today the
RDC/TMD is an international consortium with the
goal of advancing the scientific knowledge of TMDs
and related pain conditions through authenticated tools
for international use, multisite and cross-cultural
research studies based on standardized assessment,
annual meetings and international consensus work-
shops (http://www.rdc-tmdinternational.org/). RDC/
TMD has so far been lacking standardized methods
for imaging and imaging diagnosis of disorders of the
temporomandibular joint (TMJ). Recent reviews have
concluded that there is a lack of consistency in
diagnostic criteria and the classification system for
MRI of the TMJ.
3,4
Criteria have been proposed for
*Correspondence to: Dr Kristina Helle´n-Halme, Department of Oral and
Maxillofacial Radiology, Faculty of Odontology, Malmo¨ University, SE-205 06
Malmo¨, Sweden. E-mail: kristina.hellen-halme@mah.se
Received 24 October 2011; revised 29 December 2011; accepted 30 December
2011
Dentomaxillofacial Radiology (2012) 41, 656–661
’2012 The British Institute of Radiology
http://dmfr.birjournals.org
CT and MRI within the framework of RDC/TMD, and
a high reliability was found for assessing osteoarthritis
(OA) using CT and for disc position and effusion using
MRI.
5
Previously it has been shown that interobserver
agreement was increased with the aid of reference
films.
6
Publishing reference images and creating a
calibration program on the web could provide a way
of increasing the coherence among observers using
MRI of the TMJ.
The aim was therefore to test if the criteria proposed by
Ahmad et al
5
could be useful when applied by several
observers using MR images of the TMJ published on the
web. A future aim is to use the images as reference images
in calibration of observers using MRI of the TMJ. The
hypothesis was that criteria could be a way to facilitate
and increase observer agreement for MRI of the TMJ.
Materials and methods
The images in this study were obtained during 2008
and the first 6 months of 2009 at the Department
of Radiology, Ska˚ne University Hospital, Malmo¨,
Sweden. All examinations were made in a Siemens
Magnetom Vision 1.5 T machine (Siemens, Erlangen,
Germany) with a bilateral TMJ surface coil. For the
closed mouth position, the patients were instructed to
close their mouth with their teeth in maximal contact.
In the open mouth position, a stepped plastic bite block
was placed between the upper and lower incisors. Use
of the bite block was practised before the examination.
The patients were asked to open their mouths as much
as they could without experiencing unbearable pain.
The examination included sagittal and coronal imaging
of the TMJ in the closed mouth position, and sagittal
images in the open mouth position. Sagittal sections
were orientated perpendicular to the long axis of the
condyle, and coronal sections were taken parallel to the
long axis. The orientation was determined using axial
localizers, one for the closed mouth position and one
for the open mouth position. Proton density (PD) and
T
2
weighted images were acquired using a double-echo
turbo-spin echo sequence [echo time (TE), 15/105 ms;
repetition time (TR), 2400 ms (sagittal) or 2000 ms
(coronal); turbo factor, 7; field of view (FOV),
1606160 mm
2
; matrix, 2556512; and slice thickness,
3 mm]. Each sequence yielded 8–10 sections.
Selection and evaluation of images
The criteria for the non-osseous components of the TMJ
described by Ahmad et al
5
were used for selection of MR
images in order to create samples of different scenarios
relative to disc position, disc shape, effusion and
presence of loose calcified bodies in the TMJ. The aim
was to select five cases of each scenario, but for loose
calcified bodies only one case was found. The different
criteria are listed in Table 1. Disc diagnoses for the TMJ
using MRI according to Ahmad et al
5
are given in
Table 2. 25 samples of MR images exemplifying the
different criteria were published on the internet. All cases
had previously been evaluated and regarded as accep-
table examinations at the hospital. Images from different
examinations were included with differences in image
quality. The images shown were PD sagittal images,
closed and open mouth, T
2
sagittal images closed mouth
and PD coronal image closed mouth (Figure 1). Five
observers evaluated the images from the web, four of
whom had more than 15 years’ experience in evaluating
MRI of the TMJ; the fifth observer was a post-graduate
student. In five cases, one or more of the observers
thought that the image quality was unacceptable when
the cases were evaluated, and these five cases were
excluded. The remaining 20 cases were used in the
evaluation of the criteria and they are presently
published on the internet together with instructions on
how to use the program (http://www.mah.se/od/rad/tmj).
The diagnoses in Table 2 were used for determining
the disc position in the sagittal plane. In all other
evaluations the criteria shown in Table 1 were used.
The observer had the opportunity to read the criteria at
any time when observing the images (Figure 1).
Results
The results of the assessments of the five observers are
shown in Table 3. The agreements of the observers for
the 20 cases are shown in Table 4. In all cases except
one, three or more observers agreed upon the diagnosis.
In one case of joint effusion, two observers agreed that
there was no effusion; two observers indicated slight
effusion, while one thought it was frank. All observers
agreed on whether a loose calcified body was present or
not. However, the presence of a loose calcified body
was shown only in one case. The second best agreement
was obtained for diagnosis of the disc position in the
sagittal view, where all 5 observers agreed in 16 of the
20 cases. For disc position in the coronal view and the
evaluation of the disc shape, the agreement was lower.
Total agreement for the evaluation of disc shape was
achieved in only 6 of the 20 cases.
Discussion
Diagnostic radiology is part of a larger system where
the goal is to treat patients with efficacy and efficiency.
7
The accuracy of a diagnostic method can be defined in
many different ways. Sensitivity and specificity are two
measures that are often used to describe the diagnostic
accuracy. Another important measure is observer
agreement, i.e. how different observers agree when
reading the same images. A systematic literature review
of the efficacy of MRI of the TMJ in diagnosing
degenerative and inflammatory joint diseases showed
that no conclusion concerning the diagnostic efficacy
could be reached.
3
One problem was that diagnostic
criteria varied between publications. In order to
RDC/TMD: evaluation of MR images using written criteria
K Helle´n-Halme
et al
657
Dentomaxillofacial Radiology
improve the diagnostic accuracy for MR images of the
TMJ, specific imaging criteria have been developed by
Ahmad et al.
5
The criteria are intended to be used in the
RDC/TMD classification system. This system is used in
many countries in order to simplify and guide how an
examination of a patient with TMD should be
performed. The criteria can also serve as a base when
calibrating different clinicians. This aim of this study
was to evaluate whether the guidelines for MRI of the
TMJ could be used as a way to calibrate observers for
better diagnostic accuracy. The reliability of the criteria
in the study by Ahmad et al
5
was examined with
observers who were calibrated in an extensive training
programme. The mean kappa values in the study by
Ahmad et al
5
reached a value of 0.84 for any disc
displacement and 0.64 for effusion. The kappa value as
a statistical measurement is calculated from data which
form a square table from two observers with different
options to choose from. In this study, we chose
therefore to report the observer agreement in percen-
tage figures.
The internet has made it possible to reach out to
many people. Its use for the purpose of this study is,
however, rather new. The attempt was to make a
website with the purpose of presenting different
scenarios of TMJ conditions in order to evaluate if
criteria such as those published by Ahmad et al
5
can be
helpful to increase the observer agreement for MR
images of the TMJ. In order to represent a clinical
situation, the images chosen were not optimized with
regard to image quality. All MR examinations had been
used for diagnosing patients with TMD and approved
by the radiologist who made the initial report. Further,
the five observers agreed upon using the images in this
investigation. It has been reported that higher quality
MR images lead to a better interobserver agreement,
8
and therefore the procedure used in this investigation
can be subject to discussion. If, in the future, a website
for calibration and learning to diagnose different
conditions of the TMJ is published on the internet,
the highest quality MR images should be chosen and
revised continuously as the MRI technique develops.
All observers in our study except one were experienced
in the use of MRI of the TMJ. They had several years of
clinical experience and extensive research in the area.
The post-graduate student, however, had substantial
experience of oral radiography, and the decisions made
by the student did not affect the final results. This fact
could indicate that MR images of the TMJ can be
evaluated with the aid of written criteria if the observer
has previous knowledge of radiological interpretation.
The MRI technique could further be refined for
obtaining the coronal images of the disc,
9
as this
technique was used in the article by Ahmad et al.
5
Table 1 Non-osseous component assessment criteria using MRI of the temporomandibular joint from Ahmad et al
5
Disc position: closed mouth sagittal and axially corrected coronal views
Normal disc position
i. In the sagittal plane, relative to the superior aspect of the condyle, the border between the low signal of the disc and the high signal of the
retrodiscal tissue is located between the 11:30 and 12:30 clock positions
ii. In the sagittal plane, the intermediate zone is located between the anterosuperior aspect of the condyle and the posteroinferior aspect of the
articular eminence
iii. In the oblique coronal plane, the disc is centred between the condyle and eminence in the medial, central and lateral parts
Indeterminate
i. In the sagittal plane, relative to the superior aspect of the condyle, the low signal of the disc and the high signal of the retrodiscal tissue are
located anterior to the 11:30 position, but the condyle contacts the intermediate zone located between the anterosuperior aspect of the condyle
and the posteroinferior aspect of the articular eminence, or
ii. In the sagittal plane, relative to the superior aspect of the condyle, the low signal of the disc and the high signal of the retrodiscal tissue are
located between the 11:30 and 12:30 clock positions, but the intermediate zone of the disc is located anterior to the condyle, and
iii. In the axially corrected coronal plane, the disc is positioned between the condyle and eminence in the medial, central and lateral parts
Disc displacement
i. In the sagittal plane, relative to the superior aspect of the condyle, the low signal of the disc and the high signal of the retrodiscal tissue are
located anterior to the 11:30 clock position, and
ii. In the sagittal plane, the intermediate zone of the disc is located anterior to the condyle, or
iii. In the axially corrected coronal plane, the disc is not centred between the condyle and eminence in either the medial or the lateral parts
Disc not visible: Neither signal intensity nor outline make it possible to define a structure as the disc
Disc position: open mouth sagittal views
Normal disc position: The intermediate zone is located between the condyle and the articular eminence
Persistent disc displacement: The intermediate zone is located anterior to the condylar head
Disc not visible: Neither signal intensity nor outlines make it possible to define a structure as the disc
Disc shape: closed mouth sagittal views
Normal: The disc in the sagittal plane is biconcave
Deformed: All shapes other than biconcave in the sagittal plane
Disc not visible: Neither signal intensity nor outlines make it possible to define a shape of the disc
Effusion: open or closed mouth sagittal views
None: No bright signal in either joint space in the T
2
weighted images
Slight effusion: A bright signal in either joint space that conforms to the contours of the disc, fossa/articular eminence and/or condyle
Frank effusion: A bright signal in either joint space that extends beyond the osseous contours of the fossa/articular eminence and/or condyle and
has a convex configuration in the anterior or posterior recesses
Loose calcified bodies: closed mouth sagittal views
Single or multiple discrete low-signal intensity objects are present in the joint spaces, and are not attached to the condyle, fossa or eminence in any
plane
RDC/TMD: evaluation of MR images using written criteria
658 K Helle´n-Halme
et al
Dentomaxillofacial Radiology
Several studies of observer agreement at MRI of the
TMJ have been published.
10–17
It is difficult to compare
the studies as different criteria were used, for instance
for disc position. Widmalm et al
16
concluded that a
diagnosis of TMD based on MRI examination proto-
cols made by a single examiner should not be accepted,
as the diagnosis may be misleading. A recent publica-
tion concluded that examiners do not demonstrate
reproducibility in the interpretation of MRI of the
TMJ.
17
It seems that the diagnosis ‘‘anterior disc
displacement without reduction’’ reached the highest
kappa values and agreements between observers in
different publications.
8,13,15–17
In 80%of the cases the
five observers agreed on the diagnosis of the disc in the
sagittal plane, a finding that is similar to our results.
The observations of disc position in the coronal
plane, joint effusion and disc shape reached much lower
agreement between the observers. This is consistent
with the findings of Takano et al,
15
Widmalm et al,
16
Butzke et al
17
and Schmitter et al.
8
However, better
agreement of the disc shape was found by Tasaki et al.
10
One way of increasing the agreement between observers
is to reduce the number of categories. This was
proposed by Takano et al
15
regarding the finding of
joint effusion, and in our study, if the finding of joint
fluid was dichotomized to ‘‘yes’’ or ‘‘no’’, the observer
agreement would increase to 80%. Therefore, we
propose that the criteria described by Ahmad et al
5
are changed for joint effusion to be stated as present or
not. Regarding the judgement of disc deformity, a
better definition would perhaps be in case of a disc
displacement with reduction that the shape of the disc is
judged in the open (reduced) position. If the shape of
the disc is normal in the open position then the
judgement should be that of a normal disc shape, since
the deformity seen in the closed was only ‘‘temporary’’.
This was also suggested by Ahmad et al,
5
and it will be
used in the future presentation of the criteria.
Figure 1 Page on the internet showing, in the left and right columns, MR images of the temporomandibular joint (TMJ) in the closed mouth
position [proton density (PD) and T
2
], open mouth position (PD) and coronal view (PD). Images can be shown in larger magnification in the
middle fields. In the row in the bottom, alternative diagnoses can be chosen. In the corner to the left, it is possible to click on the different buttons
and the written criteria will be shown
RDC/TMD: evaluation of MR images using written criteria
K Helle´n-Halme
et al
659
Dentomaxillofacial Radiology
The MR images in the sagittal plane showed closed
and open mouth positions of the TMJ. The images
formed the bases for giving diagnoses such as disc
displacement with or without reduction. The coronal
images could only be used for identifying the disc
position in the closed mouth position. This limits the
number of possible diagnoses of the disc position in the
coronal plane and a combination of diagnoses in
the sagittal and coronal plane. However, this decision
was taken by the RDC/TMD consortium, as more
emphasis was put on disc displacement in the ante-
roposterior direction than in the mediolateral direction.
A diagnostic examination should ideally lead to
different treatment options.
7
Today there are no such
possibilities in RDC/TMD, and this might be a
suggestion for future development of the system.
The criteria proposed by Ahmad et al
5
are important
to follow in order to standardize the reporting of TMD
in MRI of the TMJ. We suggest that the criteria are
used internationally following the intentions of RDC/
Table 2 Disc diagnosis in the sagittal plane for the temporomandibular joint using MRI according to Ahmad et al
5
A. Normal: disc location is normal on closed and open mouth images
B. Disc displacement with reduction: disc location is displaced on closed mouth images but normal in open mouth images
C. Disc displacement without reduction: disc location is displaced on closed mouth and open mouth images
D. Indeterminate: disc location is not clearly normal or displaced in the closed mouth position
E. Disc not visible: neither signal intensity nor outlines make it possible to define a structure as the disc in the closed mouth and open mouth view
Table 3 Results of the five assessments for Observers a–e
Disc position sagittal Disc position coronal Disc shape Effusion Loose calcified body
Case number a b c d e a b c d e a b c d e a b c d e a b c d e
1 4444443441222222211122222
2 3333311111111112222322222
3 1111111111111112222222222
4 1111111221111112211222222
5 3333322221112212322322222
6 3333311111112213322322222
7 1111111331112211111122222
8 4444434441221122322222222
9 3333311111112223222322222
10 4444424441122222311222222
11 1122121111112211111122222
12 1111111111112211111122222
13 4444444441222232211122222
14 4444444441222221111111111
15 2244211113212231111122222
16 2233244441111131111122222
17 1111112222212211111122222
18 1111111111211111111122222
19 2444434441322221111122222
20 1111141111111111111122222
Disc position sagittal (diagnoses): 1, normal; 2, indeterminate; 3, disc displacement with reduction; 4, disc displacement without reduction; 5, disc
not visible.
Disc position coronal: 1, normal; 2, lateral; 3, medial; 4, disc not visible.
Disc shape: 1, normal; 2, deformed; 3, disc not visible.
Effusion: 1, none; 2, slight; 3, frank.
Loose calcified body: 1, yes; 2, no.
Table 4 Agreements between the five observers
Cases
Diagnosis 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 All (%)
A 55555555553555335545 80
B 35534533534544444534 30
C 55553333343345343445 30
D 34533354325535555555 55
E 55555555555555555555100
Number indicates how many observers agreed.
Diagnosis: A, disc position sagittal; B, disc position coronal; C, disc shape; D, effusion; E, loose calcified body.
The total agreement of all five observers is given in the column ‘‘all’’.
RDC/TMD: evaluation of MR images using written criteria
660 K Helle´n-Halme
et al
Dentomaxillofacial Radiology
TMD. Our internet page can be used in order to train
and calibrate observers, as in the majority of cases more
than three of five trained observers agreed upon the
diagnosis. On the website it will be possible to reach our
conclusions after the evaluations are made on a
separate page.
The conclusion of this study was that the use of
criteria was useful in order to standardize and simplify
the evaluation and thereby probably increase the
diagnostic outcome among different observers for
MRI of the TMJ. We recommend that the criteria are
used internationally to be able to compare different
studies.
Acknowledgments
We want to thank Professor Per-Lennart Westesson and Senior
Consultant Lars-Go¨ran Hansson for acting as observers.
References
1. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP,
Irwig LM, et al. Towards complete and accurate reporting of
studies of diagnostic accuracy: the STARD initiative. BMJ 2003;
326: 41–44.
2. Dworkin SF, LeResche L. Research diagnostic criteria for
temporomandibular disorders: review, criteria, examinations and
specifications, critique. J Craniomandib Disord 1992; 6: 301–355.
3. Limchaichana N, Petersson A, Rohlin M. The efficacy of
magnetic resonance imaging in the diagnosis of degenerative
and inflammatory temporomandibular joint disorders: a systema-
tic literature review. Oral Surg Oral Med Oral Pathol Oral Radiol
Endod 2006; 102: 521–536.
4. Koh KJ, List T, Petersson A, Rohlin M. Relationship between
clinical and magnetic resonance imaging diagnoses and findings in
degenerative and inflammatory temporomandibular joint diseases:
a systematic literature review. J Orofac Pain 2009; 23: 123–139.
5. Ahmad M, Hollender L, Anderson Q, Kartha K, Ohrbach R,
Truelove EL, John MT, Schiffman EL. Research diagnostic
criteria for temporomandibular disorders (RDC/TMD): develop-
ment of image analysis criteria and examiner reliability for image
analysis. Oral Surg Oral Med Oral Pathol Oral Radiol Endod
2009; 107: 844–860.
6. Rohlin M, Petersson A. Rheumatoid arthritis of the tempor-
omandibular joint: radiologic evaluation based on standard
reference films. Oral Surg Oral Med Oral Pathol Oral Radiol
Endod 1989; 67: 594–99.
7. Fryback DG, Thornbury JR. The efficacy of diagnostic imaging.
Med Decis Making 1991; 11: 88–94.
8. Schmitter M, Kress B, Ha¨hnel S, Rammelsberg P. The effect of
quality of temporomandibular joint MR images on interrater
agreement. Dentomaxillofac Radiol 2004; 33: 253–258.
9. Hollender L, Barclay P, Maravilla K, Terry V. A new coronal
imaging plane for magnetic resonance imaging of the tempor-
omandibular joint. Dentomaxillofac Radiol 1998; 27: 48–50.
10. Tasaki MM, Westesson P-L, Raubertas RF. Observer variation
in interpretation of magnetic resonance images of the tempor-
omandibular joint. Oral Surg Oral Med Oral Pathol 1993; 76:
231–234.
11. Nebbe B, Brooks SL, Hatcher D, Hollender LG, Prasad NGN,
Major PW. Interobserver reliability in quantitative MRI assess-
ment of temporomandibular joint disk status. Oral Surg Oral
Med Oral Pathol Oral Radiol Endod 1998; 86: 746–750.
12. Orsini MG, Kuboki T, Terada S, Matsuka Y, Yamashita A,
Clark GT. Diagnostic value of 4 criteria to interpret tempor-
omandibular joint normal disk position on magnetic resonance
images. Oral Surg Oral Med Oral Pathol Oral Radiol Endod 1998;
86: 489–497.
13. Nebbe B, Brooks SL, Hatcher D, Hollender LG, Prasad NGN,
Major PW. Magnetic resonance imaging of the temporomandib-
ular joint: interobserver agreement in subjective classification of
disk status. Oral Surg Oral Med Oral Pathol Oral Radiol Endod
2000; 90: 102–107.
14. Vahlensieck M, Okweschokwu S, Greven M. Magne-
tresonanztomographie (MRT) des Kiefergelenkes: Einfluss auf
Therapieentscheidung und U
¨bereinstimmung zweier Auswerten.
English abstract. Fortschr Ro¨ntgenstr 2002; 174: 1415–1421.
15. Takano Y, Honda K, Kashima M, Yotsui Y, Igarashi C,
Petersson A. Magnetic resonance imaging of the temporoman-
dibular joint: a study of inter- and intraobserver agreement. Oral
Radiol 2004; 20: 62–67.
16. Widmalm SE, Brooks SL, Sano T, Upton LG, McKay DC.
Limitation of the diagnostic value of MR images for diagnosing
temporomandibular joint disorders. Dentomaxillofac Radiol 2006;
35: 334–338.
17. Butzke KW, Batista Chaves KD, Dias da Silveira HE, Dias da
Silveira HL. Evaluation of the reproducibility in the interpreta-
tion of magnetic resonance images of the temporomandibular
joint. Dentomaxillofac Radiol 2010; 39: 157–161.
RDC/TMD: evaluation of MR images using written criteria
K Helle´n-Halme
et al
661
Dentomaxillofacial Radiology
