Ensuring inter-tester reliability of voluntary muscle and monofilament sensory testing in the INFIR Cohort Study.

Abstract

To assess the reliability of monofilament (MF) and voluntary muscle strength (VMT) testing carried out by nine physiotherapy staff recruited for the ILEP Nerve Function Impairment & Reaction (INFIR) Cohort Study in India.
A multiple pair inter-tester reliability study was carried out in Uttar Pradesh, India. Newly trained testers were paired up with an experienced physiotherapist, whose assessment served as the gold standard. Each pair completed a series of assessments. All testers had undertaken a week of specific VMT and MF training, followed by a month of practice in the hospital setting. Reliability was assessed by calculating weighted Kappa (Kw) statistics, which may be interpreted as the chance-corrected proportion of agreement between testers.
Eight newly-trained physiotherapists and one physiotechnician took part in the study. In the early stages of the study some areas of weak agreement were identified and correct assessment technique was reviewed, particularly for the eye. Good to very good reliability (Kw 0.62 to 0.99) was found for all sensory tests and most muscle strength tests. The only lower Kw scores (0-48 to 0-59, suggesting only moderate reliability) were for the VMT of muscles supplied by the median nerve in one of the study's two field centres. Even in this case, testers never varied by more than one grade, but calculation of Kw was negatively influenced by a lack of variation among the subjects. In addition, testers never varied by more than one grade from the gold standard.
Even though all testers were professionally trained and received additional specific training and practice in MF and VMT testing, discrepancies in technique required an early review and correction. This fact highlights the need for careful training and formal reliability testing. This should extend to referral centres where staff are involved in assessing the symptoms of reaction and monitoring response to treatment. Reliability testing provides the opportunity to address important discrepancies in technique that may persist even in the presence of protocols and qualified and trained staff. It is therefore a valuable tool as part of a training procedure for situations, where patients may be assessed by different testers. Overall, our results were deemed good enough to proceed with the INFIR study, using VMT and MF testing as a baseline against which to compare more sophisticated methods of nerve function testing.

Full text

Page 1

Ensuring inter-tester reliability of voluntary muscle
and monofilament sensory testing in the INFIR
Cohort Study
ANNE E. ROBERTS*, PETER G. NICHOLLS**,
PRANAVA MADDALI* & WIM H. VAN BRAKEL***
*The Leprosy Mission Trust India, CNI Bhavan, 11 Pandit Pant
Marg, New Delhi, India
**Department of Public Health, University of Aberdeen, Foresterhill,
Aberdeen, AB25 2ZD, Scotland, UK
***KIT Leprosy Unit, Wibaustraat 137J, 1097 DN Amsterdam, The
Netherlands
Accepted for publication 15 March 2007
Summary
Objective
strength (VMT) testing carried out by nine physiotherapy staff recruited for the ILEP
Nerve Function Impairment & Reaction (INFIR) Cohort Study in India.
DesignA multiple pair inter-tester reliability study was carried out in Uttar
Pradesh, India. Newly trained testers were paired up with an experienced
physiotherapist, whose assessment served as the gold standard. Each pair completed
a series of assessments. All testers had undertaken a week of specific VMT and MF
training, followed by a month of practice in the hospital setting. Reliability was
assessed by calculating weighted Kappa (Kw) statistics, which may be interpreted as
the chance-corrected proportion of agreement between testers.
ResultsEight newly-trained physiotherapists and one physiotechnician took part in
the study. In the early stages of the study some areas of weak agreement were
identified and correct assessment technique was reviewed, particularly for the eye.
Good to very good reliability (Kw 0·62 to 0·99) was found for all sensory tests and
most muscle strength tests. The only lower Kw scores (0·48 to 0·59, suggesting only
moderate reliability) were for the VMT of muscles supplied by the median nerve in
one of the study’s two field centres. Even in this case, testers never varied by more
than one grade, but calculation of Kw was negatively influenced by a lack of variation
among the subjects. In addition, testers never varied by more than one grade from the
gold standard.
ConclusionEven though all testers were professionally trained and received
additional specific training and practice in MF and VMT testing, discrepancies in
technique required an early review and correction. This fact highlights the need for
careful training and formal reliability testing. This should extend to referral centres
To assess the reliability of monofilament (MF) and voluntary muscle
Correspondence to: Wim van Brakel, KIT Leprosy Unit, Wibaustraat 137J, 1097 DN Amsterdam, The
Netherlands (E-mail: w.v.brakel@kit.nl)
Lepr Rev (2007) 78, 122–130
0305-7518//064053+09 $1.00
q Lepra
122

Page 2

where staff are involved in assessing the symptoms of reaction and monitoring
response to treatment. Reliability testing provides the opportunity to address
important discrepancies in technique that may persist even in the presence of
protocols and qualified and trained staff. It is therefore a valuable tool as part of a
training procedure for situations, where patients may be assessed by different testers.
Overall, our results were deemed good enough to proceed with the INFIR study,
using VMT and MF testing as a baseline against which to compare more sophisticated
methods of nerve function testing.
Introduction
The INFIR cohort study is a multi-centre, prospective cohort study of 303 multibacillary
leprosy patients in North India, followed for two years. The objectives of the study are to
identify predictors of neuropathy and reactions in leprosy, to determine which methods of
assessment are most sensitive for detecting peripheral neuropathy, and to study the
pathogenesis of neuropathy and reactions. The battery of neurological tests includes sensory
and motor nerve conduction, assessments of warm and cold sensation and vibration
perception thresholds, and muscle strength testing using dynamometry. The predictive value
of these neurological assessments is assessed by comparison with assessments of sensory
function using Semmes Weinstein monofilaments (MF) and assessments of motor function
using Voluntary Muscle Testing (VMT). A description of the study cohort at baseline has
been published.1
Assessments made by monofilament and voluntary muscle testing are the established
methods used to assess sensory and motor function of peripheral nerves in leprosy research.2
These methods can detect changes in nerve function and provide information that supports
the clinical diagnosis of reactions. This paper describes the combination of training given to
physiotherapists and physio technicians recruited to work in the INFIR cohort study and the
inter-tester reliability testing that demonstrated acceptable levels of agreement between
testers in their use of monofilaments and voluntary muscle testing.
Several publications have reported good inter-tester reliability for monofilament and
voluntary muscle testing.2–6The use of monofilament testing is becoming more common in
referral centres in South Asia and has been widely promoted in Brazil.7However, it is
important to verify the reliability of testing within each new setting, specifically in the
research setting.3In addition, reliability testing can be used as a tool in the training of
technicians. For this purpose, we compared the results of each technician with that of
someone with extensive experience in using these methods (the ‘gold standard’). We believe
this is the first time this method has been published and that these are the first VMT and MF
reliability data to be published from India.
Study Methods and Population
Training and reliability testing was undertaken during the period May to July 2000. The
testers involved were all newly recruited to work in the INFIR cohort study and included
eight recently graduated physiotherapists and one physiotherapy technician. Four had worked
for one year prior to recruitment, all outside the field of leprosy. Five were located in Naini
and four in Faizabad at referral hospitals run by The Leprosy Mission International (TLMI) in
Uttar Pradesh State in North India.
Inter-tester reliability in the INFIR Cohort Study123

Page 3

The testers received a full week of intensive training in the specific application of MF and
VMT to leprosy patients, and then spent a month assessing leprosy patients within the local
hospital setting prior to undertaking the reliability study. For sensory testing on the hands
they used a set of five monofilaments of 200mg, 2g, 4g, 10g and 300g respectively. For
testing of feet they used the four monofilaments from 2g up to 300g. Three test points were
used on bilateral ulnar nerves and median nerve and four test points on bilateral posterior
tibial nerves. They also assessed sensation of bilateral radial cutaneous nerves and the
bilateral sural nerves (results not presented here). The sites and monofilaments used for
sensory testing are detailed in Appendix 1. Bilateral voluntary motor testing assessed strong
closure of the eyes, little finger abduction (abductor digiti minimi; ADM), palmar abduction
of the thumb (abductor pollicis brevis; APB), wrist extension and foot dorsiflexion. In each
test muscle strength was graded using the modified Medical Research Council (MRC) six-
point scale of 0–5 with 5 representing normal strength.
For the formal reliability testing, subjects were recruited from individuals attending
leprosy, dermatology or general clinics who agreed to participate in the study. Subjects were
included even if one or more limbs were unavailable for testing due, for example, to the
presence of a plaster cast, an ulcer or limb absorption/amputation. Subjects were excluded if
they could not understand the instructions for testing.
To validate results as well as measure reliability, the assessments by each of the nine
testers were compared with those of a physiotherapist with 4 years experience who provided a
‘gold standard’. In turn, the accuracy of the gold standard assessment, referred to here as
‘Tester 1’, was assessed informally by comparison with assessments made by a consultant
physiotherapist with more than 20 years experience in VMT and MF testing of leprosy
patients. In each centre, testing was carried out in a screened area or separate rooms. Tester 1
and from two to five of the other testers tested each subject. The order in which tests were
conducted was randomised, as was the order in which testers made assessments. Testing took
from 15 to 20 minutes per tester to complete. Testers were blinded to the prior results of
others. Separate data forms identifying the subject and the tester were filled for every test.
The formal reliability testprovided the opportunity tomonitor and assess the work of each
tester. In the early stages some important discrepancies in results were identified, the reasons
discussed and testing techniques reviewed. Data from these early subjects was retained and is
included in the analysis presented here. This process of monitoring and training continued
throughout reliability testing.
In order to measure reliability, we adopted the procedure described by Altman,8using the
weighted Kappa statistic (Kw) as reliability coefficient. Kw may be interpreted as the
proportion of agreement between testers, corrected for the effect of chance. The computation
yields a value for Kw in the range 0 to 1, where 0 reflects agreement no better than that arising
purely by chance and 1 reflects perfect agreement. According to Altman, a Kw of 0·60 or
greater indicates good agreement, and a Kw in excess of 0·80 indicates very good agreement.
Where there is little variation in assessments the value of kappa cannot be calculated. We
chose to use quadratic weights that attach greater importance to large differences. In addition,
differences between pairs of testers were examined using analysis of variance (excluding the
gold standard assessment). Data was entered into a purpose-written database using Microsoft
Access. We used STATA analysis procedures to compute Kw based on the comparison of
assessments by each individual tester and that of the gold standard tester.
A. E. Roberts et al.124

Page 4

Ethical approval for all aspects of the INFIR study was obtained from the Indian Council
of Medical Research through the Research Ethics Committee of the Central JALMA Institute
for Leprosy in Agra.
Results
In total, 56 subjects were assessed at Naini and 55 at Faizabad. The ranges of test scores
obtained by the ‘gold standard tester’ in this reliability study are shown in Table 1.
The reliability statistics of monofilament testing of ulnar, median and posterior tibial
nerves are summarised in Table 2, while the statistics of voluntary muscle testing of
musculature supplied by facial, ulnar, median, radial and lateral popliteal nerves are
presented in Table 3.
For monofilament testing we found Kw for comparisons between individual testers and
the gold standard tester ranged from 0·75 upwards, reflecting agreement that was good to very
good. The percentage of agreement to within one grade ranged from 61% to 94%.
Table 1. Frequencies of ‘gold standard’ test scores in both study centres in Uttar Pradesh, India; data presented as
number of cases (percentage)
Nerves (scale)Monofilament scores FaizabadNainiTotal
Ulnar
(0–15)
0
1–2
3–7
8 þ
0
1–2
3–7
8 þ
0
1–2
3–7
8 þ
11 (48)
4 (17)
2 (8·7)
6 (26)
16 (70)
0
5 (22)
2 (8·7)
8 (35)
2 (8·7)
4 (17)
9 (39)
26 (48)
4 (7·4)
3 (5·6)
21 (39)
28 (52)
8 (15)
7 (13)
11 (20)
18 (33)
1 (1·9)
12 (22)
23 (43)
37 (48)
8 (10)
5 (6·5)
27 (35)
44 (57)
8 (10)
12 (16)
13 (17)
26 (34)
3 (3·9)
16 (21)
32 (42)
Median
(0–15)
Post. Tibial
(0–20)
VMT scores
0–1
2–3
4
5
0–1
2–3
4
5
0–1
2–3
4
5
0–1
2–3
4
5
0–1
2–3
4
5
Facial
(0–5)
0
0
0
2 (5·0)
5 (13)
6 (15)
27 (68)
8 (20)
0
6 (15)
26 (65)
15 (38)
4 (10)
10 (25)
11 (28)
0
0
0
40 (100)
4 (8·7)
6 (13)
4 (8·7)
32 (70)
2 (3·2)
5 (7·9)
6 (9·5)
50 (79)
8 (13)
1 (1·6)
11 (18)
42 (68)
18 (29)
5 (8·1)
15 (24)
24 (39)
23 (100)
0
1 (4·6)
5 (23)
16 (73)
3 (14)
1 (4·6)
5 (23)
13 (59)
0
0
1 (4·6)
21 (95)
2 (9·5)
2 (9·5)
0
17 (81)
Median
(0–5)
Ulnar
(0–5)
Radial
(0–5)
1 (1·6)
61 (98)
6 (8·9)
8 (12)
4 (6·0)
49 (73)
Peroneal
(0–5)
Inter-tester reliability in the INFIR Cohort Study125

Page 5

Table 2. Agreement between testers and the resulting weighted Kappas for monofilament testing
Monofilament testing
Ulnar nerveMedian nerve Posterior tibial nerve
Tester
Number of
data pairs
with tester 1
Number
with direct
agreement
%
agreement
within 1
grade
Weighted
Kappa
95 %
confidence
interval
Number
with direct
agreement
%
agreement
within 1
grade
Weighted
Kappa
95 %
confidence
interval
Number
with direct
agreement
%
agreement
within 1
grade
Weighted
Kappa
95 %
confidence
interval
2
3
4
5
6
7
8
9
10
28
60
74
64
78
32
24
34
32
9/28
29/60
36/74
29/64
46/78
18/32
16/24
21/34
19/32
61
65
70
63
74
72
79
74
75
0·96
0·94
0·96
0·96
0·96
0·84
0·75
0·89
0·89
0·59–1·0
0·69–1·0
0·73–1·0
0·71–1·0
0·74–1·0
0·49–1·0
0·37–1·0
0·56–1·0
0·55–1·0
14/28
34/60
41/74
35/64
50/78
24/32
20/24
26/34
23/32
75
80
76
83
81
84
88
88
84
0·97
0·93
0·92
0·96
0·95
0·90
0·79
0·80
0·86
0·60–1·0
0·68–1·0
0·69–1·0
0·72–1·0
0·73–1·0
0·56–1·0
0·42–1·0
0·48–1·0
0·51–1·0
17/28
30/60
39/74
39/64
43/78
24/32
16/24
25/34
21/32
68
67
64
73
68
94
79
85
81
0·89
0·76
0·80
0·92
0·88
0·93
0·88
0·93
0·97
0·52–1·0
0·51–1·0
0·57–1·0
0·67–1·0
0·66–1·0
0·58–1·0
0·49–1·0
0·60–1·0
0·63–1·0
A. E. Roberts et al.
126