Content uploaded by Mustafa Atee
Author content
All content in this area was uploaded by Mustafa Atee on Nov 23, 2020
Content may be subject to copyright.
© 2018 S. Karg er AG, Basel
Original Research Article
Dement Geriatr Cogn Disord
Psychometric Evaluation of the Electronic
Pain Assessment Tool: An Innovative Instrument
for Individuals with Moderate-to-Severe
Dementia
Mustafa Atee a Kreshnik Hoti a, b Jeffery D. Hughes a
a School of Pharmacy, Faculty of Health Sciences, Curtin University, Bentley, WA, Australia;
b Division of Pharmacy, Faculty of Medicine, University of Pristina, Pristina, Kosovo
Keywords
Psychometric properties · Validity · Reliability · Electronic Pain Assessment Tool · Pain ·
Dementia · Automated facial recognition · Facial action units · Automated facial analysis ·
Observational pain scales · Application
Abstract
Background/Aims: Pain is common in aged care residents with dementia; yet it often goes
undetected. A novel tool, the electronic Pain Assessment Tool (ePAT), was developed to ad-
dress this challenging problem. We investigated the psychometric properties of the ePAT.
Methods: In a 10-week prospective observational study, the ePAT was evaluated by compar-
ison against the Abbey Pain Scale (APS). Pain assessments were blindly co-performed by the
ePAT rater against the nursing staff of two residential aged care facilities. The residents were
assessed twice by each rater: at rest and following movement. Results: The study involved 34
residents aged 85.5 ± 6.3 years, predominantly with severe dementia (Psychogeriatric Assess-
ment Scale – Cognitive Impairment score = 19.7 ± 2.5). Four hundred paired assessments
(n = 204 during rest; n = 196 following movement) were performed. Concurrent validity (r =
0.911) and all reliability measures (κw = 0.857; intraclass correlation coefficient = 0.904; α =
0.950) were excellent, while discriminant validity and predictive validity were good. Conclu-
sion: The ePAT is a suitable tool for the assessment of pain in this vulnerable population.
© 2018 S. Karg er AG, Basel
Accepted: November 15, 2017
Published online: ■■■
Mustafa Atee
School of Pharmac y, Curtin University
PO Box U1987
Perth, WA 6845 (Australia)
E-Mail Mustafa.Atee @ curtin.edu.au
www.karger.com/dem
DOI: 10.1159/000485377
Downloaded by:
Curtin University Library
134.7.195.166 - 1/24/2018 2:26:22 AM
257
Dement Geriatr Cogn Disord
Atee et al.: Psychometric Evaluation of the Electronic Pain Assessment Tool: An
Innovative Instrument for Individuals with Moderate-to-Severe Dementia
www.karger.com/dem
© 2018 S. Karger AG, Basel
DOI: 10.1159/000485377
Introduction
A reduced self-reporting capacity in individuals with dementia is common due to disease-
related progressive cognitive impairment [1]. In the absence of a valid and reliable self-report
rating (i.e., gold standard), observational pain assessment tools are an appropriate
replacement to evaluate pain for nonverbal populations such as those with advanced dementia
[2, 3]. A study by Lukas et al. [4] concluded that the use of these tools improved pain recog-
nition (by up to 25.4%) and rating (by up to 42.5%) in older adults with cognitive impairment.
Because ageing predisposes individuals to a number of comorbidities, pain is very
common. In aged care facilities, up to 80% of residents experience pain at some stage during
their stay [5, 6]. In this setting, dementia affects more than 50% of aged care residents in
Australia and Germany [7, 8], and 69% in the UK [9, 10]. In the USA, over 61% of aged care
residents have moderate-to-severe cognitive impairment [11]. Nursing staff have reported
difficulty in detecting pain in these residents, and even with experience they remain poor at
interpreting facial expressions indicative of pain [12, 13]. Moreover, there is evidence that
even interdisciplinary evaluation of pain fails to assess pain correctly [14].
The importance of identifying pain in this population is critical to optimal pain
management. Failure to do so may result in a denial of appropriate medication or therapeutic
intervention, prescription of analgesics at inadequate dosages, or use of inappropriate medi-
cation (e.g., antipsychotics) [15–20]. These may lead to poor quality of life and premature
death in these individuals [21–25]. Assessing pain requires a tool with sound psychometric
properties and innovative characteristics so that timely access to appropriate pain
management is assured.
Currently, there are more than 35 observational-behavioral pain assessment tools
targeted at people with dementia [26]. Many of these tools have been criticized in a number
of reviews for being inadequate due to the paucity of systematic evaluation of their psycho-
metric properties [27–29]. Despite the abundance of tools for this population, they still suffer
from lack of innovation [27]. Since the face is the richest source of behavioral expressions, a
prudent approach is to integrate objective facial measures such as Ekman’s Facial Action
Coding System (FACS) into these tools [29, 30]. The FACS is an annotated catalogue of facial
micro-expressions which describes each facial muscle action with a unique code called an
action unit (AU) [31]. For instance, AU4 is brow lowering, while AU6 is a cheek raiser.
Current research supports this approach, which is why we have developed the electronic
Pain Assessment Tool (ePAT), a novel instrument of great potential to transform the process
of pain assessment in dementia [29, 32, 33]. The ePAT is a point-of-care tool that uses a hybrid
model: automated facial recognition and analysis, digitization, and clinical observations [33].
The tool is built as a software application that is compatible with Android and iOS smart
devices (see online suppl. File 1; see www.karger.com/doi/10.1159/000485377 for all online
suppl. material, ). Figure 1 illustrates the steps involved in pain assessment using the ePAT.
Previous psychometric evaluation of the prototype version of the app (Android V3.0) demon-
strated that ePAT had excellent concurrent validity and internal consistency when compared
to the Abbey Pain Scale (APS), while possessing good discriminant validity and interrater reli-
ability [33]. These data were evaluated using a mix of raters including nurses, health science
students, and care workers [33].
In this study the Android V4.0 app was evaluated in the setting of aged care in comparison
with APS assessments undertaken by nursing staff. The aim of the study was to evaluate the
psychometric properties (concurrent validity, discriminant validity, predictive validity,
internal consistency, and interrater reliability) of the ePAT (Android V4.0) in residents with
moderate-to-severe dementia, and to compare these findings to those reported in our
previous study.
Downloaded by:
Curtin University Library
134.7.195.166 - 1/24/2018 2:26:22 AM
258
Dement Geriatr Cogn Disord
Atee et al.: Psychometric Evaluation of the Electronic Pain Assessment Tool: An
Innovative Instrument for Individuals with Moderate-to-Severe Dementia
www.karger.com/dem
© 2018 S. Karger AG, BaselDOI: 10.1159/000485377
Methods
Study Design and Setting
This prospective observational study involved residents recruited from two accredited residential aged
care facilities (RACFs), located in the Perth metropolitan area, Western Australia. Table 1 provides a summary
of the characteristics of each RACF.
Procedure
The study was approved by the Human Research Ethics Committee of Curtin University, Western
Australia (HREC: HR10/2014) and by Mercy Health Care (R15/50AC). This study also had Clinical Trial Noti-
fication (CTN) approval (CT-2016-CTN-04886-1 v1) from Therapeutic Goods Administration (TGA). Written
invitations and information sheets were sent to authorized representatives to provide consent on behalf of
residents as they were deemed incompetent of providing their own informed written consent. All assess-
ments were completed in accord with the Declaration of Helsinki, as well as policy statements of the
Alzheimer’s Association and the World Health Organization on assistive technologies for people with
dementia. All data were de-identified and protected to maintain confidentiality.
In this study, the ePAT (Android V4.0, Samsung Note 3-N9005) was evaluated in comparison with the
APS in two RACFs. The study lasted for a period of 10 weeks, which was commenced in January and completed
in April 2017. Weekly pain ratings of residents were made concurrently by two independent raters, one using
the ePAT and the other using the APS. The ePAT rater was the primary investigator (M.A.), who is experienced
with the use of the tool, while the APS was administered by a member of the nursing staff of the two RACFs.
The raters were required to complete a questionnaire to ascertain their familiarity (i.e., prior experience
or exposure) with the APS. Before commencing the study, a test case was run and discussed between the
ePAT rater and the APS raters to cement understanding of the protocol. Paired ratings were undertaken
indoors during standard care in the afternoon hours of 1–4 p.m. The automated facial assessments with the
ePAT were done under ambient lighting conditions and within 1 m of the resident’s face. Each rater was
blinded to the other rater’s assessment/scoring and also to the drug and non-drug pain therapies received
by the residents under assessment during that day. At each encounter, the residents were assessed twice by
Step 1: The Face (Domain 1) Step 2: The Voice (Domain 2) Step 3: The Movement (Domain 3) Step 4: The Behavior (Domain 4)
Step 5: The Activity (Domain 5) Step 6: The Body (Domain 6) Step 7: Computation of Pain Scores
Fig. 1. Pain assessment process using the electronic Pain Assessment Tool (ePAT). Facial analysis is auto-
mated (step 1) in domain 1 (The Face), the app user then completes the checklists for the other 5 domains
(steps 2–6), and the app then calculates a total pain score and pain severity score (step 7).
Color version available online
Downloaded by:
Curtin University Library
134.7.195.166 - 1/24/2018 2:26:22 AM
259
Dement Geriatr Cogn Disord
Atee et al.: Psychometric Evaluation of the Electronic Pain Assessment Tool: An
Innovative Instrument for Individuals with Moderate-to-Severe Dementia
www.karger.com/dem
© 2018 S. Karger AG, Basel
DOI: 10.1159/000485377
each rater: at rest to replicate comfort conditions, followed by movement to instigate nociceptive experi-
ences. The residents were clinically stable during these assessments, which were carried out under similar
testing conditions (e.g., lighting). A similar methodology was used by the authors in a previous study [33].
Pain Assessments
The APS (see online suppl. File 2) was used as a point of reference in this study because it is widely used
in Australia for patients with dementia who cannot verbalize, has been translated into various languages, and
has sound psychometric properties [3, 27, 34–36]. The APS consists of 6 observational domains: vocalization,
facial expressions, change in body language, behavioral change, physiological change, and physical change
[37]. Scoring of the tool involves ordinal ratings that range from 0 to 3 to indicate the intensity in each
domain, where 0 = not present, 1 = mild, 2 = moderate, and 3 = severe. Total pain intensity is calculated after
adding all domain scores, which is then categorized based on the following cutoff points: 0–2 = no pain,
3–7 = mild pain, 8–13 = moderate pain, and ≥14 = severe pain [37].
The ePAT is the measure of interest in this study. Its psychometric data were investigated in comparison
with those from the APS. The ePAT is a point-of-care app that utilizes automated facial recognition and
analysis to detect pain-related facial AUs, which are then used in combination with other clinical indicators
to calculate a pain intensity score [33]. The tool has a total of 42 descriptors, which were selected based on
their association with pain according to the literature. Each descriptor is scored on a binary scale of yes = 1
and no = 0. These descriptors are contained within 6 domains: The Face, The Voice, The Movement, The
Behavior, The Activity, and The Body (Fig. 1). A summation of the domain scores results in a total numerical
score, which conforms to the following pain categories: no pain (0–6), mild pain (7–11), moderate pain
(12–15), and severe pain (≥16) [33]. A full description of the tool’s contents, scoring, and conceptual foun-
dation has been published elsewhere [33].
Cognitive Assessments
Cognition of the residents was measured using the Psychogeriatric Assessment Scale – Cognitive
Impairment Scale (PAS-Cog). The PAS-Cog is an informant-administered cognition scale which has been vali-
dated for various populations [38–40]. The scale is also endorsed by the Australian Government as a tool for
cognitive assessment for the purposes of aged care funding [41]. The cognitive scores of the residents were
extracted from their electronic profile data. Scores indicative of the stage of severity of cognitive impairment
are as follows: 0–3 = minimal, 4–9 = mild, 10–15 = moderate, and 16–21 = severe [38].
Resident s
Residents were selected for participation if they were older than 65 years; had been living in the facility
for at least 3 months; had been diagnosed with dementia by a geriatrician; had been classified as having
moderate-to-severe dementia based on a PAS-Cog score of >10; and had a medical history or presenting
complaint(s) that involved painful conditions.
Residents were excluded from the study if they were deemed medically unfit to participate (as deter-
mined by the treating physician); had a facial deformity; or were unable to display facial expressions.
Table 1. Characteristics of the RACFs involved in the study
RACF 1 RACF 2
Location of facility Metropolitan Metropolitan
State Western Australia Western Australia
Type of facility Residential Residential
Ownership of facility Non-for-profit Non-for-profit
Bed capacity 67 83
Clinical staff 4 registered nurses
1 enrolled nurse
1 clinical nurse
3 registered nurses
1 enrolled nurse
Re-accreditation due date July 2018 July 2018
RACF, residential aged care facility.
Downloaded by:
Curtin University Library
134.7.195.166 - 1/24/2018 2:26:22 AM
260
Dement Geriatr Cogn Disord
Atee et al.: Psychometric Evaluation of the Electronic Pain Assessment Tool: An
Innovative Instrument for Individuals with Moderate-to-Severe Dementia
www.karger.com/dem
© 2018 S. Karger AG, BaselDOI: 10.1159/000485377
APS Raters
APS raters were staff employed by the facility who were recruited into the study if they had been working
in the facility for at least 3 months to ensure familiarity with the residents; were familiar with or cognizant of
the use of the APS; were able to converse in English; were trained or registered nurses; were involved in
providing direct care to residents; and were willing to participate (conferred by providing a written consent).
Staff were excluded from participation if their duties did not include pain assessment and management
or if they were likely to be absent during the course of the study.
Statistical Analyses
Descriptive statistics were used in this study to provide a summary of variables for the residents, raters,
and assessments. Psychometric measures of validity and reliability were calculated using appropriate statis-
tical tests.
Concurrent validity was evaluated using Pearson’s (r) correlation coefficient between the two instru-
ments (ePAT vs. APS) for each of the following instances: at rest, with movement, and overall (i.e., at rest or
with movement). The strength of correlation is denoted by the following increment values: none to weak,
0.00–0.25; weak to moderate, 0.26–0.50; moderate to strong, 0.51–0.75; and strong to extremely strong,
0.76–0.99 [42].
Discriminant validity was examined through conducting a comparative analysis between ePAT scores
and APS scores for all residents under conditions of rest as opposed to movement. Regression analysis was
selected to show the difference in pain scores using a mixed model. The dependent variable was the difference
in pain scores, while the independent variable was timing (rest or with movement). The p value associated
with the timing of an activity – e.g., sitting (representative of rest) versus exercise (representative of
movement) – demonstrated whether the correlation of a difference in scores was activity dependent.
Predictive validity was assessed using t statistics of the mean pain scores. Other central tendency
measures (median and mode) for pain scores at rest compared to those following movement activities were
also reported.
Reliability measures explored in this study included internal consistency, interrater reliability, and the
intraclass correlation coefficient (ICC). Internal consistency determined whether the two instruments were
measuring the same construct, i.e., pain. Cronbach’s α was used to assess the internal consistency, with values
of ≥0.7 considered illustrative of good agreement between the two tools [43]. Interrater reliability was
assessed by measuring agreement using Cohen’s κ on the categorical pain scores: no pain, mild pain, moderate
pain, and severe pain. Weighted κ (κw) was used to determine the overall agreement on ordinal scores. Inter-
pretation of the κ values ranged from poor (0.0) to perfect (1.0), with moderate agreement in the range of
0.41–0.60 [44]. ICC values were calculated for continuous pain scores. Interpretation of the ICC values
followed the same guidelines as the correlation coefficients above [42].
Statistical significance was expressed as p values (<0.05) or 95% confidence intervals (CI). All analyses
were performed using the Statistical Package for the Social Sciences (SPSS) version 22 (SPSS, Inc., Apache
Software Foundation, Chicago, IL, USA).
Results
Resident Sample
Thirty-seven residents with differing dementias, pain conditions, and genders were
consented (through their proxies) to participate in the study. Three residents dropped out
before the study commenced; of these, 1 resident died and the other 2 were discharged to
other facilities/home, leaving a final sample of 34 residents. All the residents were over 68
years of age, with the majority being female (58.8%). A total of 27 residents (79.4%) were
classified as having severe dementia according to PAS-Cog scores (mean: 19.7 ± 2.5; median:
21; range: 11–21). Table 2 provides the clinical and demographic data on the residents.
APS Raters
The APS raters were nursing staff employed in one or the other of the two facilities with
a mean age of 29.4 ± 6.8 years. Four of the 5 raters were female. The raters were nurses with
Downloaded by:
Curtin University Library
134.7.195.166 - 1/24/2018 2:26:22 AM
261
Dement Geriatr Cogn Disord
Atee et al.: Psychometric Evaluation of the Electronic Pain Assessment Tool: An
Innovative Instrument for Individuals with Moderate-to-Severe Dementia
www.karger.com/dem
© 2018 S. Karger AG, Basel
DOI: 10.1159/000485377
varying roles (1 clinical nurse and 4 registered nurses) and years of experience (2.8 ± 0.8).
All 4 registered nurses were employed in the same facility (RACF 1), while the clinical nurse
was working in the other facility (RACF 2). All nurses had received pain education in the past
and were familiar with the use of the APS. Table 3 describes the demographic characteristics
of the APS raters.
Clinical Pain Assessments
The residents had a total of 400 paired pain assessments conducted over the period of
the study. Of those, 204 assessments were performed during rest, whereas 196 assessments
were done following movement. All assessments were undertaken during activities of daily
living such as sitting, walking, and repositioning.
Table 2. Clinical and demographic characteristics of the resident sample at baseline (n = 34) from the two
aged care facilities
Mean age (SD), years 85.5 (6.3)
Median age (range), years 83.8 (68–93.2)
Gender, n (%)
Female 20 (58.8)
Male 14 (41.2)
Ethnicity, n (%)
Caucasian 33 (97.1)
Other 1 (2.9)
Country of birth, n (%)
Australia 24 (70.8)
UK 5 (14.7)
Italy 1 (2.9)
The Netherlands 1 (2.9)
Poland 1 (2.9)
South Africa 1 (2.9)
USA 1 (2.9)
Primary language, n (%)
English 33 (97.1)
Italian 1 (2.9)
Secondary language, n (%)
Afrikaans 1 (2.9)
Dutch 1 (2.9)
Polish 1 (2.9)
Ukrainian 1 (2.9)
None 30 (88.4)
Mobility, n (%)
Fully ambulant 1 (2.9)
Ambulant with assistance 19 (55.9)
Non-ambulant 14 (41.2)
Mean dementia severity (PAS-Cog) score (SD) 19.7 (2.5)
Median PAS-Cog score (range) 21 (11–21)
Diagnosis of dementia, n (%)
Alzheimer disease 12 (35.3)
Unspecified dementia 15 (44.1)
Frontotemporal dementia 1 (2.9)
Lewy body dementia 2 (5.9)
Parkinson dementia 2 (5.9)
Vascular dementia 2 (5.9)
PAS-Cog, Psychogeriatric Assessment Scale – Cognitive Impairment Scale
Downloaded by:
Curtin University Library
134.7.195.166 - 1/24/2018 2:26:22 AM
262
Dement Geriatr Cogn Disord
Atee et al.: Psychometric Evaluation of the Electronic Pain Assessment Tool: An
Innovative Instrument for Individuals with Moderate-to-Severe Dementia
www.karger.com/dem
© 2018 S. Karger AG, BaselDOI: 10.1159/000485377
Validity
Concurrent Validity
All correlation coefficients had high positive values, indicating excellent concurrent
validity of the ePAT when compared with the APS (Table 4). The association between ePAT
and APS total scores is also shown in Figure 2. The graph demonstrates the strong positive
correlation between the two instruments.
Discriminant Validity
The statistical difference (expressed by the p value) was computed to demonstrate
whether the timing of activity (rest vs. movement) had any effect on the correlation between
the ePAT and the APS. It was found that the correlation between the two instruments was not
situation dependent (p = 0.243).
Predictive Validity
Both tools showed increased pain scores as a consequence of movement. On the ePAT,
the pain scores were significantly higher (p < 0.0001) with movement (mean: 11.44 ± 3.54;
median: 11; mode: 13) than at rest (mean: 8.33 ± 3.34; median: 9; mode: 10). Similarly, the
APS had significantly higher pain scores (p < 0.0001) following movement (mean: 6.96 ± 3.85;
median: 7; mode: 8) than at rest (mean: 4.34 ± 3.14; median: 4; mode: 1).
Table 3. Demographic characteristics of the APS raters (n = 5)
n (%) Mean (SD) Median (range)
Age, years 29.4 (6.8) 28 (24–41)
Gender (female) 4 (80)
Ethnicity
Caucasian 1 (20)
Asian 4 (80)
Primary language (English) 5 (100)
Employment status, h 35.4 (3.2) 36 (30–38)
Part time (<38 h) 3 (60)
Full time (≥38 h) 2 (40)
Years of experience
Nursing 2.8 (0.8) 3 (2–4)
Aged care 2.4 (0.5) 2 (2–3)
Cognitive impairment/dementia care 2.4 (0.5) 2 (2–3)
Employment in facility 1.6 (0.9) 1 (1–3)
Role in facility
Clinical nurse 1 (20)
Registered nurse 4 (80)
Past pain education
Yes 5 (100)
No 0 (0)
Last received pain education
<3 months 2 (40)
<12 months 1 (20)
>12 months 2 (40)
Familiarity with the APS
Yes 5 (100)
No 0 (0)
APS, Abbey Pain Scale.
Downloaded by:
Curtin University Library
134.7.195.166 - 1/24/2018 2:26:22 AM
263
Dement Geriatr Cogn Disord
Atee et al.: Psychometric Evaluation of the Electronic Pain Assessment Tool: An
Innovative Instrument for Individuals with Moderate-to-Severe Dementia
www.karger.com/dem
© 2018 S. Karger AG, Basel
DOI: 10.1159/000485377
Reliability
Interrater Reliability (Interrater Agreement)
Overall agreement on the categorical pain scores was excellent (κw = 0.857; 95% CI:
0.819–0.895). Greater agreement among raters was found during rest (κ = 0.840; p = 0.000)
compared to movement (κ = 0.772; p = 0.000).
Intraclass Correlation Coefficient
As a single measure, the ICC value of the ePAT was excellent (0.904; 95% CI: 0.885–0.921).
During rest, the ICC value was 0.902 (95% CI: 0.872–0.925), while following movement the
ICC was 0.879 (95% CI: 0.843–0.908). Both of these values fall within the “excellent” range.
Internal Consistency
Overall internal consistency (Cronbach’s α) of the ePAT when compared to the APS was
0.950, which is classified as excellent. The α values were greater for movement (α = 0.797)
than for the rest condition (α = 0.766).
Discussion
The findings of this validation study provide further evidence of the psychometric prop-
erties of the ePAT which make it a suitable instrument for the assessment of pain in people
with moderate-to-severe dementia who cannot verbalize pain. The significance of the current
research was to demonstrate the robustness of a previous study and to ensure the repeat-
Number of paired pain
assessments, n (%)
Pearson’s
correlation, r
Rest 204 (51) 0.896
Movement 196 (49) 0.904
Overall 400 (100) 0.911
ePAT, electronic Pain Assessment Tool; APS, Abbey Pain Scale. All
correlation values are significant at the 0.01 level (2-tailed).
15
10
5
20
0
242220181614121086420
Total score (APS)
Total score (ePAT)
Timing
Move
Rest
Table 4. Correlation values
between the ePAT and the APS
under various conditions
Fig. 2. Association between elec-
tronic Pain Assessment Tool
(ePAT) and Abbey Pain Scale
(APS) total scores. Dark-colored
dots: rest; light-colored dots:
movement. One dot may repre-
sent more than one rating score.
Downloaded by:
Curtin University Library
134.7.195.166 - 1/24/2018 2:26:22 AM
264
Dement Geriatr Cogn Disord
Atee et al.: Psychometric Evaluation of the Electronic Pain Assessment Tool: An
Innovative Instrument for Individuals with Moderate-to-Severe Dementia
www.karger.com/dem
© 2018 S. Karger AG, BaselDOI: 10.1159/000485377
ability of the initial psychometric findings. These are essential criteria for adopting a tool into
clinical practice.
Unsurprisingly, there was a strong positive correlation of the ePAT with standard
assessment of pain (i.e., by the APS) in the RACFs involved in the study. This is because both
scales have a similar construct and conceptual foundation and both measure similar aspects
of pain-related behaviors. Based on the current findings, it seems that the psychometric char-
acteristics of the ePAT have improved, as they have slightly higher correlation values when
compared to previous findings [33]. This is perhaps due to variation in the APS rater samples
between the two studies. In the current study, all APS raters were nurses (with a greater expe-
rience in pain assessment), whereas they were a mix of nurses, health science students, and
care workers in the previous study [33].
The “state-of-the-art” systematic review by Herr et al. [45] set correlation values of
0.4–0.6 as an adequate range for new pain assessment scales for nonverbal populations. The
ePAT had shown excellent concurrent validity under various testing conditions (rest: r =
0.896; movement: r = 0.904). These results are much higher than the recommended range
suggested by Herr et al. [45] and in line with our previous findings [33]. Other pain assessment
tools with good-to-excellent concurrent validity are the Rotterdam Elderly Pain Observation
Scale (REPOS) (0.61–0.75) when compared to the Pain Assessment in Advanced Dementia
(PAINAD) [46], and the Rating Pain in Dementia (RaPID) scale (0.8–0.86) when compared to
the McGill Pain Questionnaire [47]. In our study, the strength of correlation between the ePAT
and the APS was slightly better during movement than at rest. A number of studies on obser-
vational tools (the Checklist of Nonverbal Pain Indicators [CNPI], FACS, Mobilization-Obser-
vation-Behaviors-Intensity Dementia Pain Scale [MOBID], and Pain Behavior Measurement
[PBM]) have shown parallel findings in this regard [48–50].
All pain scores recorded by the ePAT following movement were higher than those under
rest conditions. Similarly, the APS had greater scores observed for each resident under the
same conditions. This is consistent with our previous work [33], which supports that the
ePAT has discriminative properties, i.e., discriminant validity. Out of 28 tools, Lichtner et al.
[27] listed only 8 (the Certified Nurse Assistant Pain Tool [CPAT], CNPI, Discomfort Scale –
Dementia of the Alzheimer’s Type [DS-DAT], Pain Assessment Checklist for Seniors with
Limited Ability to Communicate [PACSLAC], MOBID, APS, Assessment Discomfort in Dementia
[ADD] Protocol, and Behavior checklist) which showed evidence of discriminant validity.
There were also significant differences in mean ePAT and APS scores at rest (treated as
preintervention) and following movement (treated as postintervention), providing support
for the predictive validity. Abbey et al. [37] reported the predictive validity of the APS as a
significant change in mean pain scores before and after interventions.
The properties of reliability were excellent, as demonstrated by the overall values of α, κ,
and ICC. Except for internal consistency (α values), all reliability measurement values were
higher for rest than for movement. This is probably because when a resident is at rest, he/she
has fewer behaviors recorded by raters; hence the raters have a lower chance of disagreement.
Internal consistency is a reliability measure of the correlation of the subscales of an instrument
(i.e., the ePAT) to assess the construct of interest (i.e., pain behaviors). Raters may have
various interpretations as to whether certain behaviors are related to pain or part of neuro-
psychiatric symptoms of dementia. There is an overlap between pain and these symptoms,
and there is evidence to suggest that pain is linked to depressive symptoms in residents with
dementia [51, 52]. Other observational tools with excellent interrater reliability κ values
include the Pain Assessment in Noncommunicative Elderly Persons (PAINE) (0.711–0.999),
RaPID (0.97), DS-DAT (0.61–0.98), and PAINAD (0.72–0.97) [27].
Strengths of this study include the use of blinded rating and blinded knowledge of the
pain management given to reduce learning bias. Prior to the first (paired) rating, a test case
Downloaded by:
Curtin University Library
134.7.195.166 - 1/24/2018 2:26:22 AM
265
Dement Geriatr Cogn Disord
Atee et al.: Psychometric Evaluation of the Electronic Pain Assessment Tool: An
Innovative Instrument for Individuals with Moderate-to-Severe Dementia
www.karger.com/dem
© 2018 S. Karger AG, Basel
DOI: 10.1159/000485377
was used to ensure the steps of the study protocol were followed correctly. The vast majority
of assessments were matched under various conditions indicative of rest and movement. The
study was conducted in a real-world clinical environment and without interrupting the work
flow. The statistical analyses of the psychometric data were comprehensive. Important impli-
cations of using this app include (1) automated facial analysis, which has the potential to
improve the process of pain assessment through objective measures, and (2) digital docu-
mentation, which provides a pragmatic approach for use in clinical settings.
This study has similar limitations to those documented in our previous study, such as
sampling bias and reporting bias [33]. Further, cognitive evaluations were based on informant
ratings, which have their own limitations regarding subjectivity. These evaluations were
conducted at various time points prior to commencement of the study. In our sample, the
majority (79.4%) of the residents had severe cognitive impairment, as indicated by their
PAS-Cog scores. Given the fact that the course of dementia has a progressive nature, it is likely
that the cognitive status of all residents (including the remaining 20.6%) may have worsened
since it was last assessed. In addition, the clinical pain assessments were limited to certain
time periods, i.e., the afternoon, and findings may be different if they are performed during
other times.
In conclusion, this study adds further to the body of evidence regarding the psychometric
merits of the ePAT as a valid and reliable tool for the assessment of pain in residents with
moderate-to-severe dementia. Further research on technology and refinements in using the
presence and intensity of facial expressions as a means of fully automating pain assessment
are warranted.
Acknowledgements
The authors want to thank the aged care staff, residents, and their families for their involvement in the
study. Sincere thanks also go to Jenny Lalor for assisting with the statistical analyses.
Disclosure Statement
All authors are shareholders in EPAT Technologies Ltd, which is marketing the ePAT instrument (also
known as PainChekTM). They also have a patent application titled “A pain assessment method and system”
(PCT/AU2015/000501), which is currently under national phase examination since February 2, 2017. M.A.
is a Research Scientist for EPAT Technologies Ltd while serving as a Research Fellow and PhD Candidate with
the School of Pharmacy, Curtin University. K.H. is employed as a consultant by EPAT Technologies Ltd while
serving as an Assistant Professor at University of Pristina, and an Adjunct Senior Lecturer at the School of
Pharmacy, Curtin University. J.D.H. is employed as Chief Scientific Officer of EPAT Technologies Ltd while
serving as a Professor at the School of Pharmacy, Curtin University.
Funding Sources
The original research that led to the development of the ePAT tool is part of a PhD project which was
supported by the Alzheimer’s Australia Dementia Research Foundation (AADRF) through grant funding and
a stipend scholarship. The content of the article is solely the responsibility of the authors and does not neces-
sarily represent the official views of AADRF. The project has been commercialized into a spin-off start-up
company (ePAT Pty Ltd), which has been publicly listed as EPAT Technologies Ltd in the Australian Share
Securities (ASX) since October 2016. This study was sponsored by EPAT Technologies Ltd as part of its appli-
cation to list the ePAT app as a Class 1 medical device with the TGA, and receive the CE mark in Europe. The
ePAT is commercially available as PainChekTM.
Downloaded by:
Curtin University Library
134.7.195.166 - 1/24/2018 2:26:22 AM
266
Dement Geriatr Cogn Disord
Atee et al.: Psychometric Evaluation of the Electronic Pain Assessment Tool: An
Innovative Instrument for Individuals with Moderate-to-Severe Dementia
www.karger.com/dem
© 2018 S. Karger AG, BaselDOI: 10.1159/000485377
Author Contributions
All authors conceived the idea and designed the study. M.A. conducted the literature search and drafted
the manuscript. All authors reviewed the manuscript. M.A. recruited the patients and collected the data. M.A.
and J.D.H. conducted some statistical analyses and interpreted all results. All authors contributed to and
approved the final version of the manuscript.
References
1 Horgas AL, Elliott AF, Marsiske M: Pain assessment in persons with dementia: relationship between self-
report and behavioral observation. J Am Geriatr Soc 2009; 57: 126–132.
2 Malara A, De Biase GA, Bettarini F, Ceravolo F, Di Cello S, Garo M, Praino F, Settembrini V, Sgrò G, Spadea F,
Rispoli V: Pain assessment in elderly with behavioral and psychological symptoms of dementia. J Alzheimers
Dis 2016; 50: 1217–1225.
3 Neville C, Ostini R: A psychometric evaluation of three pain rating scales for people with moderate to severe
dementia. Pain Manag Nurs 2014; 15: 798–806.
4 Lukas A, Barber JB, Johnson P, Gibson SJ: Observer-rated pain assessment instruments improve both the
detection of pain and the evaluation of pain intensity in people with dementia. Eur J Pain 2013; 17: 1558–1568.
5 Tse MM, Pun SP, Benzie IF: Pain relief strategies used by older people with chronic pain: an exploratory survey
for planning patient-centred intervention. J Clin Nurs 2005; 14: 315–320.
6 van Kooten J, Binnekade TT, van der Wouden JC, Stek ML, Scherder EJ, Husebø BS, Smalbrugge M, Hertogh CM:
A review of pain prevalence in Alzheimer’s, vascular, frontotemporal and Lewy body dementias. Dement
Geriatr Cogn Disord 2016; 41: 220–232.
7 Australian Institute of Health and Welfare: Residential aged care in Australia 2010–2011: a statistical overview.
Cat No AGE 68. Canberra, Australian Institute of Health and Welfare, 2012.
8 Hoffmann F, Kaduszkiewicz H, Glaeske G, van den Bussche H, Koller D: Prevalence of dementia in nursing
home and community-dwelling older adults in Germany. Aging Clin Exp Res 2014; 26: 555–559.
9 Prince M, Knapp M, Guerchet M, McCrone P, Prina M, Comas-Herrera A, Wittenberg R, Adelaja B, Hu B, King D,
Rehill A, Salimkumar D: Dementia UK: Second Edition – Overview. London, Alzheimer’s Society, 2014.
10 Matthews FE, Arthur A, Barnes LE, Bond J, Jagger C, Robinson L, Brayne C; Medical Research Council Cognitive
Function and Ageing Collaboration: A two-decade comparison of prevalence of dementia in individuals aged
65 years and older from three geographical areas of England: results of the Cognitive Function and Ageing
Study I and II. Lancet 2013; 382: 1405–1412.
11 Centers for Medicare & Medicaid Services: Nursing Home Data Compendium 2015 Edition. Woodlawn, Centers
for Medicare & Medicaid Services, 2015, p 185.
12 Monroe TB, Parish A, Mion LC: Decision factors nurses use to assess pain in nursing home residents with
dementia. Arch Psychiatr Nurs 2015; 29: 316–320.
13 Lautenbacher S, Niewelt BG, Kunz M: Decoding pain from the facial display of patients with dementia: a
comparison of professional and nonprofessional observers. Pain Med 2013; 14: 469–477.
14 Apinis C, Tousignant M, Arcand M, Tousignant-Laflamme Y: Can adding a standardized observational tool to
interdisciplinary evaluation enhance the detection of pain in older adults with cognitive impairments? Pain
Med 2014; 15: 32–41.
15 Fisher SE, Burgio LD, Thorn BE, Allen-Burge R, Gerstle J, Roth DL, Allen SJ: Pain assessment and management
in cognitively impaired nursing home residents: association of certified nursing assistant pain report,
Minimum Data Set pain report, and analgesic medication use. J Am Geriatr Soc 2002; 50: 152–156.
16 Pautex S, Michon A, Guedira M, Emond H, Le Lous P, Samaras D, Michel JP, Herrmann F, Giannakopoulos P,
Gold G: Pain in severe dementia: self-assessment or observational scales? J Am Geriatr Soc 2006; 54: 1040–
1045.
17 Nygaard HA, Jarland M: Are nursing home patients with dementia diagnosis at increased risk for inadequate
pain treatment? Int J Geriatr Psychiatry 2005; 20: 730–737.
18 Gallini A, Andrieu S, Donohue JM, Oumouhou N, Lapeyre-Mestre M, Gardette V: Trends in use of antipsychotics
in elderly patients with dementia: impact of national safety warnings. Eur Neuropsychopharmacol 2014; 24:
95–104.
19 Nakanishi M, Nakashima T, Shindo Y, Niimura J, Nishida A: Japanese care location and medical procedures for
people with dementia in the last month of life. J Alzheimers Dis 2016; 51: 747–755.
20 Jensen-Dahm C, Palm H, Gasse C, Dahl JB, Waldemar G: Postoperative treatment of pain after hip fracture in
elderly patients with dementia. Dement Geriatr Cogn Disord 2016; 41: 181–191.
21 van Kooten J, van der Wouden JC, Sikkes SAM, Smalbrugge M, Hertogh C, Stek ML: Pain, neuropsychiatric
symptoms, and quality of life of nursing home residents with advanced dementia in the Netherlands: a cross-
sectional study. Alzheimer Dis Assoc Disord 2017; 31: 315–321.
22 Jakobsson ULF, Hallberg IR: Pain and quality of life among older people with rheumatoid arthritis and/or
osteoarthritis: a literature review. J Clin Nurs 2002; 11: 430–443.
Downloaded by:
Curtin University Library
134.7.195.166 - 1/24/2018 2:26:22 AM
267
Dement Geriatr Cogn Disord
Atee et al.: Psychometric Evaluation of the Electronic Pain Assessment Tool: An
Innovative Instrument for Individuals with Moderate-to-Severe Dementia
www.karger.com/dem
© 2018 S. Karger AG, Basel
DOI: 10.1159/000485377
23 Ibrahim JE, Murphy BJ, Bugeja L, Ranson D: Nature and extent of external-cause deaths of nursing home resi-
dents in Victoria, Australia. J Am Geriatr Soc 2015; 63: 954–962.
24 Rostad HM, Puts MTE, Cvancarova Småstuen M, Grov EK, Utne I, Halvorsrud L: Associations between pain and
quality of life in severe dementia: a Norwegian cross-sectional study. Dement Geriatr Cogn Dis Extra 2017; 7:
109–121.
25 Klapwijk MS, Caljouw MA, Pieper MJ, van der Steen JT, Achterberg WP: Characteristics associated with quality
of life in long-term care residents with dementia: a cross-sectional study. Dement Geriatr Cogn Disord 2016;
42: 186–197.
26 Corbett A, Husebø BS, Malcangio M, Staniland A, Cohen-Mansfield J, Aarsland D, Ballard C: Assessment and
treatment of pain in people with dementia. Nat Rev Neurol 2012; 8: 264–274.
27 Lichtner V, Dowding D, Esterhuizen P, Closs SJ, Long AF, Corbett A, Briggs M: Pain assessment for people with
dementia: a systematic review of systematic reviews of pain assessment tools. BMC Geriatr 2014; 14: 138.
28 Corbett A, Husebø BS, Achterberg WP, Aarsland D, Erdal A, Flo E: The importance of pain management in older
people with dementia. Br Med Bull 2014; 111: 139–148.
29 Hadjistavropoulos T, Herr K, Prkachin KM, Craig KD, Gibson SJ, Lukas A, Smith JH: Pain assessment in elderly
adults with dementia. Lancet Neurol 2014; 13: 1216–1227.
30 Prkachin KM: Assessing pain by facial expression: facial expression as nexus. Pain Res Manag 2009; 14: 53–58.
31 Ekman P, Friesen WV, Hager JC (eds): Facial Action Coding System. Salt Lake City, Research Nexus, Network
Research Information, 2002.
32 Lints-Martindale AC, Hadjistavropoulos T, Barber B, Gibson SJ: A psychophysical investigation of the facial
action coding system as an index of pain variability among older adults with and without Alzheimer’s disease.
Pain Med 2007; 8: 678–689.
33 Atee M, Hoti K, Parsons R, Hughes JD: Pain assessment in dementia: evaluation of a point-of-care technological
solution. J Alzheimers Dis 2017; 60: 137–150.
34 Australian Pain Society: Pain in residential aged care facilities: management strategies. North Sydney,
Australian Pain Society, 2005.
35 Takai Y, Yamamoto-Mitani N, Chiba Y, Nishikawa Y, Hayashi K, Sugai Y: Abbey Pain Scale: development and
validation of the Japanese version. Geriatr Gerontol Int 2010; 10: 145–153.
36 Gregersen M, Melin AS, Nygaard IS, Nielsen CH, Beedholm-Ebsen M: Reliability of the Danish Abbey Pain Scale
in severely demented and non-communicative older patients. Int J Palliat Nurs 2016; 22: 482–488.
37 Abbey J, Piller N, De Bellis A, Esterman A, Parker D, Giles L, Lowcay B: The Abbey Pain Scale: a 1-minute
numerical indicator for people with end-stage dementia. Int J Palliat Nurs 2004; 10: 6–13.
38 Jorm AF, Mackinnon AJ, Henderson AS, Scott R, Christensen H, Korten AE, Cullen JS, Mulligan R: The Psycho-
geriatric Assessment Scales: a multi-dimensional alternative to categorical diagnoses of dementia and
depression in the elderly. Psychol Med 1995; 25: 447–460.
39 Mackinnon A, Mulligan R: The Psychogeriatric Assessment Scales (PAS): psychometric properties in French
and German speaking populations. Int J Geriatr Psychiatry 2001; 16: 892–899.
40 Jorm AF, Mackinnon AJ, Christensen H, Henderson AS, Jacomb PA, Korten AE: The Psychogeriatric Assessment
Scales (PAS): further data on psychometric properties and validity from a longitudinal study of the elderly. Int
J Geriatr Psychiatry 1997; 12: 93–100.
41 Australian Government, Department of Health: Aged Care Funding Instrument (ACFI) Assessment Pack.
Canberra, Department of Health, Commonwealth of Australia, 2016.
42 Portney L, Mary P: Foundations of clinical research: applications to practice. Upper Saddle River, Pearson/
Prentice Hall, 2009.
43 Bland JM, Altman DG: Cronbach’s alpha. BMJ 1997; 314: 572.
44 Viera AJ, Garrett JM: Understanding interobserver agreement: the kappa statistic. Fam Med 2005; 37: 360–363.
45 Herr K, Bjoro K, Decker S: Tools for assessment of pain in nonverbal older adults with dementia: a state-of-
the-science review. J Pain Symptom Manage 2006; 31: 170–192.
46 Qi S, Diane J, Kay D: The psychometric properties, feasibility and utility of behavioural-observation methods
in pain assessment of cognitively impaired elderly people in acute and long-term care: a systematic review.
JBI Libr Syst Rev 2012; 10: 977–1085.
47 Zwakhalen SM, Hamers JP, Abu-Saad HH, Berger MP: Pain in elderly people with severe dementia: a systematic
review of behavioural pain assessment tools. BMC Geriatr 2006; 6: 3.
48 Hadjistavropoulos T, LaChapelle DL, MacLeod FK, Snider B, Craig KD: Measuring movement-exacerbated pain
in cognitively impaired frail elders. Clin J Pain 2000; 16: 54–63.
49 Feldt KS: The Checklist of Nonverbal Pain Indicators (CNPI). Pain Manag Nurs 2000; 1: 13–21.
50 Husebø BS, Strand LI, Moe-Nilssen R, Husebø SB, Ljunggren AE: Pain behaviour and pain intensity in older
persons with severe dementia: reliability of the MOBID Pain Scale by video uptake. Scand J Caring Sci 2009;
23: 180–189.
51 Erdal A, Flo E, Selbaek G, Aarsland D, Bergh S, Slettebø DD, Husebø BS: Associations between pain and
depression in nursing home patients at different stages of dementia. J Affect Disord 2017; 218: 8–14.
52 van Dalen-Kok AH, Pieper MJC, de Waal MWM, Lukas A, Husebø BS, Achterberg WP: Association between pain,
neuropsychiatric symptoms, and physical function in dementia: a systematic review and meta-analysis. BMC
Geriatr 2015; 15: 49.
Downloaded by:
Curtin University Library
134.7.195.166 - 1/24/2018 2:26:22 AM
