A preview of this full-text is provided by Taylor & Francis on behalf of the Scandinavian Journal of Occupational Therapy Foundation.
Content available from Scandinavian Journal of Occupational Therapy
This content is subject to copyright. Terms and conditions apply.
ORIGINAL ARTICLE
Inter-rater and test-retest reliability of the Danish version of the everyday
technology use questionnaire
Rina Juel Kaptain
a,b
, Anders Kottorp
b,c
, Ann-Helen Patomella
b
and Tina Helle
a,b
a
Department of Occupational Therapy, Department of Research and Development, University College North, Aalborg, Denmark;
b
Department of Neurobiology, Care Sciences and Society, Division of Occupational Therapy, Karolinska Institutet, Stockholm, Sweden;
c
Department of Occupational Therapy, College of Applied Health Sciences, University of Illinois at Chicago, Chicago, IL, USA
ABSTRACT
Background: Everyday technologies are naturally integrated in people’s daily life. For older
adults and adults living with a chronic disease like chronic obstructive pulmonary disease
(COPD), the ability to use technologies for health management has become increasingly import-
ant. The aim of this study was to investigate inter-rater and test-retest reliability of the Danish
version of the Everyday Technology Use Questionnaire (ETUQ) in a sample of older adults with
(n¼23) and without (n¼24) COPD.
Material and method: The ETUQ was initially translated in accordance with the dual panel
approach and then administered to a sample of 47 participants. Svensson’s method for paired
ordinal data was utilized to calculate and analyze reliability.
Results: Overall, inter-rater and test-retest reliability of the Danish version of the ETUQ demon-
strated excellent percentage agreement (PA) (>75%), although for test-retest reliability, nine
items demonstrated fair (53%) to good (73%) agreement.
Conclusion: This study supports the use of the Danish version of the ETUQ in a sample of older
adults with or without COPD.
Application to practice: The Danish version of the ETUQ is an evidence-based evaluation that
can reliably contribute to clinical occupational therapy and research in Denmark focusing on
everyday technology use.
ARTICLE HISTORY
Received 21 December 2016
Revised 17 October 2017
Accepted 18 October 2017
KEYWORDS
Agreement; everyday
technology; occupational
therapy; psychometrics;
Svensson’s method;
translation
Introduction
Everyday technology is increasingly being used in
Western society [1] in homes and in public spaces
[2]. However, research has shown that individuals,
both with and without disabilities, may encounter dif-
ficulties in using everyday technologies [3,4] such as
household appliances, remote controls, cell phones,
microwaves, online banking, ticket machines, global
positioning system (GPS) and tablets. For example,
people with acquired brain injury report difficulties
with telecommunication and computers [5], and when
performing online banking transactions and using
ticket machines, which requires a quick response or
selection among several alternatives [6]. They also
highlight difficulties in communicating with public
authorities through the internet or telephone services,
which can restrict their participation in society [6].
Therefore, due to technological developments in soci-
ety at large, it is very important to explore how
restricted use of everyday technology may affect daily
life for people with and without disabilities.
As concerns people living with chronic diseases
such as chronic obstructive pulmonary disease
(COPD), telehealth (or telemedicine) is a growing
area that also involves everyday technology.
Telehealth/telemedicine here refers to the delivery of
healthcare services from a distance [7]. These services
usually involve clients with, for example, COPD who
frequently monitor their current state of health and
send their data to healthcare professionals using a
tablet or a cell phone [8], or consult health care pro-
fessionals via a video conference call [9]. In other
words, the ability to use different everyday technolo-
gies may be an increasingly important prerequisite for
health management for people with chronic diseases.
From an occupational therapy perspective, it is there-
fore essential to evaluate a person’s ability to interact
with and use everyday technology since the ability to
CONTACT Rina Juel Kaptain rio@ucn.dk Department of Occupational Therapy and Department of Research and Development, University College
North, Denmark, Karolinska Institutet, Department of Neurobiology, Care Sciences and Society, Division of Occupational Therapy, Sweden, Bødkervej 5,
9300 Saeby, Denmark
ß2017 Informa UK Limited, trading as Taylor & Francis Group
SCANDINAVIAN JOURNAL OF OCCUPATIONAL THERAPY
2019, VOL. 26, NO. 3, 226–234
https://doi.org/10.1080/11038128.2017.1395910
do so can have a major impact on several areas of
people’s daily lives, including their health
management.
One instrument that can support the evaluation of
everyday technology use is the Swedish standardized
instrument, the Everyday Technology Use
Questionnaire (ETUQ). The ETUQ provides a system-
atic evaluation of perceived relevance of and difficul-
ties associated with everyday technology use [10,11].
The psychometric properties of the ETUQ have previ-
ously been investigated in several studies within dif-
ferent populations, primarily in Sweden, but also in
other contexts [12,13]. These studies have addressed
older adults with and without mild cognitive impair-
ment and dementia [14], adults with intellectual dis-
abilities [15,16], and adults with acquired brain injury
[17]. A Rasch rating model [18] has earlier been uti-
lized to investigate different aspects of validity, for
example, rating the scale validity, person response val-
idity, and internal scale validity [16] of the instrument
when applied to different samples and different con-
texts. So far, however, the reliability of the ETUQ has
been examined only in terms of the precision of the
generated measures (by means of separation indices)
[16], whereas no empirical reliability studies have
been conducted addressing changes over time (test-
retest reliability) or potential interviewer influence
(inter-rater reliability).
Since the ETUQ will be used in a Danish research
project targeting telehealth interventions, activity, and
participation amongst adults living with COPD, it was
necessary to translate and psychometrically test the
Danish version of the ETUQ.
The aim of this study was therefore to investigate
the inter-rater and test-retest reliability of the Danish
version of the ETUQ when used with older adults
with and without COPD.
Materials and methods
Translation and adaptation of the ETUQ to a
danish context
The dual panel approach [19] was used for the trans-
lation of the original Swedish version of the ETUQ
into Danish. Panel One worked with the translation
of the ETUQ into Danish and Panel Two worked
with the language validity [19] of the new Danish ver-
sion of the instrument. Panel One consisted of two
registered occupational therapists with PhD degrees,
with Danish as their mother tongue, but very familiar
with the Swedish language. Panel Two consisted of a
selection of potential end-users: four Danish
occupational therapists, both clinicians and lecturers,
and one older adult person. The translation and adap-
tation took place during a period of three months in
an iterative process, involving several face-to-face and
telephone meetings with the developers of the ETUQ
that resulted in a Danish version of the ETUQ that
was culturally adapted and ready to be tested for reli-
ability and stability in the Danish context. To give
one example of cultural adaptation issues, one item of
the Swedish version of ETUQ (lock-key to washing
room) does not exist in Denmark. Therefore, this
item was not translatable and was left out of the
Danish version.
Instrumentation
The ETUQ was developed in order to explore and
evaluate perceived difficulties in using everyday tech-
nologies at home and in public places. More specific-
ally, the ETUQ addresses perceived quality of the
interaction between the individual and a specific tech-
nology, considering also the relevance of the technol-
ogy in question. Relevance is here defined as follows:
(1) The technology is available to the individual, and
(2) The technology (a) has earlier been used by the
individual, (b) is currently used by the individual, or
(c) is intended to be used by the individual [20].
The ETUQ version utilized in this study consists of
93 items [21] allocated into following eight activity
areas: household activities, various activities in the
home, personal care, activities using power tools,
accessibility in activities outside the home, communi-
cation using a personal computer and telephone,
activities in economy and shopping, and activities in
transport [10].
ETUQ targets everyday technology use with respect
to the individual person’s current life situation, con-
text, interests and values. It is administered as a struc-
tured face-to-face interview where the interviewer first
examines, whether the technology is relevant for the
person’s current life situation. If so, the interviewer
next examines the extent to which, the person may or
may not find any difficulties related to the manage-
ment of the everyday technology in question. Ratings
are marked according to following ordinal five-cat-
egory rating scale: (1) The technology is not used any-
more or the person has not started using it, even if it is
available and relevant, (2) The technology is only used
together with another person, (3) The technology is
used with considerable difficulty, (4) The technology is
used with hesitation/uncertainty/minor difficulty, and
(5) The technology is used with no hesitation or diffi-
culty at all [21]. Thus, the interviewer goes through
SCANDINAVIAN JOURNAL OF OCCUPATIONAL THERAPY 227
each item as a semi-structured interview and decides
which of the five response alternatives best corre-
sponds to the participant’s replies.
Participants and raters
The sample included adult persons, aged 55 years,
who lived in ordinary dwellings, and excluded those
who had major cognitive, physical, hearing, and/or
visual functional limitations. Potential participants
were identified via health care centers, activity centers,
patient organizations, and organizations of older
adults. A combination of purposive and snowball
sampling [22] was used until a target sample of 30
cases each for the examination of inter-rater reliability
and test-retest reliability was achieved. Since we aimed
for an equal representation of adults with and without
COPD in each of the reliability groups (inter-rater
and test-retest), we purposefully sampled fifteen par-
ticipants with COPD and fifteen participants without
COPD in each of the reliability groups. As thirteen
participants volunteered to participate in both reliabil-
ity tests, we ended up with a total sample of 47 partic-
ipants (n¼24 with COPD, n¼23 without COPD).
See Table 1 for a detailed description of the sample
characteristics.
Four raters served as data collectors in the study.
The raters were all registered occupational therapists
who had 2–17 years of clinical experience. To
ensure reliable instrument use, raters had completed
the same 1-day ETUQ course in Denmark, resulting
in a certification. To investigate inter-rater reliability,
the raters were paired in different combinations of
rater pairs across the four data collectors. Rater 1 per-
formed the ETUQ interview and scorings, whereas
rater 2 only performed the scorings. Both raters made
the scorings independently and based on the partic-
ipant’s responses. This approach was chosen in order
to be time-effective as well as for ethical aspects such
as minimizing endurance problems among the partici-
pants, as taking part in interviews can be a demand-
ing task for persons living with COPD. To examine
test-retest reliability, rater 1 of the rater pair repeated
the ETUQ interview 1–4 weeks after the first home
visit. This is considered to be sufficiently long to
avoid the participant remembering the specific
responses given, but at the same time also to assure
that the participant’s health status and everyday tech-
nology repertoire has not changed [21,23], as this may
impact on the outcomes. Two of the raters conducted
the test-retest interviews. The first author (RJK) con-
ducted 20 retest interviews, whereas the last author
(TH) conducted 10 retest interviews.
Ethics
The study was conducted in accordance with the
Danish Data Protection Agency (J. No. 2014-41-3078).
Participants were informed both orally and in writing
about how the data would be anonymized and treated
confidentially, and that participants could withdraw
from the study at any time without any consequences.
Table 1. Sample characteristics (N¼47) in groups of participants with chronic obstructive pulmonary disease (COPD) (n¼23)
and without COPD (n¼24), and the number of cases of inter-rater reliability (n¼30) and test-retest reliability (n¼30) with or
without COPD.
Sample
characteristics
Participants
Total
N¼47
Participants
with COPD
n¼23
Participants
without COPD
n¼24
Cases of Inter-rater
reliability (15 with COPD
and 15 without COPD)
n¼30
Cases of Test-retest
reliability (15 with COPD
and 15 without COPD)
n¼30
Gender: n(%)
Male 15 (32) 7 (30) 8 (33) 9 (30) 9 (30)
Female 32 (68) 16 (70) 16 (67) 21 (70) 21 (70)
Living status:
Live alone 20 (43) 13 (57) 7 (29) 12 (40) 12 (40)
Live together 27 (57) 10 (43) 17 (71) 18 (60) 18 (60)
Education:
Primary school 27 (57) 15 (65) 12 (50) 16 (53) 19 (63)
High school 5 (11) 4 (17.5) 1 (4) 3 (10) 2 (7)
Higher education 15 (32) 4 (17.5) 11 (46) 11 (37) 9 (30)
Dependence on:
Walking aid(s)
a
8 (17) 6 (26) 2 (8) 3 (10) 6 (20)
Wheelchair
a
3 (6) 1 (4) 2 (8) 2 (7) 1 (3)
Physical limitations:
Stamina
a
22 (47) 18 (78) 4 (17) 14 (47) 15 (50)
Balance
a
17 (36) 9 (39) 8 (33) 10 (33) 11 (37)
Coordination
a
2 (4) 1 (4) 1 (4) 1 (3) 1 (3)
Spine and/or lower extremity
a
24 (51) 12 (52) 12 (50) 16 (53) 14 (47)
a
Items according to the Housing Enabler: Person components (41) [42].
228 R. J. KAPTAIN ET AL.
Data analysis
To investigate the inter-rater and test-retest reliability
of the ETUQ, we utilized Svensson’s method for
paired ordinal data [24–26]. Svensson’s method was
developed as a reaction to the tendency in classic test
theory applications to assume that ordinal data are
comparable to data at interval level, ignoring the dif-
ference by summing up a number of items, or distrib-
uting a total sum score as if the scores were at an
interval level [23]. Alternatively, Svensson developed a
statistical method based on the distribution of paired
ordinal data [27]. The data were entered into
Svensson’s Micro Soft-Excel file for the six categories
of the ETUQ [24] and statistical measures were calcu-
lated. Percentage agreement (PA) was calculated for
each item and evaluated against a commonly accepted
definition of agreement: <40%-poor agreement;
40–60%-fair agreement; 60–75%-good agreement;
>75%- excellent agreement [28].
Following Svensson’s method, four additional stat-
istical measures were calculated to provide a more
profound reliability analysis of the ETUQ item data:
relative position (RP), relative concentration (RC),
relative rank variation (RV), and measure of disorder
(D) [25]. RP estimates the parameter of systematic
shift in position between the pairs. That is, it is a
measure of the pattern of change for the group,
whereas RC is the statistical measure for a systematic
change in the concentration of the raters’scores and
the extent to which the scores are centered on the
scale. The RP and RC are measures of systematic dis-
agreement and have possible values from 1.00 to
1.00. If both values are .00, there is no change within
the group and agreement is complete: PA ¼100%.A
negative RP is obtained when there is a systematic
change toward lower categories from test to retest or
between Rater 1 and Rater 2 [25,29,30]. RV is the
measure of the difference between the paired ranges
[24,27,29,31]. A measure of disorder among possible
combinations of pairs, D, has possible values from .00
(total order) to 1.00 (total disorder). D and RV are
calculated to evaluate individual sources of disagree-
ment [26,32]. The individual patterns of change
between the raters’scores are presented in a cross-
tabulation within each of the 93 ETUQ items.
We also wanted to ensure that the initial decision
made in the ETUQ, as to whether a specific everyday
technology was relevant to the individual participant,
was indeed stable between raters and/or occasions.
We therefore decided to enter this initial judgment as
an additional score in our matrix and included a 0
score to our 1–5 category scale. Thus, we ended up
with 36 (6 6) possible combinations of pair-wise
data from our six category rating scale [24], including
the initial item relevance decision.
As a supplementary reliability analysis, we also
wanted to explore the relationships not only on item
level, but also on a more generic person level. First,
we therefore transformed the ETUQ data from the
participants into interval measures by the use of a
Rasch rating scale model [18], as described in other
studies [14,15,16]. As the ETUQ has not been vali-
dated in a sample of older people with and without
COPD in Denmark, a preliminary validity control in
relation to response processes was initially performed.
We used an infit Mean Square statistic of less than
1.4 associated with a standardized zvalue of less than
2.0 to indicate acceptable person goodness-of-fit to
the Rasch model [33,34]. As none of the participants
in the sample demonstrated unacceptable goodness-
of-fit to the model, we decided to use all Rasch gener-
ated interval measures from the participants as valid
estimations of everyday technology use. We then used
Pearson correlations to explore relationships between
measures concerning raters and time points.
Results
Inter-rater reliability
Concerning inter-rater reliability, all items (100%)
demonstrated a percentage agreement of 87–100%,
which is considered excellent agreement [28].
Regarding the relative position (RP) and relative con-
centration (RC) for inter-rater reliability, Table 2
demonstrates values varying between .14 to .10 (RP)
and .50 to .08 (RC) for all ETUQ items. RP and RC
were both relatively close to .00, indicating only
minor differences between the raters. Relative rank
variance (RV) was between .00 to .01 across the
ETUQ items, which indicates only minor individual
variability consistent with the measure of disorder
(D).
Test-retest reliability
The percentage agreement for the test-retest reliability
varied from 53%to 100%, which demonstrates excel-
lent agreement for 84 items (90%), good agreement
for eight items (9%), and fair agreement for one item
(1%) (see Table 3 for exact items and PA). For test-
retest reliability, the RP varied from .16 to .17 and
the RC ranged from .14 to .10, indicating minor sys-
tematic shifts in position and concentration within
the evaluated group. The RV for test-retest reliability
SCANDINAVIAN JOURNAL OF OCCUPATIONAL THERAPY 229
varied from .00 to .18, reflecting a wider distribution
of the percentage agreement across items. These find-
ings are also related to the measure of disorder (D)
(ranging from .00 to .18), which denotes up to 18%
chance of disorder within the pair. A closer examin-
ation of the test-retest results on the ETUQ items
with a PA below 75%shows a tendency to shift
between the scale steps ‘No difficulty’and ‘Not rele-
vant’, influencing the lower PA on the majority of
items (n¼56%). For the item ‘Digital camera’, shifts
occurred between the scale steps ‘No difficulty’and ‘The
everyday technology is not used anymore’, and for a few
items it looked more randomly distributed (n¼33%).
In line with the findings from Svensson’s method
on item level, the correlation coefficient between
ETUQ measures related to raters was r¼.87 (p<0.01),
whereas the correlation coefficient between ETUQ
measures related to time point was r¼.76 (p<0.01).
Discussion
The overall aim of this study was to investigate inter-
rater and test-retest reliability of the Danish version
of the ETUQ when used with older adults with and
without COPD. As demonstrated by the results, excel-
lent percentage agreement (PA) of inter-rater reliabil-
ity and excellent PA for the vast majority of ETUQ
items in test-retest reliability (90%) were found. There
was less agreement in the test-retest evaluation among
nine ETUQ items, supported by the measures of sys-
tematic disagreement (RP, RC) and individual vari-
ability (RV, D). When looking into the ETUQ items
demonstrating a PA from ‘fair’to ‘good’in the test-
retest, these items refer to everyday technologies that
are probably not used on a frequently/daily basis in
our sample, such as Digital Camera, Automatic ticket
machine, Automatic check-in at airport, Import digital
data from camera into the computer. One explanation
could be the classic element when performing test-
retest evaluations that the individual may have
reflected on the evaluation and this cognitive process
may have affected the answers in the re-test evalu-
ation, concerning the relevance of and perceived abil-
ity to use these technologies. Streiner and Norman
[23] describe that the results for the test-retest reliabil-
ity is often higher than for inter-rater reliability, yet
Table 2. Inter-rater reliability results of the frequency of items, relative position (RP), relative concentra-
tion (RC), relative rank variance (RV), and measure of disorder (D) in relation to level of percentage
agreement (PA) for scores of the 93 items of the danish version of the ETUQ using svensson’s method
(25,28,30,32).
Level of agreement PA(%) Frequency of items (%) RP RC RV D
Excellent 100 47 (51) .00 .00 .00 .00
97 26 (28) .04–.04 .04–.04 .00 .00–.01
93 13 (14) .06–.06 .02–.05 .00–.01 .00–.02
90
b
4 (4) .08–.10 .00–.08 .00–.01 .00–.02
87
a
3 (3) .14 to .01 .05 to .01 .00–.01 .00–.04
a
Smoke detector, DVD, Radio;.
b
Alarm clock, TV w. remote, Stereo, Mobile phone top on money.
Table 3. Test-retest reliability results of the relative position (RP), relative concentration (RC), relative
rank variance (RV), and measure of disorder (D) in relation to level of percentage agreement (PA) for
scores on 93 items of the danish version of the ETUQ when using the svensson’s method for analysis
(25,28,30,32).
Level of agreement PA(%) Frequency of items (%) RP RC RV D
Excellent 100 17 (19) .00 .00 .00 .00
97 12 (14) .03–.04 .04–.03 .00 .00–.04
93 17 (18) .16–.10 .09–.05 .00 .00–.01
90 12 (10) .07–.10 .10–.09 .00 .00–.02
87 6 (7) .04–.07 .00–.05 .00–.02 .00–.06
86 1 (1) .08 .09 .00 .00–.02
83 8 (8) .16–.10 .09–.05 .00–.03 .00-.06
80 7 (7) .06–.12 .08–.09 .00–.02 .00–.04
79 1 (1) .15–.16 .29 to .02 .00 .00–.01
77 3 (3) .04–.17 .05–.07 .00–.04 .00–.05
Good 73
d
3 (3) .09–.03 .08–.00 .00–.04 .00–.06
70
c
3 (3) .06–.12 .08–.08 .00–.05 .00–.07
63
b
2 (1) .04–.16 .10 to .03 .00–.08 .00–.10
Fair 53
a
1 (1) .14 .01 .18 .18
a
Tape player;.
b
Pedometer, Flushing mechanism;.
c
Camera digital, Automatic ticket machine, Automatic check-in at airport;.
d
Import digital data, Remote control, Computer game.
230 R. J. KAPTAIN ET AL.
this is not the case for this study. In addition, no lin-
guistic or wording concerns occurred in the transla-
tion of the items giving the lower scores.
In previous ETUQ studies, it is considered a non-
response if the everyday technology in question is not
perceived relevant, and hence, the item is not part of
the statistical analysis to generate an individual meas-
ure of ability to use everyday technology [11,14,16].
In this study, we decided to include this as a scale
step with the intention to ensure the stability of this
relevance decision across raters and over time.
Variations in judgment of relevance of an everyday
technology could play a critical role in estimating a
person’s overall ability to use everyday technology,
especially if such items as those described above,
when considered relevant, are perceived as more diffi-
cult to use. If such items are sometimes described as
non-relevant to the person, this will result in a risk of
over-estimating that person’s overall ability to manage
everyday technology. This potential risk scenario,
however, was not supported by our empirical data as
the systematic shift on those items was primarily
between the scale steps ‘No difficulty’and ‘Not rele-
vant’. Also, considering that only one ETUQ item
demonstrated fair test-retest agreement (1%), this sys-
tematic difference has minimal impact on the preci-
sion of the generated measures in our sample,
although the finding does provide important informa-
tion regarding the future development and use of the
ETUQ scale steps in clinical research. Therefore, the
variations in test-retest reliability found in a limited
number of ETUQ items, associated with a relatively
lower relationship between ETUQ measures in rela-
tion to time points, may not provide any major threat
to stability of the ETUQ estimations over time.
As to the overall high PA it is important to bear in
mind that the vast majority of participants perceived
few difficulties in using everyday technology, why
most scorings were marked in the category ‘No
difficulty’. The fact that our study sample, constituted
by older adults with and without COPD, perceive lim-
ited difficulties in the use of everyday technology is in
itself a clinically important outcome, and can suggest
that the use of telehealth platforms in an intervention
program for people with COPD can be beneficial for
a similar sample. With that said, for future reliability
studies on ETUQ, it is recommendable to strive for
larger variation in scores from a larger and more het-
erogeneous sample, in order to obtain even more
detailed information on the reliability of ETUQ.
We believe that important explanations of the
inter-rater results both on item and person level are
due to that the raters had been thoroughly introduced
to the ETUQ, they met during the data collection to
exchange challenges and help each other to administer
the ETUQ, which ultimately aimed at increasing reli-
ability, and that raters were used to utilize standar-
dized instruments in research and practice [23].
Furthermore, several psychometric studies of the
ETUQ have earlier validated the questionnaire in rela-
tion to both everyday technology items and the rating
scale used [14,16,35], thereby contributing to revisions
and clarification in the manual regarding scoring pro-
cedures and scale categories. Such aspects are import-
ant in facilitating a high degree of agreement across
both interviewers and time [36]. Occupational therapy
practitioners are currently also participating in a one-
day training course and specifically trained/calibrated
before using the ETUQ in practice. So when the
ETUQ is used in practice with a larger and more var-
ied group of practitioners, the rater agreement can
still be expected to be relatively high, although prob-
ably not as high as indicated in this study.
To our experience based on the present study,
there are several advantages of combining examina-
tions of inter-rater and test-retest reliability in the
same sample. One advantage is that the error variance
resulting from different raters and occasions can easily
be compared and associations can be explored further
[23]. The approach where two raters are present at
the first home visit (inter-rater reliability), but only
one rater performs the retest at the second home visit,
is time effective for the data collectors. Even more
important, it is also less disturbing to the participants,
who, in this way, avoid a third home visit. When it
comes to persons living with COPD, there is an eth-
ical consideration too, since interviewing and talking
can be demanding for this particular population. This
approach also demonstrates some methodological
benefits, like for instance that the test conditions are
very similar. That is, performing the inter-rater reli-
ability at the same occasion, the scorings were based
on the same answers, not making any interim period
where either the participant or the context could
change [37,38].
The Svensson method for paired ordinal data
makes it possible to detect the degree of systematic
disagreement separately from disagreement caused by
individual variations [25,26]. Various studies have
demonstrated the applicability of this method for reli-
ability evaluation of various clinical instruments
[25,26,32,39]. However, the approach is still relatively
limited in use and usually not highlighted specifically
in more generic guidelines to evaluate reliability data
SCANDINAVIAN JOURNAL OF OCCUPATIONAL THERAPY 231
[37,40]. One limitation with the use of the Svensson’s
method is that it only generates results on item level
and not a summarized reliability-score as other reli-
ability methods might generate (e.g. intraclass correl-
ation coefficient (ICC)). On the other hand, such
commonly used reliability measures are based upon
assumptions of linear item data responses that also
can validly be summarized, which can be theoretically
and empirically questioned with many occupational
therapy and health care evaluations using ordinal
scales. In this study we therefore decided to comple-
ment the Svensson’s method with a Rasch-based
approach, as such models also are based on similar
theoretical assumptions as Svensson.
This study can contribute to further evidence-based
use of the ETUQ in Danish occupational therapy clin-
ical practice, research, as well as education. Danish
occupational therapists can use this study to provide
evidence that, after training in the use of the ETUQ,
it is possible to obtain both valid and reliable results
regarding the client’s perceived relevance and difficul-
ties involved in using everyday technologies in daily
activities. We used a sample consisting of older adults
with and without COPD, and the results support fur-
ther use of the ETUQ in a Danish context and future
research project involving everyday technology. The
findings from this study have also informed the revi-
sion process of the ETUQ, where items that were
more often found to be not relevant have been
removed or changed in keeping up with the ongoing
development in the everyday technologies in society at
large. Clarifications have also been made in the
revised ETUQ manual that reflect some of the find-
ings of the present study [41].
The limitations of this study are its small sample
size and the relatively homogeneous and high-func-
tioning group (Table 1) of older adults with and with-
out COPD which minimized the overall variation in
ETUQ item scores [19] for the reliability analysis.
Still, in some larger studies, a small number of cases
is randomly selected to investigate more in-depth reli-
ability issues [36] with similar sample sizes as in this
study, which supports the relatively small sample used
in this study. However, based upon these findings,
future reliability studies should target a more diverse
sample in relation to everyday technology use to pro-
vide more detailed information on the scale and item
functioning of the ETUQ.
Conclusion
The Danish version of the ETUQ has been translated,
adapted, and investigated for validity in relation to
response processes, inter-rater and test-retest reliabil-
ity. With respect to the small sample size used in this
study and the fact that the group of participants with
and without COPD displayed overall high functioning
in relation to everyday technology use, the findings
indicate that the Danish version of the ETUQ is valid
and reliable for use in a Danish context.
Acknowledgements
The authors wish to thank the individuals who agreed to
participate in this study, and the raters. The authors would
also specifically like to thank Eva Ejlersen Waehrens for
contributing to the translation process of the ETUQ.
This research was financially supported by Technologies
Closely Connected to Citizens’Health and the Division of
Occupational Therapy, University College of Northern
Denmark.
Disclosure statement
The authors report no conflicts of interest.
Funding
This research was financially supported by Technologies
Closely Connected to Citizens’Health and the Division of
Occupational Therapy, University College of Northern
Denmark.
ORCID
Rina Juel Kaptain http://orcid.org/0000-0002-0603-5621
Anders Kottorp http://orcid.org/0000-0002-8976-2612
Ann-Helen Patomella http://orcid.org/0000-0003-2667-
4073
Tina Helle http://orcid.org/0000-0002-2819-4471
References
[1] Emiliani PL. Assistive Technology (AT) versus
Mainstream Technology (MST): the research per-
spective. Technol Disabil. 2006;18:19.
[2] Malinowsky C, Kottorp A, Patomella A-H, et al.
Changes in the technological landscape over time:
Relevance and difficulty levels of everyday technolo-
gies as perceived by older adults with and without
cognitive impairment. TAD. 2015;27:91–101.
[3] Ackerman SE, Bednarczyk KR, Roncolato K, et al.
The use of computer technology with older adult cli-
ents: a pilot study of occupational therapists. Phys
Occup Ther Geriatr. 2001;20:49–57.
[4] Fallahpour M, Kottorp A, Nygard L, et al. Perceived
difficulty in use of everyday technology in persons
with acquired brain injury of different severity: a
comparison with controls. J Rehabil Med. 2014;46:
635–641.
232 R. J. KAPTAIN ET AL.
[5] Lind
en A, Lexell J, Lund ML. Perceived difficulties
using everyday technology after acquired brain
injury: influence on activity and participation. Scand
J Occup Ther. 2010;17:267–275.
[6] Engstr€
om A-LL, Lexell J, Lund ML. Difficulties in
using everyday technology after acquired brain
injury: a qualitative analysis. Scand J Occup Ther.
2010;17:233–243.
[7] Craig J, Patterson V. Introduction to the practice of
telemedicine. J Telemed Telecare. 2005;11:3–9.
[8] Udsen FW, Lilholt PH, Hejlesen O, et al.
Effectiveness and cost-effectiveness of telehealthcare
for chronic obstructive pulmonary disease: study
protocol for a cluster randomized controlled trial.
Trials. 2014;15:178.
[9] Nield M, Hoo GWS. Real-time telehealth for COPD
Self-Management Using Skype
TM
. J Chronic Obstr
Pulm Dis. 2012;9:611–619.
[10] Rosenberg L, Kottorp A, Winblad B, et al. Perceived
difficulty in everyday technology use among older
adults with or without cognitive deficits. Scand J
Occup Ther. 2009;16:216–226.
[11] Rosenberg L, Kottorp A, Nygard L. Readiness for
technology use with people with dementia: the per-
spectives of significant others. J Appl Gerontol.
2012;31:510–530.
[12] Malinowsky C, Kottorp A, Tanemura R, et al.
Validation of the everyday technology use question-
naire in a Japanese context. Hong Kong J Occup
Ther. 2015;26:1–8.
[13] Patomella A-H, Kottorp A, Ferreira M, et al.
Everyday technology use among older adults in
Sweden and Portugal. Scand J Occup Ther. 2017.
DOI:10.1080/11038128.2017.1311940
[14] Rosenberg L, Nygård L, Kottorp A. Everyday tech-
nology use questionnaire: psychometric evaluation of
a new assessment of competence in technology use.
OTJR. 2009;29:52–62.
[15] H€
allgren M, Nygård L, Kottorp A. Everyday technol-
ogy use among people with mental retardation: rele-
vance, perceived difficulty, and influencing factors.
Scand J Occup Ther. 2014;21:210–218.
[16] H€
allgren M, Nygård L, Kottorp A. Technology and
everyday functioning in people with intellectual dis-
abilities: a Rasch analysis of the Everyday
Technology Use Questionnaire (ETUQ). J Intellect
Disabil Res. 2011;55:610–620.
[17] Larsson Lund M, Nygård L, Kottorp A. Perceived
difficulty in the use of everyday technology: relation-
ships with everyday functioning in people with
acquired brain injury with a special focus on return-
ing to work. Disabil Rehabil. 2014;36:1618–1625.
[18] Bond TG, Fox CM. Applying the Rasch Model: fun-
damental measurement in the human sciences. 2nd
ed. Florence: Taylor and Francis; 2012.
[19] Hagell P, Hedin PJ, Meads DM, et al. Effects of
method of translation of patient-reported health out-
come questionnaires: a randomized study of the
translation of the rheumatoid arthritis quality of life
(RAQoL) instrument for Sweden. Value Health.
2010;13:425–430.
[20] Kottorp A, Fisher AG, Griswold LA. Validity of a
new tool for assessing the discrepancy between
observed and selfreported quality of activities of
daily living (ADL) task performance. Am J Occup
Ther. 2015;69:6911500100p1.
[21] Nygård L. Manual to the questionnaire about
Everyday technology in home and society: everyday
technology use questionnaire (ETUQ II) & short
ETUQ (S-ETUQ). Department of Neurobiology,
Care Sciences and Society, Division of Occupational
Therapy, Karolinska Institutet, Stockholm; 2012.
[22] DePoy E, Gitlin LN. Introduction to research: under-
standing and applying multiple strategies. 4th ed. St
Louis: Elsevier; 2011.
[23] Streiner DL, Norman GR. Health measurement
scales: a practical guide to their development and
use. 4th ed. Oxford: Oxford University Press; 2008.
[24] AVDACIA. Anders Avdic. [Internet]. 2015 [Updated
2015 Aug 14; cited 2015 Sep 20] Manual Svenssons
metod [about 8 screens]. Available from: http://
avdic.se/svenssonsmetodsvenska.html
[25] Svensson E. Different ranking approaches defining
association and agreement measures of paired
ordinal data. Statist Med. 2012;31:3104–3117.
[26] Norrefalk J-R, Svensson E. The functional barom-
eter-a self-report questionnaire in accordance with
the International Classification of Functioning,
Disability and Health for pain related problems; val-
idity and patient-observer comparisons. BMC Health
Serv Res. 2014;14:187.
[27] Svensson E. Guidelines to statistical evaluation of
data from rating scales and questionnaires. J Rehabil
Med. 2001;33:47–48.
[28] Watkins MW, Pacheco M. Interobserver agreement
in behavioral research: importance and calculation.
J Behav Educ. 2000;10:205–212.
[29] Svensson E. Statistisk metod f€
or parade ordinaldata.
Lakartidningen. 2007;104:596–601.
[30] AVDACIA. Sweden. Avdic A. [Internet]. 2015
[Updated 2015 Aug 14; cited 2015 Sep 20]
Guidelines to calculation by the free software and
interpretation of the measures of disagreement
applied to reliability studies. [about 3 screens].
Available from: http://avdic.se/svenssonsmetodsven-
ska.html
[31] Svensson E. Val och konsekvens: m€
atnivån avg€
or
den statistiska verktygslådan. L€
akartidningen.
2005;17:1331–1337.
[32] Allvin R, Svensson E, Rawal N, et al. The postopera-
tive recovery profile (PRP): a multidimensional ques-
tionnaire for evaluation of recovery profiles. J Eval
Clin Pract. 2011;17:236–243.
[33] Nilsson I, Fisher AG. Evaluating leisure activities in
the oldest old. Scand J Occup Ther. 2006;13:31–37.
[34] Patomella AH, Tham K, Kottorp A. P-drive: assess-
ment of driving performance after stroke. J Rehabil
Med. 2006;38:273–279.
[35] Nygård L, Pantzar M, Uppgard B, et al. Detection of
activity limitations in older adults with MCI or
Alzheimer’s disease through evaluation of perceived
difficulty in use of everyday technology: a replication
study. Aging Ment Health. 2012;16:361–371.
SCANDINAVIAN JOURNAL OF OCCUPATIONAL THERAPY 233
[36] Kimberlin CL, Winterstein AG. Validity and reliabil-
ity of measurement instruments used in research.
Am J Health Syst Pharm. 2008;65:2276–2284.
[37] Terwee CB. The COSMIN checklist (COnsus-based
Standards for the selection of health status
Measurement INstruments). [Internet]. Amsterdam
2017 [cited 2017 April 1st]. Available from: http://
www.cosmin.nl/the_cosmin_checklist.html
[38] Slaug B, Schilling O, Helle T, et al. Unfolding the
phenomenon of interrater agreement: a multicompo-
nent approach for in-depth examination was pro-
posed. J Clin Epidemiol. 2012;65:1016–1025.
[39] Mårtensson L, Archenholtz B, Dahlin-Ivanoff S. The
conceptions of pain and rehabilitation questionnaire
(CPRQ): development and test of face validity and
stability over time. Scand J Occup Ther. 2007;14:
230–239.
[40] Gisev N, Bell JS, Chen TF. Interrater agreement and
interrater reliability: key concepts, approaches, and
applications. Res Social Adm Pharm. 2013;9:
330–338.
[41] Nygård L, Rosenberg L, Kottorp A. Users manual:
everyday technology use questionnaire (ETUQ)
everyday technology in activities at home and in
society. Department of Neurobiology, Care Sciences
and Society, Division of Occupational Therapy,
Karolinska Institutet, Stockholm; 2016.
[42] Iwarsson S, Slaug B. Housing enabler: en metode til
vurdering/kortlaegning og analyse af tilgaengelighed-
sproblemer i boliger. Malm€
o. Exakta; 2010.
234 R. J. KAPTAIN ET AL.
