Content uploaded by Dirk Ruwaard
Author content
All content in this area was uploaded by Dirk Ruwaard on Jun 11, 2017
Content may be subject to copyright.
SYSTEMATIC REVIEW
Health-Related Resource-Use Measurement Instruments
for Intersectoral Costs and Benefits in the Education and Criminal
Justice Sectors
Susanne Mayer
1,2
•Aggie T. G. Paulus
2
•Agata Łaszewska
1
•
Judit Simon
1
•Ruben M. W. A. Drost
2
•Dirk Ruwaard
2
•Silvia M. A. A. Evers
2,3
ÓThe Author(s) 2017. This article is an open access publication
Abstract
Background Intersectoral costs and benefits (ICBs), i.e.
costs and benefits of healthcare interventions outside the
healthcare sector, can be a crucial component in economic
evaluations from the societal perspective. Pivotal to their
estimation is the existence of sound resource-use mea-
surement (RUM) instruments; however, RUM instruments
for ICBs in the education or criminal justice sectors have
not yet been systematically collated or their psychometric
quality assessed. This review aims to fill this gap.
Methods To identify relevant instruments, the Database of
Instruments for Resource Use Measurement (DIRUM) was
searched. Additionally, a systematic literature review was
conducted in seven electronic databases to detect instru-
ments containing ICB items used in economic evaluations.
Finally, studies evaluating the psychometric quality of
these instruments were searched.
Results Twenty-six unique instruments were included.
Most frequently, ICB items measured school absenteeism,
tutoring, classroom assistance or contacts with legal
representatives, police custody/prison detainment and court
appearances, with the highest number of items listed in the
Client Service Receipt Inventory/Client Sociodemographic
and Service Receipt Inventory/Client Service Receipt
Inventory–Children’s Version (CSRI/CSSRI/CSRI-C),
Studying the Scope of Parental Expenditures (SCOPE) and
Self-Harm Intervention, Family Therapy (SHIFT) instru-
ments. ICBs in the education sector were especially rele-
vant for age-related developmental disorders and chronic
diseases, while criminal justice resource use seems more
important in mental health, including alcohol-related dis-
orders or substance abuse. Evidence on the validity or
reliability of ICB items was published for two instruments
only.
Conclusion With a heterogeneous variety of ICBs found to
be relevant for several disease areas but many ICB
instruments applied in one study only (21/26 instruments),
setting-up an international task force to, for example,
develop an internationally adaptable instrument is
recommended.
Electronic supplementary material The online version of this
article (doi:10.1007/s40273-017-0522-4) contains supplementary
material, which is available to authorized users.
&Susanne Mayer
susanne.mayer@meduniwien.ac.at
1
Department of Health Economics, Center for Public Health,
Medical University of Vienna, Kinderspitalgasse 15/1, 1090
Vienna, Austria
2
Department of Health Services Research, Care and Public
Health Research Institute (CAPHRI), Faculty of Health,
Medicine and Life Sciences, Maastricht University,
Maastricht, The Netherlands
3
Trimbos Institute, Netherlands Institute of Mental Health and
Addiction, Utrecht, The Netherlands
PharmacoEconomics
DOI 10.1007/s40273-017-0522-4
Key Points for Decision Makers
Health-related interventions often incur costs and
benefits outside the healthcare sector (i.e.
intersectoral costs and benefits [ICBs]) which, if
relevant, should be considered in economic
evaluations and cost-of-illness studies from the
broader societal analytical perspective to arrive at
unbiased conclusions.
Reliable and valid instruments to measure such
resource use in these sectors are pivotal. Based on a
systematic review of existing instruments used in
applied economic evaluations, this paper provides an
overview of generic and disease-specific resource-
use measurement instruments in the education and
criminal justice sectors, including their psychometric
properties. Relevant instruments will be included in
the Database of Instruments for Resource Use
Measurement (DIRUM; http://www.dirum.org).
Many instruments identified in this review were
specifically applied in one study only, potentially
implying considerable duplication of work across
studies. Hence, bundling individual efforts could be
a more cost-effective strategy overall, and setting-up
an international task force to support these activities,
e.g. by fostering methodological transparency and
developing an internationally adaptable, harmonized
instrument including relevant ICB items, is thus
suggested.
1 Introduction
For economic evaluations to be a useful tool to inform
policy makers regarding the allocation of scarce resources,
accurate measurement and valuation of all relevant costs
and benefits related to an intervention is key. Costs and
benefits related to healthcare interventions that occur out-
side the healthcare sector, i.e. intersectoral costs and ben-
efits (ICBs) [1,2], have been shown to be a vital
component in economic evaluations applying a societal
perspective [3–6]. At the same time, taking a societal
viewpoint in economic evaluations is recommended by
guidelines for the use of economic evaluation in several
countries [7], amounting to a proportion of approximately
two-thirds of the analyzed countries in 2010 [8], and most
recently also including the US [9]. The societal perspective
implies that all benefits and costs that are directly or
indirectly caused by the intervention (if significantly
present) should be included in the economic evaluation,
independently of who incurs them [10]. These costs and
benefits can occur outside the healthcare system, including
the education and criminal justice sectors, and household
and leisure activities, in addition to productivity loss [6].
In recent research, generic methods for valuing ICB unit
costs have been developed and applied in the context of the
Dutch education and criminal justice system [11]. How-
ever, the issue of how to measure such intersectoral
resource use in trial-based economic evaluations and cost-
of-illness studies with standardized resource-use measure-
ment (RUM) instruments has received little systematic
scientific attention to date [12]. This might hinder the
consideration of relevant ICB resource use in economic
evaluations in the first place [9], and consequently lead to
biased results from a societal perspective [13]. It also
potentially creates heterogeneity in the measurement due to
variabilities in considered cost components, which in turn
impairs the comparability of cross-study cost estimates.
In addition, it is unclear to what extent the quality of
existing ICB RUM instruments in terms of their psycho-
metric properties has been evaluated. Sound instruments
are pivotal to adequately capture the impact of healthcare
interventions in other sectors. A review of health-related
RUM instruments that led to the set-up of the Database of
Instruments for Resource Use Measurement (DIRUM) in
2011 [14] showed that limited evidence is available
regarding the validity, and especially reliability, of the
listed instruments [15]. This presumably also applies to
instruments measuring ICBs. One major exception includes
the measurement of lost productivity, for which both a
number of RUM instruments have been developed, vali-
dated, and the quality of their validation evidence critically
appraised in recent publications [16,17]. Similarly,
regarding ICBs in the household and leisure sectors,
instruments to measure informal care have been collected
and/or assessed in previous research [18–20]. This is not
yet the case for the education or criminal justice RUM
instruments.
Against this background, this research focuses on
health-related ICBs linked to resource use in the education
sector (i.e. related to the impairment of educational
achievement of individuals [2,9]) and criminal justice
sector (i.e. related to the costs of crimes [2,9]), and has two
aims. Our main aim was to provide a systematic overview
of the characteristics of current RUM instruments used in
health economic analyses that include ICB items capturing
the impacts on the education and criminal justice sectors.
Second, we sought to determine the existence of psycho-
metric quality assessments of these instruments. This will
not only provide practical guidance for researchers to
identify sound instruments for ICB RUM in specific dis-
ease areas but also help detect disease areas for which such
S. Mayer et al.
measurement in economic evaluations might be crucial. In
addition, it will provide an evidence-base to potentially
pioneer the development of standardized new items to
measure relevant costs and benefits outside the healthcare
sector.
2 Methods
2.1 Identification of Intersectoral Cost and Benefit
(ICB) Instruments
2.1.1 Databases
To identify instruments with relevant ICB items, a stepwise
approach was adopted. First, the open-access repository
DIRUM (http://www.dirum.org)[14], listing 77 RUM
instruments in July 2016, was hand-searched. DIRUM
contains full texts of health-related RUM instruments, as
well as information and/or references about respective
instrument qualities. DIRUM has a search function allow-
ing the selection of specific ‘items of resources being
measured’ (e.g. criminal justice service, educational, other
non-National Health Service), however, all of the 77 listed
instruments were examined in detail.
Second, as DIRUM is not exhaustive [21], an extensive
systematic literature search was conducted. Herein, the
adopted research approach rests on the assumption that
existing ICB instruments that measure such resource use
are cited and/or listed in economic evaluations and cost-of-
illness studies that were conducted from a societal per-
spective. Seven electronic databases were searched in July
2016, including MEDLINE (PubMed), EMBASE (Ovid),
Social Science Citation Index (SSCI; Web of Science),
PsycINFO, Econlit, Education Resources Information
Centre (ERIC) and CINAHL (EBSCOhost). This covers
two general medical literature databases (MEDLINE,
EMBASE) [22], which, in combination with CINAHL and
PsycINFO, are considered appropriate to identify economic
evaluations [23]. SSCI, Econlit and ERIC were included to
incorporate a social science, education and economic per-
spective, respectively. No date or language restrictions
were applied (electronic supplementary Appendix
Table A1). Both search strategy and database choices were
discussed and agreed with the project team and an infor-
mation scientist.
2.1.2 Inclusion and Exclusion Criteria
Eligibility criteria of the identified studies, framed around
PICOS [24], included all non-institutionalized and institu-
tionalized age groups of the population (P), regardless of
the intervention (I), comparators (C) and outcomes (O), set
up as a trial-based (non-simulation/non-model) full eco-
nomic evaluation (i.e. comparing both costs and outcomes
of the intervention group and at least one comparator) or as
a cost-of-illness study, adopting a societal perspective (i.e.
including costs in both the healthcare sector and other
sectors) as the study design (S) and measuring ICBs in their
analysis, published as a full paper or report. Consequently,
study exclusion criteria included no original research, no
full economic evaluation or cost-of-illness study, based on
model or simulation, and not adopting a societal perspec-
tive. Studies building on nationwide population surveys
were excluded, whereas articles focusing on the description
of RUM instruments were also screened. All studies ful-
filling the inclusion criteria irrespective of language were
reviewed for referenced patient/carer-reported ICB RUM
instruments in the education or criminal justice sectors (and
information on relevant psychometric evidence). Instru-
ments either had to measure education or criminal justice
resource use, or the article had to refer to a specified
(published) instrument that could potentially include such
items. Relevant information about the instrument was
extracted.
2.2 Retrieval of ICB Instruments
Full texts of instruments identified through DIRUM were
readily available from the DIRUM website [14]. Full texts
of the instruments identified through the database search of
economic evaluations and cost-of-illness studies, if not
included in the identified publication itself, were retrieved
via two channels. A Google (Scholar) search was con-
ducted first. If unsuccessful, the (corresponding) authors of
the articles were then contacted via email, and a full ver-
sion of the utilized instrument (and, in a second step,
information about the psychometric properties) was
requested. No language restrictions were applied in terms
of the included instruments, and translators were used for
extraction. Instruments had to be available as a full version
(for free).
2.3 Identification and Retrieval of Studies Assessing
Instrument Properties
Regarding psychometric properties that assess the quality
of the identified instruments, this review focuses on their
validity and reliability. Validity captures the degree to
which an instrument measures what it intends to measure,
while reliability refers to the ability of an instrument to do
this in a consistent manner [25]. To identify studies
assessing the instruments in this respect, the references of
the studies listed in DIRUM under ‘instrument qualities’
were investigated. For the instruments included based on
the systematic literature review, secondary database
ICB Resource-Use Measurement Instruments
searches were carried out based on the names of the
instruments and other relevant additional information
found in the initially reviewed studies. The same databases
were searched as for the identification of the instruments.
2.4 Screening of Studies and Instruments
The methodology and reporting of the systematic review
were generally set up to be consistent with the Guideline
for Conducting Systematic Literature Reviews in Eco-
nomic Evaluation [24], the methods proposed by the
Cochrane Collaboration [26], and the Preferred Reporting
Items for Systematic Reviews and Meta-Analyses
(PRISMA) statement [27]. The PRISMA checklist is
presented in electronic supplementary Appendix
Table A2. No review protocol was registered. Titles and
abstracts (level 1) were screened by SM. In addition, AŁ
conducted a second independent screening of the titles
and abstracts for 10% of all studies. Differences in
inclusion (level 1) were found in approximately 20% of
the studies, but did not affect the final inclusion of unique
instruments, for which close to no disagreement was
observed. This approach also seems justifiable given that
all studies that did not explicitly contradict the inclusion
criteria on level 1 were assessed further, independently of
actually mentioning a (specific or non-specific) instrument
in the title or abstract. Full-text screening of the articles
(level 2), instrument extraction, and the additional search
for psychometric evidence was independently conducted
by two reviewers (SM, AŁ) and any disagreement
resolved by discussion against the inclusion criteria. The
instrument extraction sheet, which partly follows the
taxonomy [15] designed for the set-up of DIRUM, was
discussed with the project team and piloted on 10% of the
included instruments.
3 Results
Of 3637 unique studies identified in the systematic review
of the literature, 167 full texts were included for assess-
ment of the instruments mentioned, cited or listed in these
studies (Fig. 1). Following the screening of the full
instruments, a total of 55 instruments were identified,
resulting in 26 unique RUM instruments with ICB items in
the education and/or criminal justice sectors after dedu-
plication (Table 1); 8 instruments were retrieved based on
the DIRUM search, 13 instruments were found based on
given references in economic evaluations and cost-of-ill-
ness studies, and 5 instruments came up in both searches.
Of the 26 instruments, five were included following email
correspondence with the study authors as full texts were
not publicly available. Twenty-one of the 26 instruments
were found to be used only once, i.e. in one study, for one
country.
A descriptive overview of the characteristics of the
included instruments is presented in Table 1. With the
exception of the Dutch [28] RUM instrument, all instru-
ments were published in English, mostly referring to the
UK healthcare system [29–41], the US [43–47], Australia
[48], Canada [49,50], Cambodia [51] and Kenya [52].
Three instruments [42,53,54] were designed for multi-
country studies, and were published in English and several
other European languages. Mostly, instruments were used
for research related to mental health problems or devel-
opmental disorders [28,29,31,38,39,42,43,49], alcohol-
related disorders or substance abuse [30,36,44,50,54],
chronic conditions [32,33,37,45] or other specific dis-
eases [34,35,41,46–48,51–53]. In total, 14 instruments
refer to children and/or adolescents [28,31–35,38,
39,43,46,49,51,55,56], three instruments refer to
adolescents and adults [45,47,53], and nine instruments
refer to the adult population only [29,30,36,42,44,
48,50,52,54]. Although many instruments were designed
for, or used in, multiple administration modes and forms of
recording, 9 [29,43–46,48,50,51,54] of the 26 instru-
ments were administered in an interview set-up, including
direct contact with a researcher.
3.1 ICB Instrument Content
All but two instruments not only include ICB-specific
items but also a variety of healthcare utilization RUM
questions. Education-related RUM items are included in 21
instruments, most frequently in instruments designed in the
context of various specific disease areas [34,35,41,
46–48,51–53], developmental disorders [31,38,39,
43,49] and chronic conditions [32,33,45,55]. Criminal
justice RUM items are included in 13 instruments, most
frequently related to developmental disorders [38,39,
43,49], alcohol-related disorders or substance abuse
[30,36,44,50,54] and mental health in general [29,42].
Seven instruments include items from both the education
and criminal justice sectors. Extracted details on the
instrument content regarding, for example, item wording
are presented in the electronic supplementary Appendix
Table A3.
As shown in Fig. 2, ICB items referring to the education
sector mostly cover time missed at school due to the
specific disease (15 items), followed by extra need for
tutoring activities (six items) and classroom assistance (six
items). Attendance at a special/boarding school is captured
by five items, as is social/school functioning. ICB items
referring to the criminal justice system most commonly
S. Mayer et al.
refer to the use of lawyers or general legal assistance (11
items), police custody or prison detainment (9 items),
appearances in criminal or civil courts (6 items), injuries (5
items), police contacts (5 items), or probation/correction
services (5 items) [Fig. 3].
3.2 Psychometric Properties of ICB Items
The psychometric properties were assessed for seven of the
identified instruments, namely the Child and Adolescent
Services Assessment (CASA) interview [43], Client
Fig. 1 PRISMA flowchart of electronic database search and DIRUM
search. PRISMA Preferred Reporting Items for Systematic Reviews
and Meta-Analyses, ICB intersectoral costs and benefits, DIRUM
Database of Instruments for Resource Use Measurement. Asterisk
refers to the instruments included from the DIRUM search
ICB Resource-Use Measurement Instruments
Table 1 Instrument administration and content
Q name, administration mode Language,
origin
Disease category Target
population
Person filling in
instrument
Recall period
(ICB)
Healthcare
RUM in
instrument
ICB:
education
sector
ICB:
criminal
justice
sector
Psychometric evidence, pilot
testing
Reference
and
instrument
identification
Mental health (general)
Bodden 2008: diary Dutch,
Netherlands
Anxiety disorder Children,
adolescents
Parents 2 weeks
(prospective)
4
a
4Validity
a
[28][28]; SLR
and E
Client Service Receipt Inventory
(CSRI): originally interview, paper-
based instrument
English, UK,
translated
into various
other
languages
Mental health Adults Patient/carer;
researcher
3 months 4
a
4Validity
a
[73,74]; validity of
Italian version [75], pilot
tested
[29]; D and
SLR
Client Sociodemographic and
Service Receipt Inventory—
European Version (CSSRI—EU):
paper-based instrument
English and
several other
languages,
Europe
Schizophrenia/mental
health
Adults Researcher 3 months 4
a
4Validity
a
[42][76][73,74];
reliability of German
version [77] (but no
details)
[42]; D and
SLR
Alcohol-related disorders
Alcohol: Evaluating Stepped care in
Older Populations Study (AESOPS)
Instrument: paper-based instrument
English, UK Alcoholism Adults Patient 6 months 44 [30]; D and
SLR
Health Service Utilization Inventory
(HSUI), modified: interview
English,
Canada
Fetal alcohol
spectrum disorder
Adults Parents 12 months 4
a
44Validity
a
[78][50]; SLR
and E
Parrott 2006: paper-based instrument English, UK Alcoholism Adults Patient 6 months 44 [36 ]; SLR
and E
Sommers 2011: interview English, USA Alcoholism and
substance abuse
Adults Researcher 12 months 444 [44]; SLR
and E
European Addiction Severity Index
(EuropASI): interview
English and
several other
languages,
Europe
Substance abuse Adults Researcher 1 month/
lifetime
4
a
4Pilot tested; validated in
several countries (e.g.
France, Germany, Czech
Republic) [54], see, for
example, Dutch version
[79]
[54,80];
SLR and E
Developmental disorders
Child and Adolescent Services
Assessment (CASA) child
interview/parent interview, version
5: interview/paper-based instrument
(mixture)
English, USA Developmental
problems; mental
health
Children,
adolescents
Researcher/
parent
3 months 4
a
44
a
Reliability,
a
validity (for
healthcare RUM only) [43]
[57,58]
[43]; SLR
Client Service Receipt Inventory
Children’s Version (CSRI-C):
originally paper-based instrument
English, UK Developmental,
psychosocial and
learning problems
Children Parent/carer;
researcher
12 months; per
week
44 Pilot tested [81][31]; D
Studying the Scope of Parental
Expenditures (SCOPE): web-based
survey
English,
Canada
Developmental
disorders
Children Parent 6 months; per
week
44Pilot tested [49][49]; SLR
Self-Harm Intervention, Family
Therapy (SHIFT) Parent or Carers
Questionnaire Booklet: postal
instrument
English, UK Developmental,
psychosocial and
learning problems
Adolescents Parent 3 months 444 [38]; D
S. Mayer et al.
Table 1 continued
Q name, administration mode Language,
origin
Disease category Target
population
Person filling in
instrument
Recall period
(ICB)
Healthcare
RUM in
instrument
ICB:
education
sector
ICB:
criminal
justice
sector
Psychometric evidence, pilot
testing
Reference
and
instrument
identification
Self-Harm Intervention, Family
Therapy (SHIFT) Young Person
Questionnaire Booklet: postal
instrument
English, UK Developmental,
psychosocial and
learning problems
Adolescents Patient 3 months 444 [39]; D
Chronic disease
Client Service Receipt Inventory for
Children with Diabetes (CSRI-CD):
paper-based instrument
English, UK Diabetes Children,
adolescents
Researcher (with
child or parent and
child)
1 month 44 [32]; D
Client Service Receipt Inventory for
Adolescents with Chronic Pain
(CSRI-Pain): paper-based
instrument
English, UK Chronic pain Adolescents Parents 12 months 444Pilot tested [82][33]; SLR
SubCutaneous Insulin: Pumps or
Injections (SCIPI) RUM: paper-
based instrument
English, UK Endocrine and
metabolic
Children,
adolescents
Researcher 3 months 44 [37,55]; D
Wetterneck 2006: interview English, USA Chronic hair pulling Adolescents,
adults
Researcher 3 months 44 [45]; SLR
Other diseases
Aygo
¨ren-Pu
¨rsu
¨n 2014: web- or
paper-based survey
English,
multicountry
(Spain,
Germany,
Denmark)
Hereditary
angioedema
Adolescents,
adults
Patient Specified
based on
attack;
6 months
44 Pilot tested [53][53]; SLR
Cost of Trauma Instrument (COTI):
mailed questionnaire/telephone
interview
English,
Australia
Orthopedics and
trauma
Adults Patient (parent for
child)
12 months 444 [48]; D and
SLR
HUGS V: Hemophilia Costs and
Impact of Disease Study, Version 2:
interview
English, USA Hemophilia Children,
adolescents
Parent
(patient \18 years
of age): version 2
1 month 44 [46]; SLR
and E
Huy 2009: interview English,
Cambodia
Dengue fever Children/
other
household
members
Parent/researcher Fever episode 4
a
4
a
Validity
a
(but no details
given), pilot tested [51]
[51]; SLR
MAGnesium NEbuliser Trial In
Children (MAGNETIC)
questionnaire: postal instrument
English, UK Lungs and airways Children,
adolescents
Parent 1 month 44 Pilot tested [34][34]; D
North of England and Scotland
Study of Tonsillectomy and Adeno-
tonsillectomy in Children
(NESSTAC) Parent’s
Questionnaire: postal instrument
English, UK Ear, nose and throat Children,
adolescents
Parent/carer 3 months 44 Pilot tested [35][35]; D
SafetY and Cost Effectiveness of
Adalimumab in Combination with
MethOTRExate (SYCAMORE):
diary and questionnaire
English, UK Eyes and vision Children,
adolescents
Researcher
(questionnaire);
patient (diary)
3 months
(prospective)
44 [41][56]; D
ICB Resource-Use Measurement Instruments
Service Receipt Inventory (CSRI) [29], Client Sociode-
mographic and Service Receipt Inventory—European
Version (CSSRI—EU) [42], cost diary by Bodden et al.
[28], Work Productivity and Activity Impairment (WPAI)
instrument [47], Health Service Utilization Inventory
(HSUI) [50], and European Addiction Severity Index
(EuropASI) [54]. In case of one instrument [51], validation
was mentioned but no further details were given. In the
economic evaluations citing the included instruments, most
authors provided details on the development of the
instrument, including, for example, the development team
or information on which other instrument the relevant
instrument is based on. Pilot testing of the instrument was
reported for nine instruments.
Psychometric assessment of ICB-related items could be
confirmed for two of these seven instruments; for the
CASA instrument [43], both reliability and validity were
determined. Test–retest reliability was found to be high for
the most intensively used services, including juvenile jus-
tice; however, services used in the child’s natural setting,
for example school services, were found to be reported
with low reliability [57]. The assessment of concurrent
validity (i.e. correlation of service use measured with two
instruments) was restricted to mental health service use
only, and, again, varied for individual services [58]. Con-
struct validity and reproducibility of the general WPAI
Questionnaire was established in earlier research [59]. For
the WPAI Questionnaire plus Classroom Impairment
Questions: Specific Health Problem (WPAI ?CIQ:SHP,
version 2.0) specifically, psychometric evidence is avail-
able for its allergy-specific (AS) first version (the
WPAI ?CIQ:AS), testing its discriminative and evalua-
tive validity [60]. Except for time missed from the class-
room, allergic rhinitis symptoms were found to be well-
correlated for most impairment measures (discriminative
validity). The same conclusion applies to the correlation
analysis on the change in symptoms and change in time
missed from the classroom (evaluative validity).
4 Discussion and Conclusion
Conducting economic evaluations from a broader societal
perspective rather than from a narrow healthcare viewpoint
is increasingly acknowledged as the gold standard [61], and
also manifests in pharmacoeconomic guidelines across
Europe [7] and in the US [9]. To be able to consider health-
related impacts of interventions in the education or crimi-
nal justice sectors in practice, sound instruments measuring
relevant resource use are a prerequisite. This review
identified a total of 26 unique instruments with a variety of
such ICB items. Most frequently, ICB items in the edu-
cation sector measured resource use due to school
Table 1 continued
Q name, administration mode Language,
origin
Disease category Target
population
Person filling in
instrument
Recall period
(ICB)
Healthcare
RUM in
instrument
ICB:
education
sector
ICB:
criminal
justice
sector
Psychometric evidence, pilot
testing
Reference
and
instrument
identification
The Tool to Estimate Patients’
Costs: paper-based questionnaire
English, Kenya Tuberculosis Adults Researcher During
treatment
44 Pilot tested [83][52]; D and
SLR
Work Productivity and Activity
Impairment Questionnaire plus
Classroom Impairment Questions:
Specific Health Problem Version 2.0
(WPAI ?CIQ:SHP, V2.0)
English, USA Adaptable to specific
diseases/health
problems
Adolescents,
adults
Patient 1 week 4
a
Validity
a
[60][47]; SLR
Qquestionnaire, DQ full-text retrieved via search in DIRUM, SLR Q identified via systematic literature review, EQ full-text received following e-mail correspondence with study author(s), ICB intersectoral costs and benefits, RUM
resource-use measurement, DIRUM Database of Instruments for Resource Use Measurement, 4indicates yes (included)
a
Psychometric evidence (partly) assessed for these sections of the instrument (for details, see column ‘Psychometric evidence’)
S. Mayer et al.
absenteeism, tutoring activities, classroom assistance and
school or social functioning. Resource use related to the
criminal justice sector most commonly captured legal
assistance, police custody or prison detainment, criminal or
civil court appearance, injuries, police contacts or the use
of probation/correction services. Psychometric evidence
was found to be examined for seven instruments that, with
two exceptions [43,47], focused only on healthcare RUM
items, hence generally leaving out ICB items from their
assessment. For the two exceptions, validity (discrimina-
tive and evaluative) was assessed for one instrument
(WPAI ?CIQ:AS [47]), and reliability (test–retest) was
assessed for another (CASA [43]).
From a methodological perspective, compared with
patient-reported outcome measures, methods of RUM within
economic evaluations have been less of a focal point [21].
Specifically, a culture of psychometric validation of RUM
instruments for self-reported measurement is lacking in
health economics [21,62]. In a review of UK Health Tech-
nology Assessment (HTA) program funds trials, only
approximately 30% of the studies were found to report on
validation of their RUM data [63]. This review confirms such
Fig. 2 Education-related ICB items in the instruments. Instrument
references by ICB items: (1) absenteeism from school
[28,32,34,35,39,41,43–45,47,51,53,55]; (2) tutoring
[31,33,41,43]; (3) classroom assistance [31,33,38,39,49,56]; (4)
special school/boarding school [31,33,38,39,43,49]; (5) school
functioning [35,45,46]; (6) social functioning [46]; (7) adolescence/
school counsellor [38,39,50]; (8) special school teacher [33,43]; (9)
special needs statement [31,38]; (10) special class [43]; (11) school
dropout [48,52]; (12) other educational services [38,39]
[31,35,47–49,51]. As more than one item per topic may be
contained in an instrument, the number of references do not
necessarily add up to the numbers indicated in the figure. ICB
intersectoral costs and benefits
Fig. 3 Criminal justice-related ICB items in the instruments. Instru-
ment references by ICB item: (1) lawyer/legal assistance
[33,36,38,39,49,50]; (2) police custody/prison detainment
[29,30,42,44,50,54]; (3) criminal/civil court appearance
[29,30,42,44,50]; (4) injury [36]; (5) police contact
[29,38,39,42,50]; (6) probation/correction services
[38,39,43,50,54]; (7) aggressiveness/violence [36,54]; (8) traffic
accident [36,44]; (9) psychiatric assessment in custody [29,42]; (10)
property damage [38,48,54]; (11) youth offending team [38,39];
(12) other services [29,42][36,38,44,54]. As more than one item
per topic may be contained in an instrument, the number of references
do not necessarily add up to the numbers indicated in the figure. ICB
intersectoral costs and benefits
ICB Resource-Use Measurement Instruments
lack of psychometric evidence, also specifically for ICB
items. Indeed, establishing validity, in particular for ICB in
the education or criminal justice sectors, might prove even
more challenging than for non-ICB items [64]. For example,
it might prove difficult, in practice, to get access to alterna-
tive data sources (e.g. administrative information) for items
such as the number and/or duration of police contacts, psy-
chiatric assessments, received legal assistance, or the num-
ber of traffic tickets (electronic supplementary Appendix
Table A3) to establish some degree of criterion validity [15].
This is a drawback, especially given that criminal justice
resource use is likely a sensitive topic and potentially rather
prone to, for example, measurement error [36]. For instance,
earlier research suggests that a social desirability bias is more
likely to be present in interview situations with a researcher
than, for example, self-administered questionnaires [64,89].
This issue could thus also be relevant for existing ICB items,
given that an interview-type administration mode was used
for 9 of the 26 identified instruments. Lack of psychometric
assessment was also found with regard to piloting of the
instruments, although pilot-testing is another critical step in
ensuring reliability [63,65]. For 9 of the 26 instruments
included in this review, pilot-testing was reported, which is
comparable to the proportion identified in the UK HTA
review [63].
A recent systematic literature review by Leggett et al.
[12] also analyses RUM instruments. RUM items outside
the healthcare sector, including travel expenses, out-of-
pocket costs and productivity losses, were concluded to be
particularly rare in the 15 identified instruments. Other ICB
items were not discussed in the review, which was gener-
ally restricted to publicly available, validated instruments
designed for adult populations only. Overlap between the
instruments identified by Leggett et al. [12] and this
research is consequently limited to the CSRI instrument
only. The lacking overlap is also likely due to this review’s
indirect search approach (Sect. 2.1), contrasting the com-
mon method of directly identifying validated instruments
through literature search in earlier reviews [12,16,66].
4.1 Strengths and Limitations
This review is the first to give a comprehensive overview
of instruments including ICB items for measurement of
resource use in the education or criminal justice system. It
discloses both a heterogeneous variety of existing ICB
RUM items in these sectors and reveals a lack of psycho-
metric evaluation of these instruments. The latter finding
strongly supports the choice of the adopted indirect search
strategy. Relevant new instruments identified in the course
of this review will be fed into DIRUM, which will help
future researchers identify and retrieve instruments with
ICB items in a more efficient manner. Indeed, the fact that
the majority (21 of 26) of the included instruments were
found to be specifically designed for and used in one study
only points out substantial efficiency potential for
researchers through sharing instruments. Finally, in most
cases, the identified ICB items are part of broader instru-
ments; in this sense, this review also provides an overview
of existing international instruments for healthcare RUM,
albeit not being the initial aim.
Note though that several limitations apply to the instru-
ment identification strategy. Firstly, instruments that gener-
ally contain ICB items, but were used in an economic
evaluation from a healthcare perspective only, were poten-
tially overlooked. However, all instruments that were spec-
ified in the studies were checked for relevant ICB RUM
items, even if inclusion of such elements was not to be
expected from the respective economic evaluation or cost-
of-illness analysis. At the same time, the high number of
studies using non-specified, non-referenced, non-listed
instruments detected in this review is striking. For example,
of all 167 studies included for instrument assessment,
approximately one in five articles (n=31) reported using,
for example, some ‘(adapted) standardized’ instruments or a
general ‘economic’ instrument, and failed to provide more
details. This lack of methodological transparency in instru-
ment reporting, which was also seen in earlier research [64],
hampers not only the potential comparability of these eco-
nomic evaluations but also impairs the quality assessment of
the measurement of the included cost components.
Secondly, this review was restricted to instruments
mentioned in published health-related economic evalua-
tions or cost-of-illness studies, and thus health services
research in general. Looking at other disciplines such as,
for example, education research, economic evaluations
seem to be applied rather sparsely [67]. However, addi-
tional RUM instruments might be found, for example, in
the literature on education program impacts [66]. Future
research might thus want to consider looking into instru-
ments developed in other disciplines. Trial registries could
be an additional channel to identify more recent ICB
measurement instruments.
Thirdly, comparable to the review by Leggett et al. [12],
a quality assessment of the psychometric evidence was not
carried out. This is a limitation since not only the validation
findings themselves are crucial but also the quality of the
methodology applied [16]. However, given the currently
limited evidence published on instrument quality for ICB
items (2 of 26 instruments), at this point such an assess-
ment would be premature. Once more relevant studies have
been conducted, the quality of the validation studies could
be checked, e.g. by applying the COSMIN (COnsensus-
based Standards for the selection of health Measurement
INstruments) checklist [68], as was done by recent research
[16,66,69].
S. Mayer et al.
4.2 Research and Policy Implications
When choosing an RUM instrument for use in a trial-based
economic evaluation, it is crucial that this instrument
covers the domains that are needed to capture the real-
world economic consequences of an intervention in the
specific disease area [42]. Based on the included instru-
ments, ICB RUM items in the education sector seem par-
ticularly relevant not only for age-related developmental
disorders affecting children and adolescents but also
chronic diseases, including diabetes and chronic pain. In
contrast, criminal justice service resource use seems more
important in the fields of alcoholism and substance abuse,
and developmental disorders, as well as mental health in
general. Future economic analyses, particularly in these
fields, should thus consider measuring ICBs from the
education and criminal justice sectors, respectively.
Against the backdrop of the recently developed ICB clas-
sification scheme [2], the majority of ICB items in the edu-
cation sector are captured by the existing instruments. Most
items are found in the CSRI–Children’s Version (CSRI-C)
[31], Studying the Scope of Parental Expenditures (SCOPE;
based on CSRI) [49] and Self-Harm Intervention, Family
Therapy (SHIFT) instruments [38,39]. Regarding the crimi-
nal justice sector, most ICB items are included in the CSRI
[29], CSSRI [42], SHIFT [38] and the instrument developed
by Sommers et al. [44]. However, compared with the ICB
classification scheme, existing instruments lack ICB items
regarding, for example, child maltreatment, sexual assaults
and crime consequences on victims, which should be taken
into account when developing new items.
Besides validating existing ICB items, future research
could focus on the development of a harmonized new
instrument with a broad variety of relevant ICB elements.
Indeed motivated by the fact that RUM instruments lack
validation, RUM items from the DIRUM database are cur-
rently being reviewed by Thorn et al. [62] with the aim of
developing a standardized RUM instrument for the UK. This
instrument will focus on the health and social care sector and
will exclude ICBs. Therefore, the first step in developing a
new ICB instrument would include a literature search to
identify the main cost-driving elements from relevant eco-
nomic evaluations in a specific disease area [21]. This was
selectively performed in prior research [11,70]; however,
given that the inclusion of ICBs does not seem to have a long
tradition [71], these empirical studies might be missing such
cost elements for this very reason [72]. Thus, collating ICBs
that were mentioned in studies but not necessarily measured
appears more reasonable [2]. The second step in the devel-
opment of a new instrument [21] includes the identification
of existing instruments, which was also one of the main aims
of this review. Following this step, focus groups with
healthcare professionals, experts in the education/criminal
justice sectors and patients to discuss these instruments
should be organized to develop a first version of a new
instrument. Many instruments identified in this review were
specifically developed for use in one study only, which
potentially implies considerable duplication of work hap-
pening across studies. Therefore, bundling individual efforts
could be a more cost-effective strategy overall. Setting-up an
international task force to support these activities by, for
example, reflecting on and exploring different sources of, for
instance, electronic data as a means of validation, and also
developing an internationally adaptable, validated instru-
ment, could be the next step. Such an initiative should take
into account the state of the art of RUM classifications by
detailing the key components of an RUM [64,71]. Following
a structured taxonomy such as the one developed by Ridyard
et al. [64], by providing a description of the data source, who
completes the instrument, the administration mode, and the
methods and medium of recording, clarity and method-
ological transparency could be established in the develop-
ment of such a new instrument.
Data Availability Statement The authors declare that
the data supporting the findings of this study are available
within the article and its supplementary electronic infor-
mation files. Full instruments not directly retrieved from
the literature but following email communication with the
instrument developer may be directly requested from them
based on the information provided (Table 1, ‘Reference
and instrument identification’). Relevant instruments will
be included in DIRUM (http://www.dirum.org).
Acknowledgements Many thanks to Gregor Franssen (Information
Scientist at Maastricht University) for expert advice on setting up the
search strategy, and Marianne Tilly for her support with study full-
text retrieval. The authors also gratefully acknowledge the help of
several members of the Department of Health Services Research at
Maastricht University with language translations. Feedback from
Joanna Thorn on an earlier version of the manuscript was greatly
appreciated. We also gratefully acknowledge the valuable input from
participants at the 2017 lowlands Health Economists’ Study Group
(lolaHESG) conference in Rotterdam, The Netherlands. This manu-
script was drafted during Susanne Mayer’s research stay at Maastricht
University.
Author Contributions The initial idea for this study was first
brought up by SE and AP. SM developed the study concept, to which
input was provided by AP, JS, RD, DR and SE. All co-authors were
involved in the discussion of all steps of the review. SM conducted
the literature search, with AŁ as the second screener. SM analyzed the
data and wrote the manuscript, and all authors read and approved the
final version of this article.
Compliance with Ethical Standards
Conflicts of interest Susanne Mayer, Aggie T.G. Paulus, Agata
Łaszewska, Judit Simon, Ruben M.W.A. Drost, Dirk Ruwaard, Silvia
M.A.A. Evers declare that they have no conflicts of interest relevant
to the content of this article.
ICB Resource-Use Measurement Instruments
Funding No funding from external sources was received for this
work.
Open Access This article is distributed under the terms of the
Creative Commons Attribution-NonCommercial 4.0 International
License (http://creativecommons.org/licenses/by-nc/4.0/), which per-
mits any noncommercial use, distribution, and reproduction in any
medium, provided you give appropriate credit to the original
author(s) and the source, provide a link to the Creative Commons
license, and indicate if changes were made.
References
1. Weatherly H, Drummond M, Claxton K, et al. Methods for
assessing the cost-effectiveness of public health interventions:
key challenges and recommendations. Health Policy.
2009;93(2–3):85–92.
2. Drost RM, Paulus AT, Ruwaard D, Evers SM. Inter-sectoral costs
and benefits of mental health prevention: towards a new classi-
fication scheme. J Ment Health Policy Econ. 2013;16(4):179–86.
3. Krol M, Brouwer W, Rutten F. Productivity costs in economic
evaluations: past, present, future. Pharmacoeconomics.
2013;31(7):537–49.
4. Krol M, Papenburg J, van Exel J. Does including informal care in
economic evaluations matter? A systematic review of inclusion
and impact of informal care in cost-effectiveness studies. Phar-
macoeconomics. 2015;33(2):123–35.
5. Byford S, Raftery J. Perspectives in economic evaluation. BMJ.
1998;316(7143):1529–30.
6. Drost RM, van der Putten IM, Ruwaard D, Evers SM, Paulus AT.
Conceptualizations of the societal perspective within economic
evaluations: systematic review. Int J Technol Assess Health Care
(accepted).
7. Claxton K, Walker S, Palmer S, Sculpher M. Appropriate per-
spectives for health care decisions. Working Papers no 054cherp,
Centre for Health Economics, University of York; 2010.
8. Knies S, Severens JL, Ament AJ, Evers SM. The transferability
of valuing lost productivity across jurisdictions. Differences
between national pharmacoeconomic guidelines. Value Health.
2010;13(5):519–27.
9. Sanders GD, Neumann PJ, Basu A, et al. Recommendations for
conduct, methodological practices, and reporting of cost-effec-
tiveness analyses: second panel on cost-effectiveness in health
and medicine. JAMA. 2016;316(10):1093–103.
10. Gold MR, Siegel JE, Russell LB, Weinstein MC. Cost-effec-
tiveness in health and medicine. New York: Oxford University
Press; 1996.
11. Drost RM, Paulus AT, Ruwaard D, Evers SM. Valuing inter-
sectoral costs and benefits of interventions in the healthcare
sector: methods for obtaining unit prices. Expert Rev Pharma-
coecon Outcomes Res. 2017;17(1):77–84.
12. Leggett LE, Khadaroo RG, Holroyd-Leduc J, et al. Measuring
resource utilization: a systematic review of validated self-re-
ported questionnaires. Medicine. 2016;95(10):e2759.
13. Evers SM, Hiligsmann M, Adarkwah C. Risk of bias in trial-
based economic evaluations: identification of sources and bias-
reducing strategies. Psychol Health. 2015;30(1):52–71.
14. DIRUM, Database of Instruments for Resource Use Measure-
ment. http://www.dirum.org/. Accessed 28 Jul 2016.
15. Ridyard CH, Dyfrig A, DIRUM Team. Development of a data-
base of instruments for resource-use measurement: purpose,
feasibility, and design. Value Health. 2012;15(5):650–5.
16. Noben CY, Evers SM, Nijhuis FJ, de Rijk AE. Quality appraisal
of generic self-reported instruments measuring health-related
productivity changes: a systematic review. BMC Public Health.
2014;14(1):1.
17. Tang K. Estimating productivity costs in health economic eval-
uations: a review of instruments and psychometric evidence.
Pharmacoeconomics. 2015;33(1):31–48.
18. Tanco K, Park JC, Cerana A, et al. A systematic review of
instruments assessing dimensions of distress among caregivers of
adult and pediatric cancer patients. Palliat Supp Care.
2017;15(1):110–24.
19. Hoefman RJ, van Exel J, Brouwer W. How to include informal
care in economic evaluations. Pharmacoeconomics.
2013;31(12):1105–19.
20. van Durme T, Macq J, Jeanmart C, Gobert M. Tools for mea-
suring the impact of informal caregiving of the elderly: a litera-
ture review. Int J Nurs Stud. 2012;49(4):490–504.
21. Thorn JC, Coast J, Cohen D, et al. Resource-use measurement
based on patient recall: issues and challenges for economic
evaluation. Appl Health Econ Health Policy. 2013;11(3):155–61.
22. Glanville J, Paisley S. Identifying economic evaluations for
health technology assessment. Int J Technol Assess Health Care.
2010;26(4):436–40.
23. Alton V, Eckerlund I, Norlund A. Health economic evaluations:
how to find them. Int J Technol Assess Health Care.
2006;22(4):512–7.
24. Akers J, Aguiar-Iba
´n
˜ez R, Baba-Akbari Sari A. CRD’s guidance
for undertaking reviews in health care. York: Centre for Reviews
and Dissemination (CRD); 2009.
25. Streiner DL, Norman GR, Cairney J. Health measurement scales:
a practical guide to their development and use. Oxford: Oxford
University Press; 2014.
26. Higgins JP, Green S. Cochrane Handbook for Systematic
Reviews of Interventions. Version 5.1.0 2011. http://handbook.
cochrane.org/. Accessed 21 June 2016.
27. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group.
Preferred reporting items for systematic reviews and meta-anal-
yses: the PRISMA statement. Ann Intern Med.
2009;151(4):264–9.
28. Bodden DH, Dirksen CD, Bogels SM. Societal burden of clini-
cally anxious youth referred for treatment: a cost-of-illness study.
J Abnorm Child Psychol. 2008;36(4):487–97.
29. Beecham J, Knapp M. Costing psychiatric interventions. In:
Thornicroft G, editor. Measuring mental health needs. London:
Gaskell; 2001. p. 200–24.
30. Watson JM, Crosby H, Dale VM, et al. AESOPS: a randomised
controlled trial of the clinical effectiveness and cost-effectiveness
of opportunistic screening and stepped care interventions for
older hazardous alcohol users in primary care. Health Technol
Assess. 2013;17(25):1–158.
31. Client Service Receipt Inventory Children’s Version. http://www.
dirum.org/instruments/details/45. Accessed 1 Aug 2017.
32. CSRI-CD, Client Service Receipt Inventory for Children with
Diabetes. http://www.dirum.org/instruments/details/53#. Acces-
sed 1 Aug 2017.
33. CSRI-Pain, Client Service Receipt Inventory for Adolescents
with Chronic Pain—Parent report. http://www.pssru.ac.uk/blogs/
csri/chronic-pain-csri/. Accessed 1 Aug 2017.
34. Powell CV, Kolamunnage-Dona R, Lowe J, et al. MAGNEsium
Trial In Children (MAGNETIC): a randomised, placebo-con-
trolled trial and economic evaluation of nebulised magnesium
sulphate in acute severe asthma in children. Health Technol
Assess. 2013;17(45):1–216.
35. Lock C, Wilson J, Steen N, et al. North of England and Scotland
Study of Tonsillectomy and Adeno-tonsillectomy in Children
(NESSTAC): a pragmatic randomised controlled trial with a
S. Mayer et al.
parallel non-randomised preference study. Health Technol
Assess. 2010;14(13):1–164, iii–iv.
36. Parrott S, Godfrey C, Heather N, Clark J, Ryan T. Cost and
outcome analysis of two alcohol detoxification services. Alcohol
Alcohol. 2006;41(1):84–91.
37. DIRUM: SCIPI RUM: instrument qualities 2016. http://
www.dirum.org/instruments/details/79#. Accessed 1 Aug 2017.
38. SHIFT, Self-Harm Intervention, Family Therapy (SHIFT), Parent
or Carers Questionnaire Booklet. http://www.dirum.org/assets/
downloads/634426220067245338-SHIFT%20Parent%20or%20
Carers%20Questionnaire%20Booklet.pdf. Accessed 1 Aug
2017.
39. SHIFT, Self-Harm Intervention, Family Therapy (SHIFT),
Young Person Questionnaire Booklet. http://www.dirum.org/
assets/downloads/634426213718379721-SHIFT%20Young%20
Person%20Questionnaire%20Booklet.pdf. Accessed 1 Aug
2017.
40. Barrett B, Byford S. Collecting service use data for economic
evaluation in DSPD populations: development of the Secure
Facilities Service Use Schedule. Br J Psychiatry Suppl.
2007;49:s75–8.
41. SYCAMORE CSRI Questionnaire. http://www.dirum.org/assets/
downloads/635028335262813537-SYCAMORE_CSRI_question
naire.pdf. Accessed 1 Aug 2017.
42. Chisholm D, Knapp MR, Knudsen HC, et al. Client Socio-De-
mographic and Service Receipt Inventory-European Version:
development of an instrument for international research. EPSI-
LON Study 5. European Psychiatric Services: Inputs Linked to
Outcome Domains and Needs. Br J Psychiatry Suppl.
2000;39:s28–33.
43. Burns BJ, Angold A, Magruder-Habib K, et al. The child and
adolescent services assessment (CASA) Child Interview Version
5.0. http://devepi.duhs.duke.edu/casa.html. Accessed 1 Aug 2017.
44. Sommers BD, Fargo JD, Lyons MS, Shope JT, Sommers MS.
Societal costs of risky driving: an economic analysis of high-risk
patients visiting an urban emergency department. Traffic Inj Prev.
2011;12(2):149–58.
45. Wetterneck CT, Woods DW, Norberg MM, Begotka AM. The
social and economic impact of trichotillomania: results from two
nonreferred samples. Behav Intervent. 2006;21(2):97–109.
46. Zhou ZY, Koerper MA, Johnson KA, et al. Burden of illness:
direct and indirect costs among persons with hemophilia A in the
United States. J Med Econ. 2015;18(6):457–65.
47. WPAI ?CIQ:AS. Work Productivity and Activity Impairment
Questionnaire plus Classroom Impairment Questions: Allergy
Specific, Version 2. http://www.reillyassociates.net/WPAI_AS.
html. Accessed 1 Aug 2017.
48. Connelly LB, Webber J, Rowell D. Cost of Trauma Instrument
(COTI), 2005. http://www.conrod.org.au/cms/resources-and-
tools/cost-of-trauma-instrument-coti. Accessed 1 Aug 2017.
49. Genereaux D, van Karnebeek CD, Birch PH. Costs of caring for
children with an intellectual developmental disorder. Disabil
Health J. 2015;8(4):646–51.
50. Browne G, Gafni A, Roberts J, Goldsmith A, Jamieson E.
Approach to the measurement of costs (expenditures) when
evaluating health and social programmes. Hamilton: McMaster
University; 1995.
51. Huy R, Wichmann O, Beatty M, et al. Cost of dengue and other
febrile illnesses to households in rural Cambodia: a prospective
community-based case-control study. BMC Public Health.
2009;9:155.
52. KNCV Tuberculosis Foundation. The Tool to Estimate Patients’
Costs, 2008. http://www.biomedcentral.com/content/
supplementary/1471-2458-11-43-S1.PDF. Accessed 1 Aug 2017.
53. Aygoren-Pursun E, Bygum A, Beusterien K, et al. Socioeco-
nomic burden of hereditary angioedema: results from the
hereditary angioedema burden of illness study in Europe.
Orphanet J Rare Dis. 2014;9:99.
54. Blacken PH, Pozzi G, Tempesta E, et al. European Addiction
Severity Index (EuropASI) Cost A6: A guide to training and
administering EuropASI interviews 1994. http://www.emcdda.
europa.eu/html.cfm/index3647EN.html. Accessed 1 Aug 2017.
55. SCIPI RUM, SubCutaneous Insulin: Pumps or Injections (SCIPI)
RUM, 2011. http://www.dirum.org/assets/downloads/
634968699228637202-SCIPI_Questionnaire.pdf. Accessed 1
Aug 2017.
56. Diary SC. Syncamore Treatment Diary. http://www.dirum.org/
assets/downloads/635750685168619541-Sycamore_Treatment_
Diary.pdf. Accessed 1 Aug 2017.
57. Farmer EM, Angold A, Burns BJ, Costello EJ. Reliability of self-
reported service use: test-retest consistency of children’s
responses to the Child and Adolescent Services Assessment
(CASA). J Child Fam Stud. 1994;3(3):307–25.
58. Ascher BH, Farmer EM, Burns BJ, Angold A. The child and
adolescent services assessment (CASA) description and psycho-
metrics. J Emot Behav Disord. 1996;4(1):12–20.
59. Reilly MC, Zbrozek AS, Dukes EM. The validity and repro-
ducibility of a work productivity and activity impairment
instrument. Pharmacoeconomics. 1993;4(5):353–65.
60. Reilly MC, Tanner A, Meltzer EO. Work, classroom and activity
impairment instruments: validation studies in allergic rhinitis.
Clin Drug Investig. 1996;11(5):278–88.
61. Ten Jo
¨nsson B. arguments for a societal perspective in the eco-
nomic evaluation of medical innovations. Eur J Health Econ.
2009;10(4):357–9.
62. Thorn JC, Ridyard CH, Riley R, et al. Identification of items for a
standardised resource-use measure: review of current instru-
ments. Value Health. 2015;18(7):A688.
63. Ridyard CH, Hughes DA. Methods for the collection of resource
use data within clinical trials: a systematic review of studies
funded by the UK Health Technology Assessment program.
Value Health. 2010;13(8):867–72.
64. Ridyard CH, Hughes DA, DIRUM Team. Taxonomy for methods
of resource use measurement. Health Econ. 2015;24(3):372–8.
65. Johnston K, Buxton MJ, Jones DR, Fitzpatrick R. Assessing the
costs of healthcare technologies in clinical trials. Health Technol
Assess. 1999;3(6):1–76.
66. Schellingerhout JM, Verhagen AP, Heymans MW, et al. Mea-
surement properties of disease-specific questionnaires in patients
with neck pain: a systematic review. Qual Life Res.
2012;21(4):659–70.
67. Ladd HF, Edward EB. Handbook of research in education finance
and policy. Abingdon: Routledge; 2012.
68. Terwee CB, Mokkink LB, Knol DL, et al. Rating the method-
ological quality in systematic reviews of studies on measurement
properties: a scoring system for the COSMIN checklist. Qual Life
Res. 2012;21(4):651–7.
69. van Ballegooijen W, Riper H, Cuijpers P, van Oppen P, Smit JH.
Validation of online psychometric instruments for common
mental health disorders: a systematic review. BMC Psychiatry.
2016;16:45.
70. Goodrich K, Kaambwa B, Al-Janabi H. The inclusion of informal
care in applied economic evaluation: a review. Value Health.
2012;15(6):975–81.
71. Ridyard CH, Hughes DA. Review of resource-use measures in
UK economic evaluations. In: Curtis L, Burns A, editors. Unit
Costs of Health and Social Care 2015. Canterbury: Personal
Social Services Research Unit, University of Kent; 2015.
p. 22–30.
72. Mayer S, Kiss N, Laszewska A, Simon J. Health economic
costing methods and reporting in Austria. Value Health.
2016;19(7):A363.
ICB Resource-Use Measurement Instruments
73. Patel A, Rendu A, Moran P, et al. A comparison of two methods
of collecting economic data in primary care. Fam Pract.
2005;22(3):323–7.
74. Byford S, Leese M, Knapp M, et al. Comparison of alternative
methods of collection of service use data for the economic
evaluation of health care interventions. Health Econ.
2007;16(5):531–6.
75. Mirandola M, Bisoffi G, Bonizzato P, Amaddeo F. Collecting
psychiatric resources utilisation data to calculate costs of care: a
comparison between a service receipt interview and a case reg-
ister. Soc Psychiatry Psychiatr Epidemiol. 1999;34(10):541–7.
76. Heinrich S, Deister A, Birker T, et al. Accuracy of self-reports of
mental health care utilization and calculated costs compared to
hospital records. Psychiatry Res. 2011;185(1–2):261–8.
77. Kilian R, Roick C, Bernert S, et al. Instruments for the eco-
nomical evaluation of psychiatric service systems: methodologi-
cal foundations of the european standardisation and the German
adaptation [in German]. Psychiatr Prax. 2001;28(Suppl 2):74–8.
78. Browne GB, Arpin K, Corey P, Fitch M, Gafni A. Individual
correlates of health service utilization and the cost of poor
adjustment to chronic illness. Med Care. 1990;28(1):43–58.
79. Hendriks VM, Kaplan CD, van Limbeek J, Geerlings P. The
Addiction Severity Index: reliability and validity in a Dutch
addict population. J Subst Abuse Treat. 1989;6(2):133–41.
80. Dijkgraaf MG, van der Zanden BP, de Borgie CA, et al. Cost
utility analysis of co-prescribed heroin compared with methadone
maintenance treatment in heroin addicts in two randomised trials.
BMJ. 2005;330(7503):1297–300.
81. Knapp M, Scott S, Davies J. The cost of antisocial behaviour in
younger children. Clin Child Psychol Psychiatry. 1999;4(4):
457–73.
82. Sleed M, Eccleston C, Beecham J, Knapp M, Jordan A. The
economic impact of chronic pain in adolescence: methodological
considerations and a preliminary cost-of-illness study. Pain.
2005;119(1–3):183–90.
83. Mauch V, Woods N, Kirubi B, et al. Assessing access barriers to
tuberculosis care with the tool to Estimate Patients’ Costs: pilot
results from two districts in Kenya. BMC Public Health.
2011;11(1):1.
84. UKATT Research Team. Cost effectiveness of treatment for
alcohol problems: findings of the randomised UK alcohol treat-
ment trial (UKATT). BMJ. 2005;331(7516):544.
85. STEPWICE Research Group. The effectiveness and cost-effec-
tiveness of screening and stepped care interventions for alcohol
use disorders in the primary care setting. Cardiff: Welsh Office of
Research & Development; 2003.
86. Coulton S, Watson J, Bland M, et al. The effectiveness and cost-
effectiveness of opportunistic screening and stepped care inter-
ventions for older hazardous alcohol users in primary care
(AESOPS)—a randomised control trial protocol. BMC Health
Serv Res. 2008;8:129.
87. Stade B, Ali A, Bennett D, et al. The burden of prenatal exposure
to alcohol: revised measurement of cost. Can J Clin Pharmacol.
2009;16(1):e91–102.
88. Ramanan AV, Dick AD, Benton D, et al. A randomised con-
trolled trial of the clinical effectiveness, SafetY and cost effec-
tiveness of adalimumab in combination with MethOtRExate for
the treatment of juvenile idiopathic arthritis associated uveitis
(SYCAMORE). Trials. 2014;9:14.
89. Bowling A. Mode of questionnaire administration can have
serious effects on data quality. J Public Health.
2005;27(3):281–91.
S. Mayer et al.
