ArticlePDF AvailableLiterature Review


Background: Intersectoral costs and benefits (ICBs), i.e. costs and benefits of healthcare interventions outside the healthcare sector, can be a crucial component in economic evaluations from the societal perspective. Pivotal to their estimation is the existence of sound resource-use measurement (RUM) instruments; however, RUM instruments for ICBs in the education or criminal justice sectors have not yet been systematically collated or their psychometric quality assessed. This review aims to fill this gap. Methods: To identify relevant instruments, the Database of Instruments for Resource Use Measurement (DIRUM) was searched. Additionally, a systematic literature review was conducted in seven electronic databases to detect instruments containing ICB items used in economic evaluations. Finally, studies evaluating the psychometric quality of these instruments were searched. Results: Twenty-six unique instruments were included. Most frequently, ICB items measured school absenteeism, tutoring, classroom assistance or contacts with legal representatives, police custody/prison detainment and court appearances, with the highest number of items listed in the Client Service Receipt Inventory/Client Sociodemographic and Service Receipt Inventory/Client Service Receipt Inventory-Children's Version (CSRI/CSSRI/CSRI-C), Studying the Scope of Parental Expenditures (SCOPE) and Self-Harm Intervention, Family Therapy (SHIFT) instruments. ICBs in the education sector were especially relevant for age-related developmental disorders and chronic diseases, while criminal justice resource use seems more important in mental health, including alcohol-related disorders or substance abuse. Evidence on the validity or reliability of ICB items was published for two instruments only. Conclusion: With a heterogeneous variety of ICBs found to be relevant for several disease areas but many ICB instruments applied in one study only (21/26 instruments), setting-up an international task force to, for example, develop an internationally adaptable instrument is recommended.
Health-Related Resource-Use Measurement Instruments
for Intersectoral Costs and Benefits in the Education and Criminal
Justice Sectors
Susanne Mayer
Aggie T. G. Paulus
Agata Łaszewska
Judit Simon
Ruben M. W. A. Drost
Dirk Ruwaard
Silvia M. A. A. Evers
ÓThe Author(s) 2017. This article is an open access publication
Background Intersectoral costs and benefits (ICBs), i.e.
costs and benefits of healthcare interventions outside the
healthcare sector, can be a crucial component in economic
evaluations from the societal perspective. Pivotal to their
estimation is the existence of sound resource-use mea-
surement (RUM) instruments; however, RUM instruments
for ICBs in the education or criminal justice sectors have
not yet been systematically collated or their psychometric
quality assessed. This review aims to fill this gap.
Methods To identify relevant instruments, the Database of
Instruments for Resource Use Measurement (DIRUM) was
searched. Additionally, a systematic literature review was
conducted in seven electronic databases to detect instru-
ments containing ICB items used in economic evaluations.
Finally, studies evaluating the psychometric quality of
these instruments were searched.
Results Twenty-six unique instruments were included.
Most frequently, ICB items measured school absenteeism,
tutoring, classroom assistance or contacts with legal
representatives, police custody/prison detainment and court
appearances, with the highest number of items listed in the
Client Service Receipt Inventory/Client Sociodemographic
and Service Receipt Inventory/Client Service Receipt
Inventory–Children’s Version (CSRI/CSSRI/CSRI-C),
Studying the Scope of Parental Expenditures (SCOPE) and
Self-Harm Intervention, Family Therapy (SHIFT) instru-
ments. ICBs in the education sector were especially rele-
vant for age-related developmental disorders and chronic
diseases, while criminal justice resource use seems more
important in mental health, including alcohol-related dis-
orders or substance abuse. Evidence on the validity or
reliability of ICB items was published for two instruments
Conclusion With a heterogeneous variety of ICBs found to
be relevant for several disease areas but many ICB
instruments applied in one study only (21/26 instruments),
setting-up an international task force to, for example,
develop an internationally adaptable instrument is
Electronic supplementary material The online version of this
article (doi:10.1007/s40273-017-0522-4) contains supplementary
material, which is available to authorized users.
&Susanne Mayer
Department of Health Economics, Center for Public Health,
Medical University of Vienna, Kinderspitalgasse 15/1, 1090
Vienna, Austria
Department of Health Services Research, Care and Public
Health Research Institute (CAPHRI), Faculty of Health,
Medicine and Life Sciences, Maastricht University,
Maastricht, The Netherlands
Trimbos Institute, Netherlands Institute of Mental Health and
Addiction, Utrecht, The Netherlands
DOI 10.1007/s40273-017-0522-4
Key Points for Decision Makers
Health-related interventions often incur costs and
benefits outside the healthcare sector (i.e.
intersectoral costs and benefits [ICBs]) which, if
relevant, should be considered in economic
evaluations and cost-of-illness studies from the
broader societal analytical perspective to arrive at
unbiased conclusions.
Reliable and valid instruments to measure such
resource use in these sectors are pivotal. Based on a
systematic review of existing instruments used in
applied economic evaluations, this paper provides an
overview of generic and disease-specific resource-
use measurement instruments in the education and
criminal justice sectors, including their psychometric
properties. Relevant instruments will be included in
the Database of Instruments for Resource Use
Measurement (DIRUM;
Many instruments identified in this review were
specifically applied in one study only, potentially
implying considerable duplication of work across
studies. Hence, bundling individual efforts could be
a more cost-effective strategy overall, and setting-up
an international task force to support these activities,
e.g. by fostering methodological transparency and
developing an internationally adaptable, harmonized
instrument including relevant ICB items, is thus
1 Introduction
For economic evaluations to be a useful tool to inform
policy makers regarding the allocation of scarce resources,
accurate measurement and valuation of all relevant costs
and benefits related to an intervention is key. Costs and
benefits related to healthcare interventions that occur out-
side the healthcare sector, i.e. intersectoral costs and ben-
efits (ICBs) [1,2], have been shown to be a vital
component in economic evaluations applying a societal
perspective [36]. At the same time, taking a societal
viewpoint in economic evaluations is recommended by
guidelines for the use of economic evaluation in several
countries [7], amounting to a proportion of approximately
two-thirds of the analyzed countries in 2010 [8], and most
recently also including the US [9]. The societal perspective
implies that all benefits and costs that are directly or
indirectly caused by the intervention (if significantly
present) should be included in the economic evaluation,
independently of who incurs them [10]. These costs and
benefits can occur outside the healthcare system, including
the education and criminal justice sectors, and household
and leisure activities, in addition to productivity loss [6].
In recent research, generic methods for valuing ICB unit
costs have been developed and applied in the context of the
Dutch education and criminal justice system [11]. How-
ever, the issue of how to measure such intersectoral
resource use in trial-based economic evaluations and cost-
of-illness studies with standardized resource-use measure-
ment (RUM) instruments has received little systematic
scientific attention to date [12]. This might hinder the
consideration of relevant ICB resource use in economic
evaluations in the first place [9], and consequently lead to
biased results from a societal perspective [13]. It also
potentially creates heterogeneity in the measurement due to
variabilities in considered cost components, which in turn
impairs the comparability of cross-study cost estimates.
In addition, it is unclear to what extent the quality of
existing ICB RUM instruments in terms of their psycho-
metric properties has been evaluated. Sound instruments
are pivotal to adequately capture the impact of healthcare
interventions in other sectors. A review of health-related
RUM instruments that led to the set-up of the Database of
Instruments for Resource Use Measurement (DIRUM) in
2011 [14] showed that limited evidence is available
regarding the validity, and especially reliability, of the
listed instruments [15]. This presumably also applies to
instruments measuring ICBs. One major exception includes
the measurement of lost productivity, for which both a
number of RUM instruments have been developed, vali-
dated, and the quality of their validation evidence critically
appraised in recent publications [16,17]. Similarly,
regarding ICBs in the household and leisure sectors,
instruments to measure informal care have been collected
and/or assessed in previous research [1820]. This is not
yet the case for the education or criminal justice RUM
Against this background, this research focuses on
health-related ICBs linked to resource use in the education
sector (i.e. related to the impairment of educational
achievement of individuals [2,9]) and criminal justice
sector (i.e. related to the costs of crimes [2,9]), and has two
aims. Our main aim was to provide a systematic overview
of the characteristics of current RUM instruments used in
health economic analyses that include ICB items capturing
the impacts on the education and criminal justice sectors.
Second, we sought to determine the existence of psycho-
metric quality assessments of these instruments. This will
not only provide practical guidance for researchers to
identify sound instruments for ICB RUM in specific dis-
ease areas but also help detect disease areas for which such
S. Mayer et al.
measurement in economic evaluations might be crucial. In
addition, it will provide an evidence-base to potentially
pioneer the development of standardized new items to
measure relevant costs and benefits outside the healthcare
2 Methods
2.1 Identification of Intersectoral Cost and Benefit
(ICB) Instruments
2.1.1 Databases
To identify instruments with relevant ICB items, a stepwise
approach was adopted. First, the open-access repository
DIRUM ([14], listing 77 RUM
instruments in July 2016, was hand-searched. DIRUM
contains full texts of health-related RUM instruments, as
well as information and/or references about respective
instrument qualities. DIRUM has a search function allow-
ing the selection of specific ‘items of resources being
measured’ (e.g. criminal justice service, educational, other
non-National Health Service), however, all of the 77 listed
instruments were examined in detail.
Second, as DIRUM is not exhaustive [21], an extensive
systematic literature search was conducted. Herein, the
adopted research approach rests on the assumption that
existing ICB instruments that measure such resource use
are cited and/or listed in economic evaluations and cost-of-
illness studies that were conducted from a societal per-
spective. Seven electronic databases were searched in July
2016, including MEDLINE (PubMed), EMBASE (Ovid),
Social Science Citation Index (SSCI; Web of Science),
PsycINFO, Econlit, Education Resources Information
Centre (ERIC) and CINAHL (EBSCOhost). This covers
two general medical literature databases (MEDLINE,
EMBASE) [22], which, in combination with CINAHL and
PsycINFO, are considered appropriate to identify economic
evaluations [23]. SSCI, Econlit and ERIC were included to
incorporate a social science, education and economic per-
spective, respectively. No date or language restrictions
were applied (electronic supplementary Appendix
Table A1). Both search strategy and database choices were
discussed and agreed with the project team and an infor-
mation scientist.
2.1.2 Inclusion and Exclusion Criteria
Eligibility criteria of the identified studies, framed around
PICOS [24], included all non-institutionalized and institu-
tionalized age groups of the population (P), regardless of
the intervention (I), comparators (C) and outcomes (O), set
up as a trial-based (non-simulation/non-model) full eco-
nomic evaluation (i.e. comparing both costs and outcomes
of the intervention group and at least one comparator) or as
a cost-of-illness study, adopting a societal perspective (i.e.
including costs in both the healthcare sector and other
sectors) as the study design (S) and measuring ICBs in their
analysis, published as a full paper or report. Consequently,
study exclusion criteria included no original research, no
full economic evaluation or cost-of-illness study, based on
model or simulation, and not adopting a societal perspec-
tive. Studies building on nationwide population surveys
were excluded, whereas articles focusing on the description
of RUM instruments were also screened. All studies ful-
filling the inclusion criteria irrespective of language were
reviewed for referenced patient/carer-reported ICB RUM
instruments in the education or criminal justice sectors (and
information on relevant psychometric evidence). Instru-
ments either had to measure education or criminal justice
resource use, or the article had to refer to a specified
(published) instrument that could potentially include such
items. Relevant information about the instrument was
2.2 Retrieval of ICB Instruments
Full texts of instruments identified through DIRUM were
readily available from the DIRUM website [14]. Full texts
of the instruments identified through the database search of
economic evaluations and cost-of-illness studies, if not
included in the identified publication itself, were retrieved
via two channels. A Google (Scholar) search was con-
ducted first. If unsuccessful, the (corresponding) authors of
the articles were then contacted via email, and a full ver-
sion of the utilized instrument (and, in a second step,
information about the psychometric properties) was
requested. No language restrictions were applied in terms
of the included instruments, and translators were used for
extraction. Instruments had to be available as a full version
(for free).
2.3 Identification and Retrieval of Studies Assessing
Instrument Properties
Regarding psychometric properties that assess the quality
of the identified instruments, this review focuses on their
validity and reliability. Validity captures the degree to
which an instrument measures what it intends to measure,
while reliability refers to the ability of an instrument to do
this in a consistent manner [25]. To identify studies
assessing the instruments in this respect, the references of
the studies listed in DIRUM under ‘instrument qualities’
were investigated. For the instruments included based on
the systematic literature review, secondary database
ICB Resource-Use Measurement Instruments
searches were carried out based on the names of the
instruments and other relevant additional information
found in the initially reviewed studies. The same databases
were searched as for the identification of the instruments.
2.4 Screening of Studies and Instruments
The methodology and reporting of the systematic review
were generally set up to be consistent with the Guideline
for Conducting Systematic Literature Reviews in Eco-
nomic Evaluation [24], the methods proposed by the
Cochrane Collaboration [26], and the Preferred Reporting
Items for Systematic Reviews and Meta-Analyses
(PRISMA) statement [27]. The PRISMA checklist is
presented in electronic supplementary Appendix
Table A2. No review protocol was registered. Titles and
abstracts (level 1) were screened by SM. In addition,
conducted a second independent screening of the titles
and abstracts for 10% of all studies. Differences in
inclusion (level 1) were found in approximately 20% of
the studies, but did not affect the final inclusion of unique
instruments, for which close to no disagreement was
observed. This approach also seems justifiable given that
all studies that did not explicitly contradict the inclusion
criteria on level 1 were assessed further, independently of
actually mentioning a (specific or non-specific) instrument
in the title or abstract. Full-text screening of the articles
(level 2), instrument extraction, and the additional search
for psychometric evidence was independently conducted
by two reviewers (SM, AŁ) and any disagreement
resolved by discussion against the inclusion criteria. The
instrument extraction sheet, which partly follows the
taxonomy [15] designed for the set-up of DIRUM, was
discussed with the project team and piloted on 10% of the
included instruments.
3 Results
Of 3637 unique studies identified in the systematic review
of the literature, 167 full texts were included for assess-
ment of the instruments mentioned, cited or listed in these
studies (Fig. 1). Following the screening of the full
instruments, a total of 55 instruments were identified,
resulting in 26 unique RUM instruments with ICB items in
the education and/or criminal justice sectors after dedu-
plication (Table 1); 8 instruments were retrieved based on
the DIRUM search, 13 instruments were found based on
given references in economic evaluations and cost-of-ill-
ness studies, and 5 instruments came up in both searches.
Of the 26 instruments, five were included following email
correspondence with the study authors as full texts were
not publicly available. Twenty-one of the 26 instruments
were found to be used only once, i.e. in one study, for one
A descriptive overview of the characteristics of the
included instruments is presented in Table 1. With the
exception of the Dutch [28] RUM instrument, all instru-
ments were published in English, mostly referring to the
UK healthcare system [2941], the US [4347], Australia
[48], Canada [49,50], Cambodia [51] and Kenya [52].
Three instruments [42,53,54] were designed for multi-
country studies, and were published in English and several
other European languages. Mostly, instruments were used
for research related to mental health problems or devel-
opmental disorders [28,29,31,38,39,42,43,49], alcohol-
related disorders or substance abuse [30,36,44,50,54],
chronic conditions [32,33,37,45] or other specific dis-
eases [34,35,41,4648,5153]. In total, 14 instruments
refer to children and/or adolescents [28,3135,38,
39,43,46,49,51,55,56], three instruments refer to
adolescents and adults [45,47,53], and nine instruments
refer to the adult population only [29,30,36,42,44,
48,50,52,54]. Although many instruments were designed
for, or used in, multiple administration modes and forms of
recording, 9 [29,4346,48,50,51,54] of the 26 instru-
ments were administered in an interview set-up, including
direct contact with a researcher.
3.1 ICB Instrument Content
All but two instruments not only include ICB-specific
items but also a variety of healthcare utilization RUM
questions. Education-related RUM items are included in 21
instruments, most frequently in instruments designed in the
context of various specific disease areas [34,35,41,
4648,5153], developmental disorders [31,38,39,
43,49] and chronic conditions [32,33,45,55]. Criminal
justice RUM items are included in 13 instruments, most
frequently related to developmental disorders [38,39,
43,49], alcohol-related disorders or substance abuse
[30,36,44,50,54] and mental health in general [29,42].
Seven instruments include items from both the education
and criminal justice sectors. Extracted details on the
instrument content regarding, for example, item wording
are presented in the electronic supplementary Appendix
Table A3.
As shown in Fig. 2, ICB items referring to the education
sector mostly cover time missed at school due to the
specific disease (15 items), followed by extra need for
tutoring activities (six items) and classroom assistance (six
items). Attendance at a special/boarding school is captured
by five items, as is social/school functioning. ICB items
referring to the criminal justice system most commonly
S. Mayer et al.
refer to the use of lawyers or general legal assistance (11
items), police custody or prison detainment (9 items),
appearances in criminal or civil courts (6 items), injuries (5
items), police contacts (5 items), or probation/correction
services (5 items) [Fig. 3].
3.2 Psychometric Properties of ICB Items
The psychometric properties were assessed for seven of the
identified instruments, namely the Child and Adolescent
Services Assessment (CASA) interview [43], Client
Fig. 1 PRISMA flowchart of electronic database search and DIRUM
search. PRISMA Preferred Reporting Items for Systematic Reviews
and Meta-Analyses, ICB intersectoral costs and benefits, DIRUM
Database of Instruments for Resource Use Measurement. Asterisk
refers to the instruments included from the DIRUM search
ICB Resource-Use Measurement Instruments
Table 1 Instrument administration and content
Q name, administration mode Language,
Disease category Target
Person filling in
Recall period
RUM in
Psychometric evidence, pilot
Mental health (general)
Bodden 2008: diary Dutch,
Anxiety disorder Children,
Parents 2 weeks
[28][28]; SLR
and E
Client Service Receipt Inventory
(CSRI): originally interview, paper-
based instrument
English, UK,
into various
Mental health Adults Patient/carer;
3 months 4
[73,74]; validity of
Italian version [75], pilot
[29]; D and
Client Sociodemographic and
Service Receipt Inventory—
European Version (CSSRI—EU):
paper-based instrument
English and
several other
Adults Researcher 3 months 4
reliability of German
version [77] (but no
[42]; D and
Alcohol-related disorders
Alcohol: Evaluating Stepped care in
Older Populations Study (AESOPS)
Instrument: paper-based instrument
English, UK Alcoholism Adults Patient 6 months 44 [30]; D and
Health Service Utilization Inventory
(HSUI), modified: interview
Fetal alcohol
spectrum disorder
Adults Parents 12 months 4
[78][50]; SLR
and E
Parrott 2006: paper-based instrument English, UK Alcoholism Adults Patient 6 months 44 [36 ]; SLR
and E
Sommers 2011: interview English, USA Alcoholism and
substance abuse
Adults Researcher 12 months 444 [44]; SLR
and E
European Addiction Severity Index
(EuropASI): interview
English and
several other
Substance abuse Adults Researcher 1 month/
4Pilot tested; validated in
several countries (e.g.
France, Germany, Czech
Republic) [54], see, for
example, Dutch version
SLR and E
Developmental disorders
Child and Adolescent Services
Assessment (CASA) child
interview/parent interview, version
5: interview/paper-based instrument
English, USA Developmental
problems; mental
3 months 4
validity (for
healthcare RUM only) [43]
[43]; SLR
Client Service Receipt Inventory
Children’s Version (CSRI-C):
originally paper-based instrument
English, UK Developmental,
psychosocial and
learning problems
Children Parent/carer;
12 months; per
44 Pilot tested [81][31]; D
Studying the Scope of Parental
Expenditures (SCOPE): web-based
Children Parent 6 months; per
44Pilot tested [49][49]; SLR
Self-Harm Intervention, Family
Therapy (SHIFT) Parent or Carers
Questionnaire Booklet: postal
English, UK Developmental,
psychosocial and
learning problems
Adolescents Parent 3 months 444 [38]; D
S. Mayer et al.
Table 1 continued
Q name, administration mode Language,
Disease category Target
Person filling in
Recall period
RUM in
Psychometric evidence, pilot
Self-Harm Intervention, Family
Therapy (SHIFT) Young Person
Questionnaire Booklet: postal
English, UK Developmental,
psychosocial and
learning problems
Adolescents Patient 3 months 444 [39]; D
Chronic disease
Client Service Receipt Inventory for
Children with Diabetes (CSRI-CD):
paper-based instrument
English, UK Diabetes Children,
Researcher (with
child or parent and
1 month 44 [32]; D
Client Service Receipt Inventory for
Adolescents with Chronic Pain
(CSRI-Pain): paper-based
English, UK Chronic pain Adolescents Parents 12 months 444Pilot tested [82][33]; SLR
SubCutaneous Insulin: Pumps or
Injections (SCIPI) RUM: paper-
based instrument
English, UK Endocrine and
Researcher 3 months 44 [37,55]; D
Wetterneck 2006: interview English, USA Chronic hair pulling Adolescents,
Researcher 3 months 44 [45]; SLR
Other diseases
¨n 2014: web- or
paper-based survey
Patient Specified
based on
6 months
44 Pilot tested [53][53]; SLR
Cost of Trauma Instrument (COTI):
mailed questionnaire/telephone
Orthopedics and
Adults Patient (parent for
12 months 444 [48]; D and
HUGS V: Hemophilia Costs and
Impact of Disease Study, Version 2:
English, USA Hemophilia Children,
(patient \18 years
of age): version 2
1 month 44 [46]; SLR
and E
Huy 2009: interview English,
Dengue fever Children/
Parent/researcher Fever episode 4
(but no details
given), pilot tested [51]
[51]; SLR
MAGnesium NEbuliser Trial In
Children (MAGNETIC)
questionnaire: postal instrument
English, UK Lungs and airways Children,
Parent 1 month 44 Pilot tested [34][34]; D
North of England and Scotland
Study of Tonsillectomy and Adeno-
tonsillectomy in Children
(NESSTAC) Parent’s
Questionnaire: postal instrument
English, UK Ear, nose and throat Children,
Parent/carer 3 months 44 Pilot tested [35][35]; D
SafetY and Cost Effectiveness of
Adalimumab in Combination with
diary and questionnaire
English, UK Eyes and vision Children,
patient (diary)
3 months
44 [41][56]; D
ICB Resource-Use Measurement Instruments
Service Receipt Inventory (CSRI) [29], Client Sociode-
mographic and Service Receipt Inventory—European
Version (CSSRI—EU) [42], cost diary by Bodden et al.
[28], Work Productivity and Activity Impairment (WPAI)
instrument [47], Health Service Utilization Inventory
(HSUI) [50], and European Addiction Severity Index
(EuropASI) [54]. In case of one instrument [51], validation
was mentioned but no further details were given. In the
economic evaluations citing the included instruments, most
authors provided details on the development of the
instrument, including, for example, the development team
or information on which other instrument the relevant
instrument is based on. Pilot testing of the instrument was
reported for nine instruments.
Psychometric assessment of ICB-related items could be
confirmed for two of these seven instruments; for the
CASA instrument [43], both reliability and validity were
determined. Test–retest reliability was found to be high for
the most intensively used services, including juvenile jus-
tice; however, services used in the child’s natural setting,
for example school services, were found to be reported
with low reliability [57]. The assessment of concurrent
validity (i.e. correlation of service use measured with two
instruments) was restricted to mental health service use
only, and, again, varied for individual services [58]. Con-
struct validity and reproducibility of the general WPAI
Questionnaire was established in earlier research [59]. For
the WPAI Questionnaire plus Classroom Impairment
Questions: Specific Health Problem (WPAI ?CIQ:SHP,
version 2.0) specifically, psychometric evidence is avail-
able for its allergy-specific (AS) first version (the
WPAI ?CIQ:AS), testing its discriminative and evalua-
tive validity [60]. Except for time missed from the class-
room, allergic rhinitis symptoms were found to be well-
correlated for most impairment measures (discriminative
validity). The same conclusion applies to the correlation
analysis on the change in symptoms and change in time
missed from the classroom (evaluative validity).
4 Discussion and Conclusion
Conducting economic evaluations from a broader societal
perspective rather than from a narrow healthcare viewpoint
is increasingly acknowledged as the gold standard [61], and
also manifests in pharmacoeconomic guidelines across
Europe [7] and in the US [9]. To be able to consider health-
related impacts of interventions in the education or crimi-
nal justice sectors in practice, sound instruments measuring
relevant resource use are a prerequisite. This review
identified a total of 26 unique instruments with a variety of
such ICB items. Most frequently, ICB items in the edu-
cation sector measured resource use due to school
Table 1 continued
Q name, administration mode Language,
Disease category Target
Person filling in
Recall period
RUM in
Psychometric evidence, pilot
The Tool to Estimate Patients’
Costs: paper-based questionnaire
English, Kenya Tuberculosis Adults Researcher During
44 Pilot tested [83][52]; D and
Work Productivity and Activity
Impairment Questionnaire plus
Classroom Impairment Questions:
Specific Health Problem Version 2.0
English, USA Adaptable to specific
Patient 1 week 4
[60][47]; SLR
Qquestionnaire, DQ full-text retrieved via search in DIRUM, SLR Q identified via systematic literature review, EQ full-text received following e-mail correspondence with study author(s), ICB intersectoral costs and benefits, RUM
resource-use measurement, DIRUM Database of Instruments for Resource Use Measurement, 4indicates yes (included)
Psychometric evidence (partly) assessed for these sections of the instrument (for details, see column ‘Psychometric evidence’)
S. Mayer et al.
absenteeism, tutoring activities, classroom assistance and
school or social functioning. Resource use related to the
criminal justice sector most commonly captured legal
assistance, police custody or prison detainment, criminal or
civil court appearance, injuries, police contacts or the use
of probation/correction services. Psychometric evidence
was found to be examined for seven instruments that, with
two exceptions [43,47], focused only on healthcare RUM
items, hence generally leaving out ICB items from their
assessment. For the two exceptions, validity (discrimina-
tive and evaluative) was assessed for one instrument
(WPAI ?CIQ:AS [47]), and reliability (test–retest) was
assessed for another (CASA [43]).
From a methodological perspective, compared with
patient-reported outcome measures, methods of RUM within
economic evaluations have been less of a focal point [21].
Specifically, a culture of psychometric validation of RUM
instruments for self-reported measurement is lacking in
health economics [21,62]. In a review of UK Health Tech-
nology Assessment (HTA) program funds trials, only
approximately 30% of the studies were found to report on
validation of their RUM data [63]. This review confirms such
Fig. 2 Education-related ICB items in the instruments. Instrument
references by ICB items: (1) absenteeism from school
[28,32,34,35,39,41,4345,47,51,53,55]; (2) tutoring
[31,33,41,43]; (3) classroom assistance [31,33,38,39,49,56]; (4)
special school/boarding school [31,33,38,39,43,49]; (5) school
functioning [35,45,46]; (6) social functioning [46]; (7) adolescence/
school counsellor [38,39,50]; (8) special school teacher [33,43]; (9)
special needs statement [31,38]; (10) special class [43]; (11) school
dropout [48,52]; (12) other educational services [38,39]
[31,35,4749,51]. As more than one item per topic may be
contained in an instrument, the number of references do not
necessarily add up to the numbers indicated in the figure. ICB
intersectoral costs and benefits
Fig. 3 Criminal justice-related ICB items in the instruments. Instru-
ment references by ICB item: (1) lawyer/legal assistance
[33,36,38,39,49,50]; (2) police custody/prison detainment
[29,30,42,44,50,54]; (3) criminal/civil court appearance
[29,30,42,44,50]; (4) injury [36]; (5) police contact
[29,38,39,42,50]; (6) probation/correction services
[38,39,43,50,54]; (7) aggressiveness/violence [36,54]; (8) traffic
accident [36,44]; (9) psychiatric assessment in custody [29,42]; (10)
property damage [38,48,54]; (11) youth offending team [38,39];
(12) other services [29,42][36,38,44,54]. As more than one item
per topic may be contained in an instrument, the number of references
do not necessarily add up to the numbers indicated in the figure. ICB
intersectoral costs and benefits
ICB Resource-Use Measurement Instruments
lack of psychometric evidence, also specifically for ICB
items. Indeed, establishing validity, in particular for ICB in
the education or criminal justice sectors, might prove even
more challenging than for non-ICB items [64]. For example,
it might prove difficult, in practice, to get access to alterna-
tive data sources (e.g. administrative information) for items
such as the number and/or duration of police contacts, psy-
chiatric assessments, received legal assistance, or the num-
ber of traffic tickets (electronic supplementary Appendix
Table A3) to establish some degree of criterion validity [15].
This is a drawback, especially given that criminal justice
resource use is likely a sensitive topic and potentially rather
prone to, for example, measurement error [36]. For instance,
earlier research suggests that a social desirability bias is more
likely to be present in interview situations with a researcher
than, for example, self-administered questionnaires [64,89].
This issue could thus also be relevant for existing ICB items,
given that an interview-type administration mode was used
for 9 of the 26 identified instruments. Lack of psychometric
assessment was also found with regard to piloting of the
instruments, although pilot-testing is another critical step in
ensuring reliability [63,65]. For 9 of the 26 instruments
included in this review, pilot-testing was reported, which is
comparable to the proportion identified in the UK HTA
review [63].
A recent systematic literature review by Leggett et al.
[12] also analyses RUM instruments. RUM items outside
the healthcare sector, including travel expenses, out-of-
pocket costs and productivity losses, were concluded to be
particularly rare in the 15 identified instruments. Other ICB
items were not discussed in the review, which was gener-
ally restricted to publicly available, validated instruments
designed for adult populations only. Overlap between the
instruments identified by Leggett et al. [12] and this
research is consequently limited to the CSRI instrument
only. The lacking overlap is also likely due to this review’s
indirect search approach (Sect. 2.1), contrasting the com-
mon method of directly identifying validated instruments
through literature search in earlier reviews [12,16,66].
4.1 Strengths and Limitations
This review is the first to give a comprehensive overview
of instruments including ICB items for measurement of
resource use in the education or criminal justice system. It
discloses both a heterogeneous variety of existing ICB
RUM items in these sectors and reveals a lack of psycho-
metric evaluation of these instruments. The latter finding
strongly supports the choice of the adopted indirect search
strategy. Relevant new instruments identified in the course
of this review will be fed into DIRUM, which will help
future researchers identify and retrieve instruments with
ICB items in a more efficient manner. Indeed, the fact that
the majority (21 of 26) of the included instruments were
found to be specifically designed for and used in one study
only points out substantial efficiency potential for
researchers through sharing instruments. Finally, in most
cases, the identified ICB items are part of broader instru-
ments; in this sense, this review also provides an overview
of existing international instruments for healthcare RUM,
albeit not being the initial aim.
Note though that several limitations apply to the instru-
ment identification strategy. Firstly, instruments that gener-
ally contain ICB items, but were used in an economic
evaluation from a healthcare perspective only, were poten-
tially overlooked. However, all instruments that were spec-
ified in the studies were checked for relevant ICB RUM
items, even if inclusion of such elements was not to be
expected from the respective economic evaluation or cost-
of-illness analysis. At the same time, the high number of
studies using non-specified, non-referenced, non-listed
instruments detected in this review is striking. For example,
of all 167 studies included for instrument assessment,
approximately one in five articles (n=31) reported using,
for example, some ‘(adapted) standardized’ instruments or a
general ‘economic’ instrument, and failed to provide more
details. This lack of methodological transparency in instru-
ment reporting, which was also seen in earlier research [64],
hampers not only the potential comparability of these eco-
nomic evaluations but also impairs the quality assessment of
the measurement of the included cost components.
Secondly, this review was restricted to instruments
mentioned in published health-related economic evalua-
tions or cost-of-illness studies, and thus health services
research in general. Looking at other disciplines such as,
for example, education research, economic evaluations
seem to be applied rather sparsely [67]. However, addi-
tional RUM instruments might be found, for example, in
the literature on education program impacts [66]. Future
research might thus want to consider looking into instru-
ments developed in other disciplines. Trial registries could
be an additional channel to identify more recent ICB
measurement instruments.
Thirdly, comparable to the review by Leggett et al. [12],
a quality assessment of the psychometric evidence was not
carried out. This is a limitation since not only the validation
findings themselves are crucial but also the quality of the
methodology applied [16]. However, given the currently
limited evidence published on instrument quality for ICB
items (2 of 26 instruments), at this point such an assess-
ment would be premature. Once more relevant studies have
been conducted, the quality of the validation studies could
be checked, e.g. by applying the COSMIN (COnsensus-
based Standards for the selection of health Measurement
INstruments) checklist [68], as was done by recent research
S. Mayer et al.
4.2 Research and Policy Implications
When choosing an RUM instrument for use in a trial-based
economic evaluation, it is crucial that this instrument
covers the domains that are needed to capture the real-
world economic consequences of an intervention in the
specific disease area [42]. Based on the included instru-
ments, ICB RUM items in the education sector seem par-
ticularly relevant not only for age-related developmental
disorders affecting children and adolescents but also
chronic diseases, including diabetes and chronic pain. In
contrast, criminal justice service resource use seems more
important in the fields of alcoholism and substance abuse,
and developmental disorders, as well as mental health in
general. Future economic analyses, particularly in these
fields, should thus consider measuring ICBs from the
education and criminal justice sectors, respectively.
Against the backdrop of the recently developed ICB clas-
sification scheme [2], the majority of ICB items in the edu-
cation sector are captured by the existing instruments. Most
items are found in the CSRI–Children’s Version (CSRI-C)
[31], Studying the Scope of Parental Expenditures (SCOPE;
based on CSRI) [49] and Self-Harm Intervention, Family
Therapy (SHIFT) instruments [38,39]. Regarding the crimi-
nal justice sector, most ICB items are included in the CSRI
[29], CSSRI [42], SHIFT [38] and the instrument developed
by Sommers et al. [44]. However, compared with the ICB
classification scheme, existing instruments lack ICB items
regarding, for example, child maltreatment, sexual assaults
and crime consequences on victims, which should be taken
into account when developing new items.
Besides validating existing ICB items, future research
could focus on the development of a harmonized new
instrument with a broad variety of relevant ICB elements.
Indeed motivated by the fact that RUM instruments lack
validation, RUM items from the DIRUM database are cur-
rently being reviewed by Thorn et al. [62] with the aim of
developing a standardized RUM instrument for the UK. This
instrument will focus on the health and social care sector and
will exclude ICBs. Therefore, the first step in developing a
new ICB instrument would include a literature search to
identify the main cost-driving elements from relevant eco-
nomic evaluations in a specific disease area [21]. This was
selectively performed in prior research [11,70]; however,
given that the inclusion of ICBs does not seem to have a long
tradition [71], these empirical studies might be missing such
cost elements for this very reason [72]. Thus, collating ICBs
that were mentioned in studies but not necessarily measured
appears more reasonable [2]. The second step in the devel-
opment of a new instrument [21] includes the identification
of existing instruments, which was also one of the main aims
of this review. Following this step, focus groups with
healthcare professionals, experts in the education/criminal
justice sectors and patients to discuss these instruments
should be organized to develop a first version of a new
instrument. Many instruments identified in this review were
specifically developed for use in one study only, which
potentially implies considerable duplication of work hap-
pening across studies. Therefore, bundling individual efforts
could be a more cost-effective strategy overall. Setting-up an
international task force to support these activities by, for
example, reflecting on and exploring different sources of, for
instance, electronic data as a means of validation, and also
developing an internationally adaptable, validated instru-
ment, could be the next step. Such an initiative should take
into account the state of the art of RUM classifications by
detailing the key components of an RUM [64,71]. Following
a structured taxonomy such as the one developed by Ridyard
et al. [64], by providing a description of the data source, who
completes the instrument, the administration mode, and the
methods and medium of recording, clarity and method-
ological transparency could be established in the develop-
ment of such a new instrument.
Data Availability Statement The authors declare that
the data supporting the findings of this study are available
within the article and its supplementary electronic infor-
mation files. Full instruments not directly retrieved from
the literature but following email communication with the
instrument developer may be directly requested from them
based on the information provided (Table 1, ‘Reference
and instrument identification’). Relevant instruments will
be included in DIRUM (
Acknowledgements Many thanks to Gregor Franssen (Information
Scientist at Maastricht University) for expert advice on setting up the
search strategy, and Marianne Tilly for her support with study full-
text retrieval. The authors also gratefully acknowledge the help of
several members of the Department of Health Services Research at
Maastricht University with language translations. Feedback from
Joanna Thorn on an earlier version of the manuscript was greatly
appreciated. We also gratefully acknowledge the valuable input from
participants at the 2017 lowlands Health Economists’ Study Group
(lolaHESG) conference in Rotterdam, The Netherlands. This manu-
script was drafted during Susanne Mayer’s research stay at Maastricht
Author Contributions The initial idea for this study was first
brought up by SE and AP. SM developed the study concept, to which
input was provided by AP, JS, RD, DR and SE. All co-authors were
involved in the discussion of all steps of the review. SM conducted
the literature search, with as the second screener. SM analyzed the
data and wrote the manuscript, and all authors read and approved the
final version of this article.
Compliance with Ethical Standards
Conflicts of interest Susanne Mayer, Aggie T.G. Paulus, Agata
Łaszewska, Judit Simon, Ruben M.W.A. Drost, Dirk Ruwaard, Silvia
M.A.A. Evers declare that they have no conflicts of interest relevant
to the content of this article.
ICB Resource-Use Measurement Instruments
Funding No funding from external sources was received for this
Open Access This article is distributed under the terms of the
Creative Commons Attribution-NonCommercial 4.0 International
License (, which per-
mits any noncommercial use, distribution, and reproduction in any
medium, provided you give appropriate credit to the original
author(s) and the source, provide a link to the Creative Commons
license, and indicate if changes were made.
1. Weatherly H, Drummond M, Claxton K, et al. Methods for
assessing the cost-effectiveness of public health interventions:
key challenges and recommendations. Health Policy.
2. Drost RM, Paulus AT, Ruwaard D, Evers SM. Inter-sectoral costs
and benefits of mental health prevention: towards a new classi-
fication scheme. J Ment Health Policy Econ. 2013;16(4):179–86.
3. Krol M, Brouwer W, Rutten F. Productivity costs in economic
evaluations: past, present, future. Pharmacoeconomics.
4. Krol M, Papenburg J, van Exel J. Does including informal care in
economic evaluations matter? A systematic review of inclusion
and impact of informal care in cost-effectiveness studies. Phar-
macoeconomics. 2015;33(2):123–35.
5. Byford S, Raftery J. Perspectives in economic evaluation. BMJ.
6. Drost RM, van der Putten IM, Ruwaard D, Evers SM, Paulus AT.
Conceptualizations of the societal perspective within economic
evaluations: systematic review. Int J Technol Assess Health Care
7. Claxton K, Walker S, Palmer S, Sculpher M. Appropriate per-
spectives for health care decisions. Working Papers no 054cherp,
Centre for Health Economics, University of York; 2010.
8. Knies S, Severens JL, Ament AJ, Evers SM. The transferability
of valuing lost productivity across jurisdictions. Differences
between national pharmacoeconomic guidelines. Value Health.
9. Sanders GD, Neumann PJ, Basu A, et al. Recommendations for
conduct, methodological practices, and reporting of cost-effec-
tiveness analyses: second panel on cost-effectiveness in health
and medicine. JAMA. 2016;316(10):1093–103.
10. Gold MR, Siegel JE, Russell LB, Weinstein MC. Cost-effec-
tiveness in health and medicine. New York: Oxford University
Press; 1996.
11. Drost RM, Paulus AT, Ruwaard D, Evers SM. Valuing inter-
sectoral costs and benefits of interventions in the healthcare
sector: methods for obtaining unit prices. Expert Rev Pharma-
coecon Outcomes Res. 2017;17(1):77–84.
12. Leggett LE, Khadaroo RG, Holroyd-Leduc J, et al. Measuring
resource utilization: a systematic review of validated self-re-
ported questionnaires. Medicine. 2016;95(10):e2759.
13. Evers SM, Hiligsmann M, Adarkwah C. Risk of bias in trial-
based economic evaluations: identification of sources and bias-
reducing strategies. Psychol Health. 2015;30(1):52–71.
14. DIRUM, Database of Instruments for Resource Use Measure-
ment. Accessed 28 Jul 2016.
15. Ridyard CH, Dyfrig A, DIRUM Team. Development of a data-
base of instruments for resource-use measurement: purpose,
feasibility, and design. Value Health. 2012;15(5):650–5.
16. Noben CY, Evers SM, Nijhuis FJ, de Rijk AE. Quality appraisal
of generic self-reported instruments measuring health-related
productivity changes: a systematic review. BMC Public Health.
17. Tang K. Estimating productivity costs in health economic eval-
uations: a review of instruments and psychometric evidence.
Pharmacoeconomics. 2015;33(1):31–48.
18. Tanco K, Park JC, Cerana A, et al. A systematic review of
instruments assessing dimensions of distress among caregivers of
adult and pediatric cancer patients. Palliat Supp Care.
19. Hoefman RJ, van Exel J, Brouwer W. How to include informal
care in economic evaluations. Pharmacoeconomics.
20. van Durme T, Macq J, Jeanmart C, Gobert M. Tools for mea-
suring the impact of informal caregiving of the elderly: a litera-
ture review. Int J Nurs Stud. 2012;49(4):490–504.
21. Thorn JC, Coast J, Cohen D, et al. Resource-use measurement
based on patient recall: issues and challenges for economic
evaluation. Appl Health Econ Health Policy. 2013;11(3):155–61.
22. Glanville J, Paisley S. Identifying economic evaluations for
health technology assessment. Int J Technol Assess Health Care.
23. Alton V, Eckerlund I, Norlund A. Health economic evaluations:
how to find them. Int J Technol Assess Health Care.
24. Akers J, Aguiar-Iba
˜ez R, Baba-Akbari Sari A. CRD’s guidance
for undertaking reviews in health care. York: Centre for Reviews
and Dissemination (CRD); 2009.
25. Streiner DL, Norman GR, Cairney J. Health measurement scales:
a practical guide to their development and use. Oxford: Oxford
University Press; 2014.
26. Higgins JP, Green S. Cochrane Handbook for Systematic
Reviews of Interventions. Version 5.1.0 2011. http://handbook. Accessed 21 June 2016.
27. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group.
Preferred reporting items for systematic reviews and meta-anal-
yses: the PRISMA statement. Ann Intern Med.
28. Bodden DH, Dirksen CD, Bogels SM. Societal burden of clini-
cally anxious youth referred for treatment: a cost-of-illness study.
J Abnorm Child Psychol. 2008;36(4):487–97.
29. Beecham J, Knapp M. Costing psychiatric interventions. In:
Thornicroft G, editor. Measuring mental health needs. London:
Gaskell; 2001. p. 200–24.
30. Watson JM, Crosby H, Dale VM, et al. AESOPS: a randomised
controlled trial of the clinical effectiveness and cost-effectiveness
of opportunistic screening and stepped care interventions for
older hazardous alcohol users in primary care. Health Technol
Assess. 2013;17(25):1–158.
31. Client Service Receipt Inventory Children’s Version. http://www. Accessed 1 Aug 2017.
32. CSRI-CD, Client Service Receipt Inventory for Children with
Diabetes. Acces-
sed 1 Aug 2017.
33. CSRI-Pain, Client Service Receipt Inventory for Adolescents
with Chronic Pain—Parent report.
csri/chronic-pain-csri/. Accessed 1 Aug 2017.
34. Powell CV, Kolamunnage-Dona R, Lowe J, et al. MAGNEsium
Trial In Children (MAGNETIC): a randomised, placebo-con-
trolled trial and economic evaluation of nebulised magnesium
sulphate in acute severe asthma in children. Health Technol
Assess. 2013;17(45):1–216.
35. Lock C, Wilson J, Steen N, et al. North of England and Scotland
Study of Tonsillectomy and Adeno-tonsillectomy in Children
(NESSTAC): a pragmatic randomised controlled trial with a
S. Mayer et al.
parallel non-randomised preference study. Health Technol
Assess. 2010;14(13):1–164, iii–iv.
36. Parrott S, Godfrey C, Heather N, Clark J, Ryan T. Cost and
outcome analysis of two alcohol detoxification services. Alcohol
Alcohol. 2006;41(1):84–91.
37. DIRUM: SCIPI RUM: instrument qualities 2016. http:// Accessed 1 Aug 2017.
38. SHIFT, Self-Harm Intervention, Family Therapy (SHIFT), Parent
or Carers Questionnaire Booklet.
Carers%20Questionnaire%20Booklet.pdf. Accessed 1 Aug
39. SHIFT, Self-Harm Intervention, Family Therapy (SHIFT),
Young Person Questionnaire Booklet.
Person%20Questionnaire%20Booklet.pdf. Accessed 1 Aug
40. Barrett B, Byford S. Collecting service use data for economic
evaluation in DSPD populations: development of the Secure
Facilities Service Use Schedule. Br J Psychiatry Suppl.
41. SYCAMORE CSRI Questionnaire.
naire.pdf. Accessed 1 Aug 2017.
42. Chisholm D, Knapp MR, Knudsen HC, et al. Client Socio-De-
mographic and Service Receipt Inventory-European Version:
development of an instrument for international research. EPSI-
LON Study 5. European Psychiatric Services: Inputs Linked to
Outcome Domains and Needs. Br J Psychiatry Suppl.
43. Burns BJ, Angold A, Magruder-Habib K, et al. The child and
adolescent services assessment (CASA) Child Interview Version
5.0. Accessed 1 Aug 2017.
44. Sommers BD, Fargo JD, Lyons MS, Shope JT, Sommers MS.
Societal costs of risky driving: an economic analysis of high-risk
patients visiting an urban emergency department. Traffic Inj Prev.
45. Wetterneck CT, Woods DW, Norberg MM, Begotka AM. The
social and economic impact of trichotillomania: results from two
nonreferred samples. Behav Intervent. 2006;21(2):97–109.
46. Zhou ZY, Koerper MA, Johnson KA, et al. Burden of illness:
direct and indirect costs among persons with hemophilia A in the
United States. J Med Econ. 2015;18(6):457–65.
47. WPAI ?CIQ:AS. Work Productivity and Activity Impairment
Questionnaire plus Classroom Impairment Questions: Allergy
Specific, Version 2.
html. Accessed 1 Aug 2017.
48. Connelly LB, Webber J, Rowell D. Cost of Trauma Instrument
(COTI), 2005.
tools/cost-of-trauma-instrument-coti. Accessed 1 Aug 2017.
49. Genereaux D, van Karnebeek CD, Birch PH. Costs of caring for
children with an intellectual developmental disorder. Disabil
Health J. 2015;8(4):646–51.
50. Browne G, Gafni A, Roberts J, Goldsmith A, Jamieson E.
Approach to the measurement of costs (expenditures) when
evaluating health and social programmes. Hamilton: McMaster
University; 1995.
51. Huy R, Wichmann O, Beatty M, et al. Cost of dengue and other
febrile illnesses to households in rural Cambodia: a prospective
community-based case-control study. BMC Public Health.
52. KNCV Tuberculosis Foundation. The Tool to Estimate Patients’
Costs, 2008.
supplementary/1471-2458-11-43-S1.PDF. Accessed 1 Aug 2017.
53. Aygoren-Pursun E, Bygum A, Beusterien K, et al. Socioeco-
nomic burden of hereditary angioedema: results from the
hereditary angioedema burden of illness study in Europe.
Orphanet J Rare Dis. 2014;9:99.
54. Blacken PH, Pozzi G, Tempesta E, et al. European Addiction
Severity Index (EuropASI) Cost A6: A guide to training and
administering EuropASI interviews 1994. http://www.emcdda. Accessed 1 Aug 2017.
55. SCIPI RUM, SubCutaneous Insulin: Pumps or Injections (SCIPI)
RUM, 2011.
634968699228637202-SCIPI_Questionnaire.pdf. Accessed 1
Aug 2017.
56. Diary SC. Syncamore Treatment Diary.
Diary.pdf. Accessed 1 Aug 2017.
57. Farmer EM, Angold A, Burns BJ, Costello EJ. Reliability of self-
reported service use: test-retest consistency of children’s
responses to the Child and Adolescent Services Assessment
(CASA). J Child Fam Stud. 1994;3(3):307–25.
58. Ascher BH, Farmer EM, Burns BJ, Angold A. The child and
adolescent services assessment (CASA) description and psycho-
metrics. J Emot Behav Disord. 1996;4(1):12–20.
59. Reilly MC, Zbrozek AS, Dukes EM. The validity and repro-
ducibility of a work productivity and activity impairment
instrument. Pharmacoeconomics. 1993;4(5):353–65.
60. Reilly MC, Tanner A, Meltzer EO. Work, classroom and activity
impairment instruments: validation studies in allergic rhinitis.
Clin Drug Investig. 1996;11(5):278–88.
61. Ten Jo
¨nsson B. arguments for a societal perspective in the eco-
nomic evaluation of medical innovations. Eur J Health Econ.
62. Thorn JC, Ridyard CH, Riley R, et al. Identification of items for a
standardised resource-use measure: review of current instru-
ments. Value Health. 2015;18(7):A688.
63. Ridyard CH, Hughes DA. Methods for the collection of resource
use data within clinical trials: a systematic review of studies
funded by the UK Health Technology Assessment program.
Value Health. 2010;13(8):867–72.
64. Ridyard CH, Hughes DA, DIRUM Team. Taxonomy for methods
of resource use measurement. Health Econ. 2015;24(3):372–8.
65. Johnston K, Buxton MJ, Jones DR, Fitzpatrick R. Assessing the
costs of healthcare technologies in clinical trials. Health Technol
Assess. 1999;3(6):1–76.
66. Schellingerhout JM, Verhagen AP, Heymans MW, et al. Mea-
surement properties of disease-specific questionnaires in patients
with neck pain: a systematic review. Qual Life Res.
67. Ladd HF, Edward EB. Handbook of research in education finance
and policy. Abingdon: Routledge; 2012.
68. Terwee CB, Mokkink LB, Knol DL, et al. Rating the method-
ological quality in systematic reviews of studies on measurement
properties: a scoring system for the COSMIN checklist. Qual Life
Res. 2012;21(4):651–7.
69. van Ballegooijen W, Riper H, Cuijpers P, van Oppen P, Smit JH.
Validation of online psychometric instruments for common
mental health disorders: a systematic review. BMC Psychiatry.
70. Goodrich K, Kaambwa B, Al-Janabi H. The inclusion of informal
care in applied economic evaluation: a review. Value Health.
71. Ridyard CH, Hughes DA. Review of resource-use measures in
UK economic evaluations. In: Curtis L, Burns A, editors. Unit
Costs of Health and Social Care 2015. Canterbury: Personal
Social Services Research Unit, University of Kent; 2015.
p. 22–30.
72. Mayer S, Kiss N, Laszewska A, Simon J. Health economic
costing methods and reporting in Austria. Value Health.
ICB Resource-Use Measurement Instruments
73. Patel A, Rendu A, Moran P, et al. A comparison of two methods
of collecting economic data in primary care. Fam Pract.
74. Byford S, Leese M, Knapp M, et al. Comparison of alternative
methods of collection of service use data for the economic
evaluation of health care interventions. Health Econ.
75. Mirandola M, Bisoffi G, Bonizzato P, Amaddeo F. Collecting
psychiatric resources utilisation data to calculate costs of care: a
comparison between a service receipt interview and a case reg-
ister. Soc Psychiatry Psychiatr Epidemiol. 1999;34(10):541–7.
76. Heinrich S, Deister A, Birker T, et al. Accuracy of self-reports of
mental health care utilization and calculated costs compared to
hospital records. Psychiatry Res. 2011;185(1–2):261–8.
77. Kilian R, Roick C, Bernert S, et al. Instruments for the eco-
nomical evaluation of psychiatric service systems: methodologi-
cal foundations of the european standardisation and the German
adaptation [in German]. Psychiatr Prax. 2001;28(Suppl 2):74–8.
78. Browne GB, Arpin K, Corey P, Fitch M, Gafni A. Individual
correlates of health service utilization and the cost of poor
adjustment to chronic illness. Med Care. 1990;28(1):43–58.
79. Hendriks VM, Kaplan CD, van Limbeek J, Geerlings P. The
Addiction Severity Index: reliability and validity in a Dutch
addict population. J Subst Abuse Treat. 1989;6(2):133–41.
80. Dijkgraaf MG, van der Zanden BP, de Borgie CA, et al. Cost
utility analysis of co-prescribed heroin compared with methadone
maintenance treatment in heroin addicts in two randomised trials.
BMJ. 2005;330(7503):1297–300.
81. Knapp M, Scott S, Davies J. The cost of antisocial behaviour in
younger children. Clin Child Psychol Psychiatry. 1999;4(4):
82. Sleed M, Eccleston C, Beecham J, Knapp M, Jordan A. The
economic impact of chronic pain in adolescence: methodological
considerations and a preliminary cost-of-illness study. Pain.
83. Mauch V, Woods N, Kirubi B, et al. Assessing access barriers to
tuberculosis care with the tool to Estimate Patients’ Costs: pilot
results from two districts in Kenya. BMC Public Health.
84. UKATT Research Team. Cost effectiveness of treatment for
alcohol problems: findings of the randomised UK alcohol treat-
ment trial (UKATT). BMJ. 2005;331(7516):544.
85. STEPWICE Research Group. The effectiveness and cost-effec-
tiveness of screening and stepped care interventions for alcohol
use disorders in the primary care setting. Cardiff: Welsh Office of
Research & Development; 2003.
86. Coulton S, Watson J, Bland M, et al. The effectiveness and cost-
effectiveness of opportunistic screening and stepped care inter-
ventions for older hazardous alcohol users in primary care
(AESOPS)—a randomised control trial protocol. BMC Health
Serv Res. 2008;8:129.
87. Stade B, Ali A, Bennett D, et al. The burden of prenatal exposure
to alcohol: revised measurement of cost. Can J Clin Pharmacol.
88. Ramanan AV, Dick AD, Benton D, et al. A randomised con-
trolled trial of the clinical effectiveness, SafetY and cost effec-
tiveness of adalimumab in combination with MethOtRExate for
the treatment of juvenile idiopathic arthritis associated uveitis
(SYCAMORE). Trials. 2014;9:14.
89. Bowling A. Mode of questionnaire administration can have
serious effects on data quality. J Public Health.
S. Mayer et al.

Supplementary resource (1)

... A workgroup consisting of seven health economists (LJ, SE, AP, CD, WH, SN, JT) brainstormed to identify existing RUM aspects in health economics, i.e. all concepts that are part of the complex phenomenon of RUM. Afterwards, the four methodological studies [3,4,20,21] on RUM, familiar with the workgroup, were read full-text by one researcher (LJ) to validate the identified RUM aspects and to complement the list with other relevant RUM aspects. The RUM aspects identified during the expert meeting and the RUM aspects identified in the literature provided the input for the draft framework, developed in the next phase. ...
... First, a structured literature search was conducted. The search scopes of four existing systematic reviews addressing RUM that were identified by the experts in Phase 1 were used as a basis for the current literature search [3,4,20,21]. Databases were chosen based on the scope and topic coverage and, as RUM is a relatively new and underexplored area of research, multiple databases were chosen. ...
... The face-to-face expert meeting and four systematic reviews addressing RUM [3,4,20,21] provided the input for the draft framework. Clustering of the RUM aspects resulted in a framework with six main methodological RUM domains and corresponding aspects (ESM 3): (1) What to measure? ...
Full-text available
Background While the methods for conducting health economics research in general are improving, current guidelines provide limited guidance regarding resource use measurement (RUM). Consequently, a variety of methods exists, yet there is no overview of aspects to consider when deciding on the most appropriate RUM methodology. Therefore, this study aims to (1) identify and categorize existing knowledge regarding aspects of RUM, and (2) develop a framework that provides a comprehensive overview of methodological aspects regarding RUM.Methods Relevant articles were identified by enrolling a search string in six databases and handsearching the DIRUM database. Included articles were descriptively reviewed and served as input for a comprehensive framework. Health economics experts were involved during the process to establish the framework’s face validity.ResultsForty articles were included in the scoping review. The RUM framework consists of four methodological RUM domains: ‘Whom to measure’, addressing whom to ask and whom to measure; ‘How to measure’, addressing the different approaches of measurement; ‘How often to measure’, addressing recall period and measurement patterns; and ‘Additional considerations’, which covers additional aspects that are essential for further refining the methodologies for measurement. Evidence retrieved from the scoping review was categorized according to these domains.Conclusion This study clustered the aspects of RUM methodology in health economics into a comprehensive framework. The results may guide health economists in their decision making regarding the selection of appropriate RUM methods and developing instruments for RUM. Furthermore, policy makers may use these findings to review study results from an evidence-based perspective.
... Existing instruments were used as illustrative examples, but no parts of questionnaires or questions were used verbatim, with the exception of the module on employment and productivity, which was fully based on an existing RUM instrument (iPCQ). Furthermore, many existing RUM instruments were developed for use in specific contexts [52], while the PECUNIA RUM was meant to be a generic, internationally applicable instrument. Third, two additional steps in the planning phase of the Thorn et al. method [2] were needed to account for the complexity of developing a multi-sectoral RUM instrument, i.e. the scoping review (step 3) and establishing a uniform methodological basis (step 4), as well as one additional step in the development phase, i.e. harmonization of instrument modules (step 6). ...
... Although the PECUNIA RUM will initially be tested among former mental health service users and carers, the cost categories included in the instrument are also relevant for other conditions (e.g. chronic diseases) [52]. Third, instruments used for data collection in research are developed by researchers with high educational level and specialized knowledge, while the instruments are usually completed by patients and caregivers with varying levels of education and (health) literacy. ...
Full-text available
Background: Measuring objective resource-use quantities is important for generating valid cost estimates in economic evaluations. In the absence of acknowledged guidelines, measurement methods are often chosen based on practicality rather than methodological evidence. Furthermore, few resource-use measurement (RUM) instruments focus on the measurement of resource use in multiple societal sectors and their development process is rarely described. Thorn and colleagues proposed a stepwise approach to the development of RUM instruments, which has been used for developing cost questionnaires for specific trials. However, it remains unclear how this approach can be translated into practice and whether it is applicable to the development of generic self-reported RUM instruments and instruments measuring resource use in multiple sectors. This study provides a detailed description of the practical application of this stepwise approach to the development of a multi-sectoral RUM instrument developed within the ProgrammE in Costing, resource use measurement and outcome valuation for Use in multi-sectoral National and International health economic evaluAtions (PECUNIA) project. Methods: For the development of the PECUNIA RUM, the methodological approach was based on best practice guidelines. The process included six steps, including the definition of the instrument attributes, identification of cost-driving elements in each sector, review of methodological literature and development of a harmonized cross-sectorial approach, development of questionnaire modules and their subsequent harmonization. Results: The selected development approach was, overall, applicable to the development of the PECUNIA RUM. However, due to the complexity of the development of a multi-sectoral RUM instrument, additional steps such as establishing a uniform methodological basis, harmonization of questionnaire modules and involvement of a broader range of stakeholders (healthcare professionals, sector-specific experts, health economists) were needed. Conclusion: This is the first study that transparently describes the development process of a generic multi-sectoral RUM instrument in health economics and provides insights into the methodological aspects and overall validity of its development process.
... Examples for intersectoral costs and benefits (ICBs) resulting from healthcare interventions that affect the education sector could include special education services and the costs for student transport to the education facility [56,57]. Criminal justice inter-sectoral costs compromise costs for court proceedings, police services, or forensic services [56,58]. ...
... Examples for intersectoral costs and benefits (ICBs) resulting from healthcare interventions that affect the education sector could include special education services and the costs for student transport to the education facility [56,57]. Criminal justice inter-sectoral costs compromise costs for court proceedings, police services, or forensic services [56,58]. The methodology for the valuation of health-related service use in the education and (criminal) justice sectors is less established. ...
Full-text available
Background Valuation is a critical part of the costing process in health economic evaluations. However, an overview of specific issues relevant to the European context on harmonizing methodological requirements for the valuation of costs to be used in health economic evaluation is lacking. We aimed to inform the development of an international, harmonized and multi-sectoral costing framework, as sought in the European PECUNIA (ProgrammE in Costing, resource use measurement and outcome valuation for Use in multi-sectoral National and International health economic evaluAtions) project. Methods We conducted a scoping review (information extraction 2008–2021) to a) to demonstrate the degree of heterogeneity that currently exists in the literature regarding central terminology, b) to generate an overview of the most relevant areas for harmonization in multi-sectoral and multi-national costing processes for health economic evaluations, and c) to provide insights into country level variation regarding economic evaluation guidance. A complex search strategy was applied covering key publications on costing methods, glossaries, and international costing recommendations augmented by a targeted author and reference search as well as snowballing. Six European countries served as case studies to describe country-specific harmonization issues. Identified information was qualitatively synthesized and cross-checked using a newly developed, pilot-tested data extraction form. Results Costing methods for services were found to be heterogeneous between sectors and country guidelines and may, in practice, be often driven by data availability and reimbursement systems in place. The lack of detailed guidance regarding specific costing methods, recommended data sources, double-counting of costs between sectors, adjustment of unit costs for inflation, transparent handling of overhead costs as well as the unavailability of standardized unit costing estimates in most countries were identified as main drivers of country specific differences in costing methods with a major impact on valuation and cost-effectiveness evidence. Conclusion This review provides a basic summary of existing costing practices for evaluative purposes across sectors and countries and highlights several common methodological factors influencing divergence in cost valuation methods that would need to be systematically incorporated and addressed in future costing practices to achieve more comparable, harmonized health economic evaluation evidence.
... Because inter-sectoral costs and consequences were excluded in economic studies of DPHP for a long time [5,19,28,29], several researchers highlighted the need to consider these costs [5,14,19,32,41]. However, the tools developed were heterogeneous and showed limited evidence on validity and reliability [42]. ...
... Although validated and well-accepted tools for the inclusion of these inter-sectoral costs are lacking, the adoption and consistent application of a societal perspective would stimulate efforts to include costs and effects beyond the health sector [42]. ...
Full-text available
Background We aimed to provide a comprehensive overview of methodological challenges in economic evaluations of disease prevention and health promotion (DPHP)-measures. Methods We conducted an overview of reviews searching MEDLINE, EMBASE, NHS Economic Evaluation Database, Database of Promoting Health Effectiveness Reviews, Cochrane Database of Systematic Reviews (CDSR) and Database of Promoting Health Effectiveness Reviews (DOPHER) (from their inception to October 2021). We included both systematic and scoping reviews of economic evaluations in DPHP addressing following methodological aspects: (i) attribution of effects, (ii) outcomes, (iii) inter-sectoral (accruing to non-health sectors of society) costs and consequences and (iv) equity. Data were extracted according to the associated sub-criteria of the four methodological aspects including study design economic evaluation (e.g. model-based), type/scope of the outcomes (e.g. outcomes beyond health), perspective, cost categories related to non-health sectors of society, and consideration of equity (method of inclusion). Two reviewers independently screened all citations, full-text articles, and extracted data. A narrative synthesis without a meta-analysis or other statistical synthesis methods was conducted. Results The reviewing process resulted in ten systematic and one scoping review summarizing 494 health economic evaluations. A lifelong time horizon was adopted in about 23% of DPHP evaluations, while 64% of trial-based evaluations had a time horizon up to 2 years. Preference-based outcomes (36%) and non-health outcomes (8%) were only applied in a minority of studies. Although the inclusion of inter-sectoral costs (i.e. costs accruing to non-health sectors of society) has increased in recent years, these were often neglected (between 6 and 23% depending on the cost category). Consideration to equity was barely given in economic evaluations, and only addressed in six of the eleven reviews. Conclusions Economic evaluations of DPHP measures give only little attention to the specific methodological challenges related to this area. For future economic DPHP evaluations a tool with structured guidance should be developed. This overview of reviews was not registered and a published protocol does not exist.
... Despite recent methodological developments in identifying, measuring, and valuing education costs [12][13][14][15][16], their inclusion in costing research remains limited [17]. This could be attributed to several reasons. ...
... It is important to note that the costs associated with reduced school attainment become more apparent in the longer term, which makes it more challenging to capture these costs in studies with shorter time horizons of 1 or 2 years. In addition, the most frequently included education cost items, "special education," "absenteeism from school," and "additional educational support (e.g., tutoring, homework help)," correspond with the findings of the recent review of health-related resource-use measurement instruments by Mayer et al. [13]. However, less tangible resource-use items such as "school functioning" and "social functioning" were also not included in the reviewed studies. ...
Full-text available
Background Psychosocial (e.g., anxiety or behavior) problems lead to costs not only in the healthcare sector but also in education and other sectors. As psychosocial problems develop during the critical period of establishing educational trajectories, education costs are particularly relevant in the context of psychosocial problems among children and adolescents.Objectives This study aimed to gain insights into the methods used for the inclusion of education costs in health economics studies and into the proportion of the education costs in relation to the total costs associated with a condition or an intervention.Methods We systematically searched the PubMed, Embase, SSCI, CINAHL, PsycINFO, ERIC, and Econlit databases in August 2019 for economic evaluations of mental health, psychosocial and educational interventions, and cost-of-illness studies of mental, behavioral, and neurodevelopmental disorders conducted from a societal perspective in populations of children and adolescents. An additional search was conducted in February 2021 to update the review.ResultsIn total, 49 articles were included in the analysis. The most common cost items were special education, school absenteeism, and various educational professionals (educational psychologist). A variety of methods were employed for the identification, measurement, and/or valuation of education costs. The proportion of education costs to the total costs of condition/intervention ranged from 0 to 67%, with the mean being 18.5%.DiscussionSince education costs can constitute a significant proportion of the total costs of an intervention or condition, including them in health economics studies might be important in informing optimal resource allocation decisions. Although various methods are available for including education costs in health economics studies, further research is needed to develop evidence-based methods for producing comparable estimates.
... Analysts should attempt to quantify in the impact inventory as many relevant costs and outcomes as possible within time and resources available, unless these are likely to have a negligible effect on the result of the analysis (Hill et al., 2017;Sanders et al., 2016). Because criminal justice service resource use is particularly important in the field of alcoholism (Mayer et al., 2017), it would be advisable for future trial-based economic evaluations of alcohol interventions to consider the routine adoption of resource-use measurement instruments to measure impacts falling on criminal justice system. Lists of resource-use measurement instruments in the criminal justice sector have been recently proposed (Drost et al., 2013;Mayer et al., 2017). ...
... Because criminal justice service resource use is particularly important in the field of alcoholism (Mayer et al., 2017), it would be advisable for future trial-based economic evaluations of alcohol interventions to consider the routine adoption of resource-use measurement instruments to measure impacts falling on criminal justice system. Lists of resource-use measurement instruments in the criminal justice sector have been recently proposed (Drost et al., 2013;Mayer et al., 2017). Unit costs estimates for the criminal justice system are available for the United Kingdom (Brand & Price, 2000;Dubourg et al., 2005;Newton, May, Eames, & Ahmed, 2019). ...
Full-text available
Cost-effectiveness analyses of health care programmes often focus on maximising health and ignore non-health impacts. Assessing the cost-effectiveness of public health interventions from a narrow health care perspective would likely underestimate their full impact, and potentially lead to inefficient decisions about funding. The aim of this study is to provide a practical application of a recently proposed framework for the economic evaluation of public health interventions, evaluating an intervention to reduce alcohol misuse in criminal offenders. This cross-sectoral analysis distinguishes benefits and opportunity costs for different sectors, makes explicit the value judgements required to consider alternative perspectives, and can inform heterogeneous decision makers with different objectives in a transparent manner. Three interventions of increasing intensity are compared: client information leaflet; brief advice; brief lifestyle counselling. Health outcomes are measured in quality-adjusted life-years and criminal justice outcomes in re-convictions. Costs considered include intervention costs, costs to the NHS and costs to the criminal justice system. The results are presented for four different perspectives: ‘narrow’ health care perspective; criminal justice system perspective; ‘full’ health care perspective; and joint ‘full’ health and criminal justice perspective. Conclusions and recommendations differ according to the normative judgement on the appropriate perspective for the evaluation.
... Hence, there is a need for harmonized RUMs to improve the overall methodology of both cost-effectiveness and comparative effectiveness studies [11,13] in terms of international comparability. According to Mayer et al. [16], the first step in a RUM development process is the identification of economically relevant items by means of a literature review. ...
Full-text available
Background: Health economic research is still facing significant problems regarding the standardization and international comparability of health care services. As a result, comparative effectiveness studies and cost-effectiveness analyses are often not comparable. This study is part of the PECUNIA project, which aimed to improve the comparability of economic evaluations by developing instruments for the internationally standardized measurement and valuation of health care services for mental disorders. The aim of this study was to identify internationally relevant services in the health and social care sectors relevant for health economic studies for mental disorders. Methods: A systematic literature review on cost-of-illness studies and economic evaluations was conducted to identify relevant services, complemented by an additional grey literature search and a search of resource use measurement (RUM) questionnaires. A preliminary long-list of identified services was explored and reduced to a short-list by multiple consolidation rounds within the international research team and an external international expert survey in six European countries. Results: After duplicate removal, the systematic search yielded 15,218 hits. From these 295 potential services could be identified. The grey literature search led to 368 and the RUM search to 36 additional potential services. The consolidation process resulted in a preliminary list of 186 health and social care services which underwent an external expert survey. A final consolidation step led to a basic list of 56 services grouped into residential care, daycare, outpatient care, information for care, accessibility to care, and self-help and voluntary care. Conclusions: The initial literature searches led to an extensive number of potential service items for health and social care. Many of these items turned out to be procedures, interventions or providing professionals rather than services and were removed from further analysis. The resulting list was used as a basis for typological coding, the development of RUM questionnaires and corresponding unit costs for international mental health economic studies in the PECUNIA project.
... Selfreported data generally include quantitative data used in large population-based studies collected via questionnaires or interviews involving face-to-face or telephone conversations, or via self-assessment forms [4,6]. Strengths and limitations of using both data sources for cost estimation have already been extensively discussed [6][7][8][9]. ...
Full-text available
Background: Data on resourceuse are frequently required for health economic evaluation. Studies on health care utilization in individuals with mental disorders have analyzed both self-reports and administrative data, each of which with strengths and limitations. Source of data may affect the quality of cost analysis and compromise the accuracy of results. We sought to ascertain the degree of agreement between self-reports and statutory health insurance (SHI) fund claims data from patients with mental disorders to aid in the selection of data collection methods. Methods:Claims data from six German SHI and self-reported data were obtained along with a cost-effectiveness analysis performed as a part of a controlled prospective multicenter cohort study conducted in 18 psychiatric hospitals in Germany (PsychCare), including patients with pre-defined common and/or severe psychiatric disorders. Self-reported data were collected using the German adaption of the Client Sociodemographic and Service Receipt Inventory (CSSRI-D) questionnaire with a 6-month recall period. Data linkage was performed using a unique pseudonymized identifier. Healthcare utilization (HCU) was calculated for inpatient and outpatient care, day-care services, home treatment, and pharmaceuticals. Concordance was measured using Cohen’s Kappa and intraclass correlation coefficient. Regression approaches were used to investigate the effect of independent variables on the dichotomous and quantitative agreements. Results: In total 274 participants (mean age 47.8 [SD = 14.2] years; 47.08% women) were included in the analysis. Kappa values were 0.03 for outpatient contacts, 0.25 for medication use, 0.56 for inpatient days and 0.67 for day-care services. There was varied quantitative agreement between data sources, with the poorest agreement for outpatient care (ICC [95% CI] = 0.22 [0.10-0.33]) and the best for psychiatric day-care services (ICC [95% CI] = 0.72 [0.66-0.78]). Marital status and time since first treatment positively affected the chance of agreement on any use of outpatient services. Conclusions: Concordance between administrative records and patient self-reports was fair to moderate for most of the healthcare services analyzed. Health economic studies should consider using linked or at least different data sources to estimate HCU or focus the primary data-based surveys in specific utilization areas, where unbiased information can be expected.
... To conduct economic evaluations from a societal viewpoint, unit costs from various sectors are a prerequisite. However, the DHE Unit Cost Online Database currently lacks estimates for resource use beyond the health and social care sectors for so-called inter-sectoral costs and benefits (ICBs) [48]. For instance, for any study investigating the costs and outcomes of a mental-health intervention on the educational attainment of children or the use of criminal justice services by adults, no unit cost would be readily available in the current version of the Austrian database due to the lack of inclusion in the underlying studies. ...
Full-text available
Evidence-informed healthcare decision-making relies on high quality data inputs, including robust unit costs, which in many countries are not readily available. The objective of the Department of Health Economics’ Unit Cost Online Database, developed based on systematic reviews of Austrian costing studies, is to make conducting economic evaluations from healthcare and societal perspectives more feasible with publicly available unit cost information in Austria. This article aims to describe trends in unit cost data sources and reporting using this comprehensive database as a case study to encourage relevant national and international methodological discussions. Database analysis and synthesis included publication/study characteristics and costing reporting details in line with the Consolidated Health Economic Evaluation Reporting Standards (CHEERS 2022) with the year of the database launch as the cut-off point to assess how the methods have developed over time. Forty-two full economic evaluations and 278 unit costs were analyzed (2004–2016: 34 studies/232 unit costs, 2017–2022: 8 studies/46 unit costs). Although the reporting quality of costing details including the study perspective, unit cost sources and years has improved since 2017, the unit cost estimates and sources remained heterogeneous in Austria. While methodologically standardized national-level unit costs would be the gold standard, a systematically collated list of unit costs is a first step towards supporting health economic evaluations nationally.
This study aims to develop multi-media based products on thematic learning sub themes "cleanliness at home". This research was applied in class II of elementary school. Multimedia products produced are short animations with the theme of hand washing. This research method uses Research and Development (R & D) with the Borg and Goal model. Three stages of development research are: first, preliminary research into the two development models and third the model dissemination. The results of the preliminary study concluded that multi-media based applications are needed for thematic learning in elementary schools. The development of multi-media animation by giving a message of six steps of hand washing was developed through the stages of product development. The first stage of expert judgment is carried out by media, material and language experts. The second stage is one to one and small group. This study produced a valid multi-media assisted thematic learning model applied to elementary school students.
Full-text available
Objectives: The aim of this study was to investigate how the societal perspective is conceptualized in economic evaluations and to assess how intersectoral costs and benefits (ICBs), that is, the costs and benefits pertaining to sectors outside the healthcare sector, impact their results. Methods: Based on a search in July 2015 using PubMed, Embase, CINAHL, and PsychINFO, a systematic literature review was performed for economic evaluations which were conducted from a societal perspective. Conceptualizations were assessed in NVivo version 11 using conventional and directed content analysis. Trial-based evaluations in the fields of musculoskeletal and mental disorders were analyzed further, focusing on the way ICBs impact the results of economic evaluations. Results: A total of 107 studies were assessed, of which 74 (69.1 percent) provided conceptualizations of the societal perspective. These varied in types of costs included and in descriptions of cost bearers. Labor productivity costs were included in seventy-two studies (67.3 percent), while only thirty-eight studies (35.5 percent) included other ICBs, most of which entailed informal care and/or social care costs. ICBs within the educational and criminal justice sectors were each included five times. Most of the trial-based evaluations analyzed further ( n = 21 of 28) reported productivity costs. In nine, these took up more than 50 percent of total costs. In several studies, criminal justice and informal care costs were also important. Conclusions: There is great variety in the way the societal perspective is conceptualized and interpreted within economic evaluations. Use of the term “societal perspective” is often related to including merely productivity costs, while other ICBs could be relevant as well.
Full-text available
A variety of methods may be used to obtain costing data. Although administrative data are most commonly used, the data available in these datasets are often limited. An alternative method of obtaining costing is through self-reported questionnaires. Currently, there are no systematic reviews that summarize self-reported resource utilization instruments from the published literature. The aim of the study was to identify validated self-report healthcare resource use instruments and to map their attributes. A systematic review was conducted. The search identified articles using terms like “healthcare utilization” and “questionnaire.” All abstracts and full texts were considered in duplicate. For inclusion, studies had to assess the validity of a self-reported resource use questionnaire, to report original data, include adult populations, and the questionnaire had to be publically available. Data such as type of resource utilization assessed by each questionnaire, and validation findings were extracted from each study. In all, 2343 unique citations were retrieved; 2297 were excluded during abstract review. Forty-six studies were reviewed in full text, and 15 studies were included in this systematic review. Six assessed resource utilization of patients with chronic conditions; 5 assessed mental health service utilization; 3 assessed resource utilization by a general population; and 1 assessed utilization in older populations. The most frequently measured resources included visits to general practitioners and inpatient stays; nonmedical resources were least frequently measured. Self-reported questionnaires on resource utilization had good agreement with administrative data, although, visits to general practitioners, outpatient days, and nurse visits had poorer agreement. Self-reported questionnaires are a valid method of collecting data on healthcare resource utilization.
Full-text available
Background: Online questionnaires for measuring common mental health disorders such as depression and anxiety disorders are increasingly used. The psychometrics of several pen-and-paper questionnaires have been re-examined for online use and new online instruments have been developed and tested for validity as well. This study aims to review and synthesise the literature on this subject and provide a framework for future research. Methods: We searched Medline and PsycINFO for psychometric studies on online instruments for common mental health disorders and extracted the psychometric data. Studies were coded and assessed for quality by independent raters. Results: We included 56 studies on 62 online instruments. For common instruments such as the CES-D, MADRS-S and HADS there is mounting evidence for adequate psychometric properties. Further results are scattered over different instruments and different psychometric characteristics. Few studies included patient populations. Conclusions: We found at least one online measure for each of the included mental health disorders and symptoms. A small number of online questionnaires have been studied thoroughly. This study provides an overview of online instruments to refer to when choosing an instrument for assessing common mental health disorders online, and can structure future psychometric research. Keywords: Internet, Depression, Anxiety, Measurement, Psychometrics, Systematic review