Available via license: CC BY 4.0
Content may be subject to copyright.
Page 1/30
Self-reported versus administrative data records:
implications for assessing healthcare resource
utilization of mental disorders
Tarcyane Barata Garcia ( tarcyane.garcia@wig2.de )
WIG2 Institute for Health Economics and Health System Research
Roman Kliemt
WIG2 Institute for Health Economics and Health System Research
Franziska Claus
WIG2 Institute for Health Economics and Health System Research
Anne Neumann
Center of Evidence-based Health Care, Medizinische Fakultät Carl Gustav Carus, Technische Universität
Dresden, Germany
Bettina Soltmann
Department of Psychiatry and Psychotherapy, Universitätsklinikum und Medizinische Fakultät Carl
Gustav Carus, Technische Universität Dresden, Dresden, Germany, Germany
Fabian Baum
Center of Evidence-based Health Care, Medizinische Fakultät Carl Gustav Carus, Technische Universität
Dresden, Germany
Julian Schwarz
University Clinic for Psychiatry and Psychotherapy, Immanuel Hospital Rüdersdorf, Brandenburg
Medical School, Rüdersdorf, Germany
Enno Swart
Institute of Social Medicine and Health Systems Research, Medical Faculty, Otto-von-Guericke-
University Magdeburg
Jochen Schmitt
Center of Evidence-based Health Care, Medizinische Fakultät Carl Gustav Carus, Technische Universität
Dresden, Germany
Andrea Pfennig
Department of Psychiatry and Psychotherapy, Universitätsklinikum und Medizinische Fakultät Carl
Gustav Carus, Technische Universität Dresden, Dresden, Germany, Germany
Dennis Häckl
Institute of Public Finance and Public Management, Faculty of Economics and Management Science,
Leipzig University
Ines Weinhold
Page 2/30
WIG2 Institute for Health Economics and Health System Research
Research Article
Keywords: Mental health, self-report, data linkage, administrative data, validity
Posted Date: March 23rd, 2023
DOI: https://doi.org/10.21203/rs.3.rs-2634643/v1
License: This work is licensed under a Creative Commons Attribution 4.0 International License.
Read Full License
Page 3/30
Abstract
Background: Data on resourceuse are frequently required for health economic evaluation. Studies on
health care utilization in individuals with mental disorders have analyzed both self-reports and
administrative data, each of which with strengths and limitations. Source of data may affect the quality
of cost analysis and compromise the accuracy of results. We sought to ascertain the degree of
agreement between self-reports and statutory health insurance (SHI) fund claims data from patients with
mental disorders to aid in the selection of data collection methods.
Methods:Claims data from six German SHI and self-reported data were obtained along with a cost-
effectiveness analysis performed as a part of a controlled prospective multicenter cohort study
conducted in 18 psychiatric hospitals in Germany (PsychCare), including patients with pre-dened
common and/or severe psychiatric disorders. Self-reported data were collected using the German
adaption of the Client Sociodemographic and Service Receipt Inventory (CSSRI-D) questionnaire with a 6-
month recall period. Data linkage was performed using a unique pseudonymized identier. Healthcare
utilization (HCU) was calculated for inpatient and outpatient care, day-care services, home treatment, and
pharmaceuticals. Concordance was measured using Cohen’s Kappa and intraclass correlation coecient.
Regression approaches were used to investigate the effect of independent variables on the dichotomous
and quantitative agreements.
Results: In total 274 participants (mean age 47.8 [SD = 14.2] years; 47.08% women) were included in the
analysis. Kappa values were 0.03 for outpatient contacts, 0.25 for medication use, 0.56 for inpatient days
and 0.67 for day-care services. There was varied quantitative agreement between data sources, with the
poorest agreement for outpatient care (ICC [95% CI] = 0.22 [0.10-0.33]) and the best for psychiatric day-
care services (ICC [95% CI] = 0.72 [0.66-0.78]). Marital status and time since rst treatment positively
affected the chance of agreement on any use of outpatient services.
Conclusions: Concordance between administrative records and patient self-reports was fair to moderate
for most of the healthcare services analyzed. Health economic studies should consider using linked or at
least different data sources to estimate HCU or focus the primary data-based surveys in specic
utilization areas, where unbiased information can be expected.
1. Introduction
Mental disorders are highly prevalent and have profound economic consequences. In 2019 almost one
billion people worldwide were affected by mental disorders, making up approximately 5% of disability-
adjusted life years (DALYs) and 14% of years lived with disability (YLD) globally [1]. The costs of mental
health conditions around the world were estimated to be approximately US$ 2.5 trillion in 2010 and are
expected to increase to US$ 6.0 trillion by 2030 [2]. In Germany in 2020, 118million days of incapacity to
work due to mental disorders led to a loss in production costs of EUR 14.6billion according to the
calculations of the Federal Institute for Occupational Safety and Health (BAuA) [3].
Page 4/30
Over the last decade, mental health economic studies in Germany have been based on administrative
data and self-reports [4, 5]. Administrative data are recorded within the healthcare system for reasons
other than research purposes, such as billing and reimbursement. Some examples include routinely
collected data from hospitals or insurance companies, pharmacy claims and national registries. Self-
reported data generally include quantitative data used in large population-based studies collected via
questionnaires or interviews involving face-to-face or telephone conversations, or via self-assessment
forms [4, 6]. Strengths and limitations of using both data sources for cost estimation have already been
extensively discussed [6–9].
In economic evaluations conducted from the societal perspective, self-reported data on resource
utilization is usually the data source of choice, since requesting information directly from participants
can potentially facilitate access to data on a more comprehensive set of healthcare services and
assistance provided in other sectors such as special education, social care, voluntary work, and justice
systems [7, 10]. A common measurement method for patient-reported resource use in mental health care
are standardized questionnaires [11, 12], such as the Client Service Receipt Inventory (CSRI) [13]. CSRI
was originally developed in the United Kingdom to collect information on resource use and cost-related
data from patients with mental health conditions. It has been adapted many times for different
languages and has been broadly used throughout many research topics [11].
Previous studies have corroborated various aspects of reliability and validity of CSRI [14, 15] and its
different versions, e.g. the Italian version (ICAP) [16], the European version – the Client Socio-
Demographic and Service Receipt Inventory (CSSRI-EU) [17] – and more recently the Chinese version for
the rare genetic disease population (CSRI-Ra) [18]. Likewise, CSSRI-EU has been translated and validated
in other languages, such as German (CSSRI-D) [19–21] and Portuguese (ISDUCS) [22].
Regarding mental disorders, some of these studies are not representative of a broader psychiatric
population, including only participants with a specic mental disorder [14], those insured by one
individual statutory health insurance (SHI) fund [21], or only those attending hospitals in one particular
district [16, 20], which may limit the generalizability of ndings.
In a rst validation study of CSSRI-D including 330 participants with mental disorders, Heinrich et al.
(2011) [20] observed that the agreement between self-reports and hospital records ranged from poor to
excellent, depending on the type of service investigated. However, that study focused on psychiatric
inpatient and day-care service utilization only and did not include other service categories in the analyses.
In addition, self-reports were collected using interviewer-administrated instruments (telephone interviews),
which may induce social desirability bias when compared to self-administered approaches [10].
To provide an extended validation of the CSSRI-D, Zentner et al. (2012) [21] investigated the differences
and correlations between questionnaire and health insurance claims data regarding costs for inpatient
and outpatient treatments as well as for medication use for adults with mental disorders. Although rank
correlations between data sources were signicant for all cost categories, correlation coecients ranged
Page 5/30
only from low to moderate (0.53 for inpatient/day-care services), probably due to the small sample size
of the administrative records.
The purpose of the current study is to expand upon the literature comparing self-reported and
administrative data on resource utilization. We examined the psychometric properties of the self-
completed CSSRI-D questionnaire and empirically assessed agreement measures for dichotomous and
quantitative data in a mental health population of adult patients. Specically, this study was designed to:
(1) examine levels of agreement for dichotomous (yes/no) resource utilization (hospital stay, contact with
a health professional in the outpatient setting and home treatment, and medication use); (2) investigate
the concordance for volume utilization measures between data sources (i.e., number of days in hospital
and number of contacts with a health professional); (3) evaluate factors associated with agreement
between self-reported and routinely collected administrative data.
2. Methods
2.1 Study Population
The data used in this research were obtained along with a cost-effectiveness analysis performed as a
part of a controlled prospective multicenter cohort study (PsychCare, German Clinical Trials Register No.
DRKS00022535) conducted in 10 model hospitals offering exible and integrative psychiatric treatment
according to §64b German Social Code V (FIT hospitals), and eight control hospitals offering psychiatric
treatment as usual. Financing of FIT hospitals is based on a global treatment budget (GTB) covering
costs for all psychiatric hospital services and is related to the number of patients treated [23, 24]. Patients
with dened mental disorders (i.e., mental and behavioral disorders due to use of alcohol [ICD-10 F10],
schizophrenia, schizotypal disorder, delusional disorders or brief psychotic disorders [ICD-10 F20-23], or
mood affective disorders [ICD-10 F30-39]) who were insured by one of the six German health insurance
funds (SHI) cooperating with the research consortium were included in the study. Exclusion criteria were
severe intellectual disabilities, acute suicidality, and severe organic brain dysfunction including
impairment of cognitive function. Full details of the multicenter study including ethics approval and
consent to participate in the health surveys are available in the study protocol [25]. In total, 1150 patients
who met inclusion criteria were eligible for the analysis. Of these, 274 (23.8%) individuals with valid
informed consent to using claims data for scientic purposes and medically insured by one of the
cooperating SHIs were successfully linked to administrative records and therefore included in the current
study.
2.2. Data Sources
Self-reported healthcare utilization was assessed with a tailored German adaption of CSSRI [17, 19]
comprising distinct categories including sociodemographic data, usual living situation, accommodation
details, employment, income, and use of different health and social care services. To cover all
components of innovative FIT hospital care, the questionnaire was slightly modied by the authors,
including additional services such as home treatment and greater differentiation (i.e., inpatient care, day-
Page 6/30
care, outpatient services, complementary health services, and pharmaceuticals (see supplementary table
1 in Additional File 1)) in contrast to the original version’s broader categories.
In the inpatient sector, the following units were recorded: days in general hospital, days in psychiatric and
psychosomatic hospitals, days in departments of psychotherapy, and days in departments of addiction
and substance misuse. For day-care facilities, resource use was subdivided into psychiatric and non-
psychiatric day-care and recorded via the number of days. In the outpatient sector, visits to
psychiatrists/neurologists, psychologists/psychotherapists (including both outpatient practitioners and
psychiatric outpatient departments (PIA)), general practitioners (GPs) and other medical specialists were
considered. Medication names were coded according to the Anatomical Therapeutic Chemical (ATC)
classication system and grouped in categories according to their therapeutic class (psychotropic drugs
and non-psychotropic drugs). Assessments were retrospective for all resources and services with a recall
period of 6 months, except for consumption of medication where the period was 1 month only.
Primary data collection lasted from March 2018 to September 2019. Prior to the analyses the returned
questionnaires were examined for inconsistent or illogical answers, such as out-of-range responses, and
corrected, if possible, e.g., by calculating lengths of hospital stay from patient-reported information on
admission and discharge dates. Answers that could not be plausibly transferred were dened as missing
values. Insurance claims data were obtained from six German SHI funds, covering states from the whole
federal territory. Structure and content of claims data corresponded to the EVA64 study [26], including
patient related sociodemographic and morbidity data (e.g., age, sex, diagnoses, disability), sick leave, use
of inpatient and outpatient services, day-care, pharmaceutical and non-pharmaceutical treatments. SHI
data covered the period of 2016–2019 to allow for a pre-baseline period of 2 years for each patient.
Again, the data were subjected to quality and plausibility checks; implausible values were deleted and
coded as missing. Multiple claims made to the same healthcare professional within the same day were
counted as a single contact.
Data linkage of self-reported and claims data was performed using a unique pseudonymized individual-
level identication key. Patients’ individual health insurance numbers were collected by an independent
trust center, which sorted the numbers and requested the corresponding data from the participating
health insurance funds. Pseudonymized data were transferred to the research unit and linked with
primary data by study identication number. The procedure is in line with Good Practice in Secondary
Data Analysis and reporting [27, 28] as well as Good Practice Data Linkage [29] and was permitted by the
regulatory authorities of the six SHIs. Additionally, written informed consent for using health insurance
claims data and linking it with primary data was obtained from patients.
2.3 Data Analysis
All analyses were carried out using Microsoft SQL Server 2012 and R Software version 3.2.3 [30]. Our
procedure for calculating lengths of stay and contact frequencies from patient data, taking into account
plausibility criteria (e.g., maximum number of weekdays per year), is documented in Additional File 1. To
measure resource utilization, variables from the two data sources were dichotomized (yes/no) indicating
Page 7/30
any service utilization in the past 6 months (e.g., outpatient visit, stay in hospital or home treatment
contact) or any use of medication in the previous month. Next, the level of concordance was calculated
using the sum of the proportions of absolute agreement (n, %), where both the self-reported and the
administrative data indicated the same result (i.e., both indicated the occurrence of an event in the same
period, or both indicated no event). Beyond chance agreement was demonstrated using Cohen’s kappa
statistic (κ) computed using the
confusionMatrix( )
function from the R caret package [31]. The
magnitude of the frequently used kappa statistic is greatly affected by the prevalence of a condition in
the population and by bias (i.e., the extent to which there is difference in the proportion of positive or
negative cases between data sources), which has been widely criticized by researchers [32]; low kappa
values, thus, do not necessarily reect low proportions of overall agreement. To address this potential
bias, the prevalence-adjusted bias-adjusted kappa (PABAK) was reported [33], which informs the rates of
agreement regardless of an unbalanced proportion of positive or negative cases. Sensitivity (true positive
rate) and specicity (true negative rate) were also calculated for each setting and medication use using
the
diagnostic( )
function from the R ThresholdROC package [34]. For calculation purposes,
administrative records were treated as the reference for resource utilization.
For the volume measure of resource utilization (i.e., length of stay in hospital, number of outpatient visits,
and home treatment contacts), the number of self-reported events were subtracted from the number of
events in claims record to obtain the concordance between the two data sources. When the result was “0”,
“total agreement” was assigned to the patient-reported information. When the result was negative, the
information provided by the participant was considered an overestimation of utilization, because the
number of self-reported events was higher than their corresponding administrative claims. When the
result was positive, an underestimation was assumed. To estimate the quantitative concordance between
self-reported and administrative data, intraclass correlation coecients (ICC) were calculated using the
icc( )
function from the R irr package [35]. The Landis and Koch standards for strength of agreement [36]
were applied to both Cohen’s kappa and ICC (< 0.01 poor, 0.01–0.20 slight, 0.21–0.40 fair, 0.41–0.60
moderate, 0.61–0.80 substantial, and 0.81–1.0 almost perfect). To provide a visual assessment of
differences between self-reported events and those recorded in administrative data Bland-Altman plots
were used. The correlation between the number of utilizations from the two data sources was assessed
using the Spearman correlation coecient (ρ), computed by the
cor.test( )
function from the R stats
package [30].
In addition to the analyses of healthcare utilization categories, which combine several single services, the
divergences between the data sources were investigated for the subcategories within inpatient and
outpatient settings, and subclasses of medications. Additionally, healthcare services were grouped into
two broad subgroups: psychiatric and somatic services. Finally, to analyze the effect of distinct variables
on differences in the concordance of resource use between the two data sources, univariate and
multivariate logistic regression models reporting odds ratios and 95% condence intervals were specied.
Each dependent variable was coded binary (1 indicating agreement; 0 indicating disagreement). Linear
regression models were used to assess factors associated with overreporting or underreporting the
number of events (days/contacts). Here, the dependent variable was the difference between self-reported
Page 8/30
and administrative data. For all models, several characteristics that have been suggested to affect the
accuracy of self-reports or administrative data were evaluated [37], including age, sex, living situation,
education status, the length of stay in hospitals or day-care facilities and the number of outpatient visits.
As performed in previous reports [38, 39], item-level missing values were coded as “no”, and accordingly
zero-imputation was used for the analyses of quantitative agreements. Signicance was set at a value of
p
≤ 0.05.
3. Results
3.1 Baseline Characteristics
Descriptive statistics of participants are presented in Table 1.
Table 1. Sociodemographic and clinical characteristic of participants
Page 9/30
Variables Overall (n=274)
Sex - n (%)
Male 145 (52.92)
Female 129 (47.08)
Age in years - mean (SD) 47.8 (14.2)
Age in years - median [Min-Max] 49 [19-85]
Age groups - n (%)
18 - 59 years 226 (82.48)
60 years+ 48 (17.52)
(a)Index year – n (%)
2018 138 (50.36)
2019 136 (49.64)
Years since rst psychiatric treatment - n (%)
≤ 5 years 90 (32.85)
> 5 years 184 (67.15)
Main diagnosis - n (%)
F10 59 (21.53)
F20-23 50 (18.25)
F30-39 165 (60.22)
Marital status – n (%)
Single 126 (45.99)
Married 64 (23.36)
Married, but living separated 10 (3.65)
Divorced 53 (19.34)
Widowed 12 (4.38)
Missing 9 (3.28)
(b)Education Level - n (%)
Primary 65 (23.72)
Secondary 160 (58.39)
Page 10/30
Tertiary 34 (12.41)
Missing 15 (5.47)
Living situation - n (%)
Independent accommodation 238 (86.86)
Supervised accommodation 16 (5.84)
Homeless 6 (2.19)
Other 6 (2.19)
Missing 8 (2.92)
F10, mental and behavioral disorders due to use of alcohol; F20-F23, Schizophrenia, schizotypal and
delusional disorders; F30-F39, mood affective disorders. (a)Year of baseline assessment; (b)CASMIN
classication: Primary (1a), Secondary (1b, 1c, 2a, 2b, 2c_gen, 2c_voc) and Tertiary (3a, 3b).
The average age of participants was 47.8 years (SD ± 14.2, range = 19–85, median = 49 years),
approximately half of them were male (52.92%), and 67.15% were in psychiatric treatment for more than
5 years. In total 21.53% were identied with mental and behavioral disorders due to use of alcohol,
18.25% suffered from schizophrenia, schizotypal and delusional disorders, and 60.22% were diagnosed
with mood affective disorders. Participants were mostly single (45.99%) and lived predominantly in an
independent accommodation (86.86%).
3.2 Agreement between self-reported and administrative data for dichotomous reporting on healthcare
utilization
Table 2 shows the comparisons of self-report and administrative measures for any use (yes/no) of health
service utilization over 6 months prior to the baseline assessment or any use of medication in the
previous month.
Table 2. Proportion of patients by source and concordance between self-reported and administrative data
for any use of health care services (n=274)
Page 11/30
Source Inpatient Day-care Outpatient Home
Treatment
Medications
(a)Total utilization - n (%) 215
(78.47) 100 (36.5) 268
(97.81) 25 (9.12) 238 (86.86)
Utilization in SR - n (%) 183
(85.12) 85 (85) 176
(65.67) 6 (24.00) 188 (78.99)
Utilization in AR - n (%) 195 (90.7) 77 (77) 261
(97.39) 20 (80.00) 204 (85.71)
Utilization in AR and in SR - n
(%) 163
(75.81) 62 (62) 169
(63.06) 1 (4.00) 154 (64.71)
Utilization in SR only - n (%)
20 (9.3) 23 (23) 7 (2.61) 5 (20.00) 34 (14.29)
Utilization in AR only - n (%) 32 (14.88) 15 (15) 92 (34.33) 19 (76.00) 50 (21.01)
No utilization in SR,
no utilization in AR- n (%)
59 (21.53) 174 (63.5) 6 (2.19) 249
(90.88) 36 (13.14)
(b)Agreement - n (%) 222
(81.02) 236
(86.13) 175
(63.87) 250
(91.24) 190 (69.34)
Kappa [95% CI] 0.56
[0.45-0.66]
0.67
[0.57-
0.76]
0.03
[-0.04-0.1]
0.04
[-0.1-0.19]
0.25
[0.13-0.37]
PABAK 0.62 0.72 0.28 0.82 0.39
Sensitivity - % [95% CI] 83.59
[77.5-88.3]
80.52
[69.6-
88.3]
64.75
[58.6-70.5]
5
[0.3-26.9]
75.49
[68.9-81.1]
Specicity - % [95% CI] 74.68
[63.4-83.5]
88.32
[82.8-
92.3]
46.15
[20.4-73.9]
98.03
[95.2-99.3]
51.43
[39.3-63.4]
(a)Total number of participants using medical services who were identied by self-reported and/or
administrative data; (b)The percentage of agreement indicates concordance between self-reported and
administrative data based on the same result (both indicated an event, or both indicated no event).
Abbreviations: AR, administrative records; SR, self-reported data; CI, Condence Interval; kappa, Cohen’s
kappa measure of inter-rater agreement; PABAK, Prevalence and Bias Adjusted Kappa
The prevalence of utilization based on administrative records was higher than that based on self-reported
data, except for day-care services. There was a high degree of concordance between data sources, with
values ranging from approximately 91.24% for home treatment to 63.87% for outpatient care (table 2).
Page 12/30
Kappa-values varied across settings, ranging from 0.03 for overall outpatient services to 0.67 for use of
day-care services.After considering prevalence and bias, PABAK ranged from 0.28 (outpatient) to 0.82
(home treatment) and was markedly higher than the unadjusted kappa values for most of the resource
categories. Self-reported use of inpatient and outpatient services, and use of medications had higher
levels of sensitivity than specicity, whereas self-reported home treatment and day-care service had
higher specicity than sensitivity. Based on the kappa score, concordance between self-reported use of
medication and prescribing data across drug classes was fair (hypnotics and sedatives, k = 0.24) to
moderate (antipsychotics, k = 0.48) (see supplementary table 2 in Additional File 1).
3.3 Agreement between self-reported healthcare utilization and administrative data for quantity reporting
Table 3 shows the accuracy of self-reports in terms of the resource utilization volume, i.e., length of stay
in hospital in days and number of outpatient contacts.
Table 3. Differences in healthcare resource use between administrative records and self-reported data for
different medical services from the inpatient, day-care, and outpatient settings (n=274)
Page 13/30
Settings Psychiatric care Somatic Services All-cause
Inpatient
Number of days SR- Mean (SD) 32.21 (43.12) 0.97 (7.07) 33.18 (43.57)
Number of days AR - Mean (SD) 36.26 (38.24) 1.97 (7.06) 38.23 (39.47)
Underestimation - n (%) 121 (44.16) 42 (15.33) 126 (45.99)
Overestimation - n (%) 54 (19.71) 9 (3.28) 61 (22.26)
Total agreement - n (%) 99 (36.13) 223 (81.39) 87 (31.75)
(a)Zero in both sources - n (%) 66 (66.67) 220 (98.65) 63 (72.41)
Correlation - Spearman's rho 0.568 (p ≤ 0.001) 0.44 (p ≤ 0.001) 0.575 (p ≤ 0.001)
ICC [95% CI] 0.512 [0.42-0.59] 0.423 [0.32-0.52] 0.508 [0.41-0.59]
(b)
Day-care
Number of days SR- Mean (SD) 13.58 (29.41) 1.35 (11.88) 14.93 (31.25)
Number of days AR - Mean (SD) 10.53 (23.24) 0 10.53 (23.24)
Underestimation - n (%) 29 (10.58) 0 28 (10.22)
Overestimation - n (%) 54 (19.71) 5 (1.82) 58 (21.17)
Total agreement - n (%) 191 (69.71) 269 (98.18) 188 (68.61)
(a)Zero in both sources - n (%) 181 (94.76) 269 (100) 178 (94.68)
Correlation - Spearman's rho 0.718 (p ≤ 0.001) * 0.69 (p ≤ 0.001)
ICC [95% CI] 0.724 [0.66-0.78] * 0.653 [0.58-0.72]
Outpatient
Number of contacts SR- Mean (SD) 4.47 (8.06) 2.64 (4.42) 7.12 (9.89)
Number of contacts AR - Mean (SD) 4.65 (10.12) 7.31 (6.59) 11.96 (12.07)
Underestimation - n (%) 97 (35.4) 206 (75.18) 184 (67.15)
Overestimation - n (%) 89 (32.48) 33 (12.04) 72 (26.28)
Total agreement - n (%) 88 (32.12) 35 (12.77) 18 (6.57)
(a)Zero in both sources - n (%) 75 (85.23) 22 (62.86) 6 (33.33)
Correlation - Spearman's rho 0.429 (p ≤ 0.001) 0.271 (p ≤ 0.001) 0.309 (p ≤ 0.001)
ICC [95% CI] 0.219 [0.1-0.33] 0.232 [0.03-0.4] 0.217 [0.1-0.33]
Page 14/30
For calculation purposes, administrative records were treated as the reference for resource utilization.
Abbreviations: AR, administrative records; SR, self-reported data; ICC, intraclass correlation coecient;
95% CI, 95% condence interval; SD, Standard Deviation. (a)Self-reported data and administrative data
indicate no event; (b)Somatic services were not found in administrative data. *Too few cases to analyze.
Overall, participants self-reported on average 33.18 (SD ± 43.57) inpatient days, 14.93 (SD ± 31.25) days
in day-care hospitals, and 7.12 (SD ± 9.89) outpatient visits, while the administrative claims data
indicated on average 38.23 (SD ± 39.47) inpatient days, 10.53 (SD ± 23.24) days in day-care hospitals,
and 11.96 (SD ± 12.07) outpatient visits. When considering administrative records as a reference, most
participants accurately estimated the length of stay in day-care hospitals (68.61%) but tended to
predominantly under-report both inpatient days (45.99%) and outpatients visits (67.15%). The upper
panel of gure 1 (a-c) displays Bland-Altman plots comparing the number of events (hospital
days/outpatient visits) by self-reported data with the number of events in administrative records.
The frequency of higher positive average differences between data sources suggests a bias towards
under-reporting the true number of inpatient days and outpatient contacts. Correlation coecients
between administrative and self-reported data (table 3, gure 1d-e) were low for the outpatient setting, (ρ
= 0.31), moderate for the inpatient setting (ρ= 0.58), and high for day-care services (ρ = 0.69). Agreement
between data sources ranged from fair for outpatient contacts (ICC = 0.22) to substantial for day-care
services (ICC = 0.65). When psychiatric and somatic service utilization were examined separately, ICC was
only slightly better for inpatient psychiatric care (psychiatric departments = 0.57 vs somatic services =
0.42), and outpatient somatic care (somatic services = 0.23 vs psychiatric care = 0.22). Quantitative
agreement between data sources for the different components of inpatient and outpatient services are
shown in supplementary table 3 and supplementary table 4 (see Additional File 1), respectively. Due to
the small number of patients reporting on the number of contacts with healthcare professional in home
treatment (less than 10 %) and use of somatic services in day-care hospitals (less than 2 %), ICCs of
these categories could not be calculated (see supplementary table 5 and table 3 in Additional File 1).
3.4 Inuence of variables on difference in agreement of healthcare utilization
Logistic regression results for the agreement of any resource utilization (concordance on any resource
use: no = 0/yes = 1) are shown in Table 4.
Table 4. Odds ratio and corresponding 95% condence intervals from the logistic regression models for
the overall agreement on any utilization (yes/no) by healthcare resource
Page 15/30
Predictor Inpatient Day-care Outpatient Medication
Univariate models
Sex
Male(a)
0.87[0.47-
1.59] 1.01[0.51-
2.01] 0.66[0.4-1.08] 1.18[0.71-
1.98]
Age (years) 1.01[0.99-
1.03] 1.01[0.98-
1.03] 1.01[0.99-
1.03] 1.01[0.99-
1.03]
Age group
65+ years
0.75[0.35-
1.58] 0.93[0.38-
2.26] 1.16[0.6-2.24] 0.69[0.36-
1.32]
Time since 1sttreatment(b)
> 5 years
0.99[0.52-
1.89] 0.93[0.45-
1.95] 1.47[0.88-
2.48] 1.3[0.76-2.23]
Marital Status(c)
Married/living as married
2.48[1.00-
6.14] 0.87[0.4-1.91] 2.81[1.41-
5.59] 0.85[0.47-
1.56]
Education Level(d)
Tertiary
3.89[0.9-
16.85] 1.71[0.49-
5.93] 1.32[0.6-2.91] 1.41[0.61-
3.27]
Living situation(e)
Independent
accommodation
1.98[0.82-
4.81] 1[0.33-3.06] 2.01[0.91-
4.43] 0.36[0.12-
1.06]
Multivariate models
Sex
Male(a)
0.9[0.44-1.82] 0.96[0.45-
2.07] 0.7[0.39-1.24] 1.05[0.59-
1.89]
Age (years) 1.01[0.98-
1.05] 1.02[0.98-
1.06] 1[0.98-1.03] 1.02[0.99-
1.05]
Age group
65+ years
0.37[0.11-
1.21] 0.59[0.16-
2.16] 0.65[0.25-
1.72] 0.39[0.15-
1.03]
Time since 1sttreatment(b)
> 5 years
1.21[0.59-
2.46] 0.74[0.33-
1.66] 1.82[1.03-
3.23] 1.38[0.77-
2.47]
Marital Status(c)
Married
1.99[0.75-
5.25] 0.74[0.3-1.81] 2.34[1.12-
4.91] 0.77[0.4-1.5]
Education Level(d) 3.2[0.72- 1.73[0.48- 1.13[0.49-2.6] 1.46[0.61-
Page 16/30
Tertiary 14.26] 6.18] 3.52]
Living situation(e)
Independent
accommodation
1.62[0.61-
4.33] 1.14[0.35-
3.71] 1.86[0.79-
4.39] 0.32[0.09-
1.14]
Nagelkerkes R2
0.22 0.12 0.22 0.19
Reference: (a)female. (b)≤ 5 years. (c)Single/Married but living separated/ Divorced/ Widowed.
(d)Primary/Secondary; (e)Supervised accommodation/Homeless/Other. Signicant associations (P ≤
0.05) are denoted in bold.
In the univariate analyses, the only statistically signicant associations found were between marital
status and any use of inpatient and outpatient services. Married or living as married adults were more
likely to recall the occurrence of any inpatient event (OR [95%-CI] = 2.48[1.00-6.14]) and outpatient visit
(OR [95%-CI] = 2.81[1.41-5.59]) compared with their counterparts who were either single, married but living
separated, divorced, or widowed. In terms of the magnitude and direction of effect, the results of
multivariate analysis are similar. Marital status again attained statistical signicance for the association
with outpatient services. Additionally, individuals in psychiatric treatment longer than 5 years were also
more likely to correctly report any outpatient visit in the multivariate model (OR [95%-CI] = 1.82[1.03-
3.23]).For inpatient and day-care services, and medication use, no predictor had a consistent association
with the agreement between data sources.
Next, linear regression models were used to assess the predictors of under- or overreporting for volume
utilization measures. Table 5 shows regression coecients for socio-demographic predictors of the
difference in the number of events (days/contacts) measured with administrative records versus self-
reported data. For the outpatient sector, the statistically signicant factors were sex, age, and number of
events. Males tended to overreport outpatient visits, while increase in age resulted in underreporting of
outpatient contacts. An increase in the number of inpatient days and outpatient contacts resulted in
overreporting of events in the respective sectors.
Table 5. Multivariate linear regression models for predictors of differences between self-reported and
administrative data for the volume of utilization (number of days/contacts) by healthcare resource
Page 17/30
Predictor Inpatient Day-care Outpatient
Beta SE Beta SE Beta SE
Sex
(0 = Female, 1 = Male)
-0.08 5.23 -1.79 3.15 -4.88 1.17
Age (years) 0.30 0.24 -0.05 0.15 0.11 0.05
Age group
(0 = 18-64, 1 = 65+)
-0.86 8.82 3.73 5.32 -1.37 1.98
Time since 1sttreatment
(0 = ≤ 5 years, 1 = > 5 years)
3.14 5.24 -1.13 3.16 2.07 1.18
Marital Status
(0 = Married/living as married, 1 = other)
-1.50 6.02 -1.95 3.63 -1.45 1.35
Education Level
(0 = primary/secondary, 1 = tertiary)
7.08 7.35 -0.60 4.44 -0.71 1.65
Living situation
(0 = independent accommodation, 1 = other)
-15.60 8.38 1.11 5.05 1.54 1.88
(a)Number of inpatient days -0.42 0.06 - - - -
(a)Number of days in day-care - - -0.08 0.07 - -
(a)Number of outpatient contacts - - - - -0.62 0.05
Explained variance -
R2
0.2 0.01 0.47
(a)Days/contacts from administrative records. Signicant associations (P ≤ 0.05) are denoted in bold.
4. Discussion
4.1 Inpatient and day-care services
Our ndings regarding concordance on any stay in hospital or day-care facilities and their volume of
utilization are partially consistent with previous studies in individuals with mental and behavioral
disorders [15,20,40–42]. The agreements observed for inpatient and day-care services were moderate and
substantial respectively. Kappa values and correlation coecients were slightly lower than that reported
in studies using hospital computerized claims databases [20,40], but higher than that using GP records
[15,41] or administrative data from medical service plans [42].
Page 18/30
A direct quantitative comparison to these other investigations is dicult, however, due to variations in
study design (e.g., self-administered vs. interviewer-administrated approaches) and measures of validity.
Limitations in comparability constitute potential explanations for the observed differences in results
between the different studies. For instance, some reports did not include non-psychiatric care at all; others
combined data from hospital admissions and day-care facilities in one single variable. Of those that
examined psychiatric and psychosomatic services separately for any resource use, Mistry et al. (2005)
[41] found that agreement between self-reported data and GP records was fair forpsychiatric hospital
and community services (κw= 0.278) and moderate for other hospital and specialist services (κw = 0.430,
including emergency department and day hospitals). Somers et al. (2016) [42] found that concordance
between self-reports and administrative data ranged from fair to moderate for psychiatric hospitalization
(κ = 0.21 for hospitalization over 6 months; κ = 0.5 for ≥ 2 hospitalizations) and moderate for any
hospitalization (κ = 0.44, without emergency department contacts). Heinrich et al. (2011) [20] reported a
substantial agreement (κ = 0.78) between CSSRI data and hospital records for any use of psychiatric
inpatient and day-care services combined; the authors suggest that hospital data could be superior to GP
records. In the current study, when considering admission reports from hospitals and day-care facilities
separately and adjusting for prevalence and bias, substantial agreement coecients were found for any
utilization (PABAK = 0.62, inpatient sector; 0.72, day-care sector).
Regarding volume of service utilization, estimates of agreement for overall inpatient service utilization
varied from substantial for length of stay (ρc = 0.66) to almost perfect for the number of admissions (ICC
= 0.9; ρc =0.95) [15,16,40]. For agreement on the number of days in psychiatric settings, correlation
coecients ranged from fair (ρc = 0.27) to almost perfect (CCC = 0.89), while concordance for the
number of days in non-psychiatric hospitals was found to be only fair (ρc = 0.27) [15,20]. Here again, a
more comprehensive analysis of concordance between self-reported data and administrative records was
performed, separately considering admission reports from hospitals and day-care facilities for both
psychiatric and somatic services, and overall admissions. Moderate concordance was found for all
subdivisions of inpatient services, and again our correlation values were lower than those found in the
study using hospital records [20] and higher or similar to those using GP records [15].
Agreement on the length of stay in day-care facilities was the strongest found in our analysis (ICC=0.72,
psychiatry; 0.65, overall), though this nding is not supported by previous investigations which reported
very low values of agreement for day-care services [16,20]. Despite the fact that agreement on day-care
services mainly resulted from a large proportion of respondents reporting no service use, 36.5% of
patients received day-care treatment in at least one of the data sources, which may have contributed to
the greater values of correlation in our study. Of note is that 59.8% of our sample was composed of
patients receiving treatment in FIT hospitals, where inpatient treatment intensity was shown to be
reduced in association with an increase in day-care [43]. Here, a positive effect on the recall of patients
experiencing day-care services in FIT hospitalscannot be ruled out. Further work will be required to clarify
this issue.
4.2 Outpatient services and medication use
Page 19/30
In stark contrast to inpatient and day-care settings, the concordance between self-reports and claims-
based data on any use of outpatient services and medications was just slight to fair (κ = 0.03 and 0.25,
respectively). The available literature contains little information on the accuracy of self-reports compared
to data records with regard to combined outpatient services and prescribed medications for mental health
populations [15,21,44]; kappa values are not reported in any study.
For the general population, kappa values for the concordance of outpatient events have typically ranged
from moderate to substantial depending on the medical specialty [45–47]. Overall agreement for
combined outpatient services, including visits to a psychiatrist, has been found to be fair [48].
Furthermore, concordance based on the kappa statistic has been substantial to almost perfect in terms of
overall medication utilization [49,50], and from fair to almost perfect concerning the use of psychotropic
medications in both general populations [51–53] and in a mixed sample of unaffected and mentally ill
subjects [54].
Our results on the dichotomous agreement between data sources for outpatient services and medication
use are similar to those found in individuals with chronic conditions [55–57]. As both outpatient service
utilization and medication use were underreported when compared to administrative data, poor health
status may have affected the recall of utilization and consequently the agreement for less signicant or
salient events. On the other hand, a previous study has found no association between disagreement and
psychiatric diagnosis for the recall of outpatient events [48]. In addition, since we coded missing data as
negative self-reports of resource utilization, it was not possible to differentiate between participants who
intended to deny outpatient services or medication use (by leaving the question blank) and those with no
motivation to record and report their data. Alternatively, the lower prevalence of any medication use found
in self-reports may be a result of inaccuracies in administrative data. Firstly, patients may not have taken
the drugs they were prescribed. For instance, more than 25% of individuals who receive a rst-time
antidepressant prescription decline treatment either by not starting treatment or by terminating treatment
prematurely [58]. Secondly, in the claims database any prescription within 3 months prior to baseline
assessment was considered a positive event, whereas CSSRI specically referred to medications for
mentaldisorders. Notably,although psychotropic medications have been considered a drug class
vulnerable to stigmatization bias [59], we found only a small difference in kappa values for psychotropic
drugs compared to other classes of medications (κ= 0.35 vs. 0.37 respectively), indicating that
participants were not reluctant to report the use of psychoactive drugs.
Regarding volume of service utilization for outpatient contacts, agreement for the number of GP contacts
andpsychiatrist/psychotherapist contactswas fair, contrasting with the substantial agreement (CCC =
0.76, ρc = 0.63, GP contacts; ρ = 0.79, psychotherapist contacts) reported by previous research using
provider records as a gold standard, such as GP records [14,15] and psychotherapists registrations [60].
Agreements on the number of contacts and time spent with outpatient psychiatrists/psychologists were
moderate (ρc = 0.54, contacts) in studies in which psychiatric case registers were used for reference [16].
On the other hand, studies using billing data to compare self-reports on the number of GP contacts found
only a small agreement (ρc = 0.1, 10.9% agreement) between data sources [61,62]. Interestingly, another
Page 20/30
study found that agreement between CSRI data and GP records for the number of psychiatric visits, other
outpatient services and combined outpatient services ranged from slight to fair (ρc = 0.17, 0.27 and 0.23,
respectively) suggesting that GP records are more reliable and may provide more accurate information on
GP-contacts than on contacts with other outpatient providers [15]. Likewise, only a fair agreement was
found in our study for specialist and outpatient services combined using insurance claims as a
comparator. Correlation and agreement were found by our study to be better for predominant care (i.e.,
psychiatric services) than somatic services (total agreement = 89%, ρ=0.43 vs total agreement = 33%,
ρ=0.27), similarly to what has previously been reported for epilepsy patients [61]. The observed
underestimation of the number of overall outpatient contacts (67.15%, -4.84 contacts) was attributed to
reports on the number of GP visits. This nding is supported by one [15], but not by other previous
investigations [14,61,62]. In some cases, underreporting of GP contacts may occur when patients who
also have regular contacts in mental health facilities misremember whether the doctor visit was a general
or a psychiatric contact.
The number of self-reported outpatient contacts was observed to have an important impact on the total
agreement of the number of events. However, in contrast to previous investigations [37], patients in our
study with a greater number of outpatient visits were more likely to overreport than underreport the
number of outpatient contacts. On the other hand, there are also plausible explanations for this result; our
sample consisted of participants with mentaldisorderswho use many different types of health service,
such as complementary and therapeutic care. It is possible that subjects confused these with outpatient
visits, which were then consequently over-reported. In fact, similar ndings on overreporting have been
found for individuals with self-rated poor health status in general population-based studies [46,63].
Consistent with previous ndings [37,48,64] and as expected due to memory impairment, increasing age
was a signicant determinant of underreporting outpatient contacts. Concerning the agreement on any
use of outpatient services, married status and being in psychiatric treatment for more than 5 years were
associated with a greater likelihood of concordance. Being married can be associated with positive social
support for a patient’s engagement in medical treatment [65] and possibly with better recall of the
services used, at least in the outpatient setting. Likewise, people in a longer period of psychiatric
treatment could be more concerned with their health and more engaged in their treatment providing a
more correct estimation on the use of services. However, more research is needed to conrm these
conjectures.
4.3 Limitations
As most questions in CSSRI were questions without response alternatives, missingness could not be
established if an individual item was blank, and non-use was therefore assumed. This may have caused
potential misclassication for some users, but we do not believe that disagreement between data sources
was a consequence of this treatment. Once the last section of the questionnaire included “yes/no”
checkboxes, it could be observed that 96% of participants attempted to solve at least one of the
subsequent items. Some of the observed missing data were then more likely to be structurally missing
data (missing by design) or omitted responses due to zero occurrences, instead of refusals to respond.
Page 21/30
For example, dates of admission and discharge would have been structurally missing for all patients
without hospitalization events. In situations where the non-responses were missing by design or where
omitted responses resulted from lack of utilization, it was reasonable to assume that items left missing
represented non-use. In a previous report, only a slightly reduced agreement on the use of medication was
found when missing responses were recoded as non-use [53].
Apart from the medication, inpatient and outpatient services that were evaluated and compared in this
study, the CSSRI data collected in the PsychCare study contain information on further non-medical
service utilization (e.g., contacts to social workers, self-help groups or debt advice centers) relevant for
cost recording from a societal perspective. However, it was not possible to assess the accuracy of this
patient-reported information in this study, as most of these service providers are used outside the scope
covered by German SHI funds. Thus, there is no administrative data collection for billing purposes.
Additionally, it should be noted that self-reported data are based on the subject’s recall and perception.
Therefore, misinterpretation by the patient as to the type of healthcare service referred to in the
questionnaire cannot be ruled out (for instance, psychiatrist vs. psychotherapist).
Utilization of outpatient services and home treatment was assessed either as the exact number of
contacts or as a frequency, which was then multiplied by the recall period length to estimate the total
number of contacts. Consequently, inaccuracies may be caused by the exclusion of patients who gave
responses lacking interpretability (e.g., answers such as “now and then”, “barely” etc.). Another important
aspect in our analyses is the 6-month recall period, which is rather long for psychiatric patients with high
service uptake and thus a source of potential bias.
Finally, unadjusted kappa values remain low even in the occurrence of a large proportion of concordant
pairs, e.g., in case of outpatient contacts and medication use. In addition, most of our data showed a
large number of zeros (i.e., no occurrence of events), which can be challenging for the calculation of the
ICC, which assumes a normal distribution of the data. In terms of generalizability, kappa and ICC values
might not be directly cross-country comparable because service utilization is potentially subject to health
care systems variability. On the other hand, outcomes of interest (e.g., inpatient days and outpatient
visits) are usual services in many countries. In this context, our ndings were presented in an easily
interpretable manner providing absolute numbers for both data sources and cross tables so as to support
comparability with future studies.
5. Conclusion
In summary, the results of our study extend previous ndings based on much smaller samples. The aim
was to assess the degree of agreement between self-reported and administrative data on healthcare
utilization by persons with mentaldisorders. The ndings of the PsychCare study considerably expand
the evidence base regarding the potential accuracy of CSSRI-D. There was relatively high absolute
concordance on any resource use across all settings (up to 91%), but due to the differences among
positive and negative agreements, the kappa values were generally low.Inclusion of indicators less
Page 22/30
sensitive to sampling bias and prevalence, such as the adjusted PABAK depending on observed
agreement, resulted in consistently higher agreement rates in our study.Frequent events, such as
outpatient appointments, were less accurately reported than less frequent and possibly more salient
events, such as hospital admissions (inpatient and day-care). The two data sources had fair agreement
for the number of outpatient contacts, moderate agreement for the number of the inpatient days, and
substantial agreement for the length of stay in day-care facilities in the past 6 months.Based on these
results, we conclude that, for a 6-month recall period, CSSRI-D and SHI funds data on hospital
admissions (inpatient and day-care) are readily exchangeable, while for outpatient visits and medication
use they are not. Results derived from investigations relying on just one of these data sources must be
interpreted with caution. Alternatively, conducting individual-level linkages of primary and secondary data
could improve data quality and strengthen the ndings. For statistical evaluation of healthcare utilization
data derived from different sources, the choice of statistical indicators adjusting for the expected degree
of agreement solely by chance must also be discussed. It seems recommendable to perform statistical
evaluation based on a combination of agreement indicators rather than on single measures alone. To
reduce potential bias in self-reported data, shortening the recall periods may be an option; however, at the
same time, valid health economic cost measurements require longer time horizons to validly assess the
impact of innovative forms of care. Shorter observation periods would also require more frequent data
collection points in longitudinal studies; it should be carefully considered whether this is a benecial
option, taking into account the risk of a loss-to-follow-up, particularly in vulnerable patient groups.
Abbreviations
ATC: Anatomical Therapeutic Chemical Classication System; CSRI: Client Service Receipt Inventory;
CSSRI-EU: European version of the Client Socio-Demographic and Service Receipt Inventory; FIT: exible
and integrative psychiatric treatment; GPs: general practitioners; GTB: global treatment budget; SHI:
statutory health insurance funds; ICD: International Statistical Classication of Diseases and Related
Health Problems; ICC: intraclass correlation coecients; PABAK: prevalence-adjusted bias-adjusted
kappa; PIA: psychiatric outpatient departments.
Declarations
7.1. Ethics approval and consent to participate
PsychCare trial was reviewed and approved by Institutional Review Board of the Medical Faculty of the
Technical University Dresden and at each site where a separate approval was mandatory. All methods
were performed in accordance with relevant guidelines and regulations, including the 1964 Declaration of
Helsinki and its later amendments. The authors arm that signed informed consent was obtained from
all individual participants included in this study.
7.2. Consent for publication
Page 23/30
Not applicable.
7.3. Availability of data and materials
The datasets generated and/or analyzed during the current study are not publicly available.
The datasets used during the current study are available from the corresponding author on reasonable
request.
7.4. Competing interests
Unrelated to this study, Prof. Schmitt reports institutional grants for investigator-initiated research from
the German GBA, the BMG, BMBF, EU, Federal State of Saxony, Novartis, Sano, ALK, and Pzer. He also
participated in advisory board meetings as a paid consultant for Sano, Lilly, and ALK. The other authors
declare that they have no conict of interest.
7.5. Funding
This study was part of the multi-center controlled PsychCare trial, which was funded by the Innovation
Fund at the Federal Joint Committee Germany under reference number 01VSF16053. The funder had no
role in the study design and is not involved in its execution, data analysis, and dissemination of results.
7.6. Authors' contributions
All authors read the manuscript and approved the nal version submitted for publication. TBG conducted
the statistical analysis and data interpretation, performed literature searches, and wrote the rst draft of
the manuscript. RK contributed to the study design and data interpretation, drafted, and critically reviewed
the manuscript. FC contributed to the study design, drafted and critically reviewed the manuscript. AN co-
coordinated the study, contributed to the study design, drafted, and critically reviewed the manuscript. BS
co-coordinated the study and contributed to the study design and data interpretation. FB conducted the
statistical analyses and contributed to data interpretation, drafted and critically reviewed the manuscript.
JuS contributed to the study design and critically reviewed the manuscript. ES co-coordinated the study,
was responsible for data management, contributed substantially to the study design and critically
reviewed the manuscript. JoS was co-chief investigator, contributed to the study design, and critically
reviewed the manuscript. AP was co-chief investigator, contributed to the study design and critically
reviewed the manuscript. DH co-coordinated the study, contributed substantially to the study design, and
critically reviewed the manuscript. IW contributed to the study design and data interpretation, drafted, and
critically reviewed the manuscript.
7.7. Acknowledgements
The multi-center controlled PsychCare trial was funded by the Innovation Fund at the Federal Joint
Committee Germany. The authors would like to thank the six German health insurance funds for
Page 24/30
providing the necessary claims data and cooperating with the research consortium. We are also grateful
to all participants in the PsychCare trial. Many thanks to Alfred Newman for writing and editing support.
References
1. Global, regional, and national burden of 12 mental disorders in 204 countries and territories, 1990–
2019: a systematic analysis for the Global Burden of Disease Study 2019. The Lancet Psychiatry
2022;9(2):137–50.
2. Bloom JR, Wang H, Kang SH, Wallace NT, Hyun JK, Hu T. Capitation of public mental health services
in Colorado: a ve-year follow-up of system-level effects. Psychiatric services (Washington, D.C.)
2011;62(2):179–85.
3. Bundesanstalt für Arbeitsschutz und Arbeitsmedizin (BAuA). Volkswirtschaftliche Kosten durch
Arbeitsunfähigkeit 2020; Bundesanstalt für Arbeitsschutz und Arbeitsmedizin (BAuA): Dortmund,
Germany, 2020. 2020.
4. Gansen FM. Health economic evaluations based on routine data in Germany: a systematic review.
BMC health services research 2018;18(1):268.
5. König H, König H-H, Konnopka A. The excess costs of depression: a systematic review and meta-
analysis. Epidemiology and psychiatric sciences 2019;29:e30.
. Evans E, Grella CE, Murphy DA, Hser Y-I. Using administrative data for longitudinal substance abuse
research. The journal of behavioral health services & research 2010;37(2):252–71.
7. Noben CY, Rijk A de, Nijhuis F, Kottner J, Evers S. The exchangeability of self-reports and
administrative health care resource use measurements: assessement of the methodological
reporting quality. Journal of clinical epidemiology 2016;74:93-106.e2.
. Leggett LE, Khadaroo RG, Holroyd-Leduc J, Lorenzetti DL, Hanson H, Wagg A et al. Measuring
Resource Utilization: A Systematic Review of Validated Self-Reported Questionnaires. Medicine
2016;95(10):e2759.
9. Mayer S, Paulus ATG, Łaszewska A, Simon J, Drost RMWA, Ruwaard D et al. Health-Related
Resource-Use Measurement Instruments for Intersectoral Costs and Benets in the Education and
Criminal Justice Sectors. PharmacoEconomics 2017;35(9):895–908.
10. Janssen LMM, Drost RMWA, Paulus ATG, Gareld K, Hollingworth W, Noble S et al. Aspects and
Challenges of Resource Use Measurement in Health Economics: Towards a Comprehensive
Measurement Framework. PharmacoEconomics 2021;39(9):983–93.
11. Thorn JC, Coast J, Cohen D, Hollingworth W, Knapp M, Noble SM et al. Resource-use measurement
based on patient recall: issues and challenges for economic evaluation. Appl Health Econ Health
Policy 2013;11(3):155–61.
12. Franklin M, Thorn J. Self-reported and routinely collected electronic healthcare resource-use data for
trial-based economic evaluations: the current state of play in England and considerations for the
future. BMC medical research methodology 2019;19(1):8.
Page 25/30
13. Knapp M, Beecham J, Gordon K. Predicting the community cost of closing psychiatric hospitals:
National extrapolations. Journal of Mental Health 1992;1(4):315–25.
14. Patel A, Rendu A, Moran P, Leese M, Mann A, Knapp M. A comparison of two methods of collecting
economic data in primary care. Family practice 2005;22(3):323–7.
15. Byford S, Leese M, Knapp M, Seivewright H, Cameron S, Jones V et al. Comparison of alternative
methods of collection of service use data for the economic evaluation of health care interventions.
Health Econ. 2007;16(5):531–6.
1. Mirandola M, Biso G, Bonizzato P, Amaddeo F. Collecting psychiatric resources utilisation data to
calculate costs of care: a comparison between a service receipt interview and a case register. Social
psychiatry and psychiatric epidemiology 1999;34(10):541–7.
17. Chisholm D, Knapp MR, Knudsen HC, Amaddeo F, Gaite L, van Wijngaarden B. Client Socio-
Demographic and Service Receipt Inventory--European Version: development of an instrument for
international research. EPSILON Study 5. European Psychiatric Services: Inputs Linked to Outcome
Domains and Needs. The British journal of psychiatry. Supplement 2000(39):s28-33.
1. Chung CCY, Fung JLF, Lui ACY, Chan MCY, Ng YNC, Wong WHS et al. Client Service Receipt Inventory
as a standardised tool for measurement of socio-economic costs in the rare genetic disease
population (CSRI-Ra). Sci Rep 2021;11(1):23837.
19. Roick C, Kilian R, Matschinger H, Bernert S, Mory C, Angermeyer MC. Die deutsche Version des Client
Sociodemographic and Service Receipt Inventory - Ein Instrument zur Erfassung psychiatrischer
Versorgungskosten. Psychiatrische Praxis 2001;28 Suppl 2:S84-90.
20. Heinrich S, Deister A, Birker T, Hierholzer C, Weigelt I, Zeichner D et al. Accuracy of self-reports of
mental health care utilization and calculated costs compared to hospital records. Psychiatry
research 2011;185(1-2):261–8.
21. Zentner N, Baumgartner I, Becker T, Puschner B. Kosten medizinischer Leistungen bei Menschen mit
schweren psychischen Erkrankungen: Selbstbericht vs. Kostenträgerangaben. Psychiatrische Praxis
2012;39(3):122–8.
22. Sousa A, Cardoso AA, Kayo M, Gregorio G, Jesus Mari J de, Razzouk D. The Adaption of the Client
Sociodemographic and Service Receipt Inventory for Costing Mental Health Services in Brazil. The
journal of mental health policy and economics 2018;21(3):131–42.
23. Schwarz J, Schmid C, Neumann A, Pfennig A, Soltmann B, Heinze M et al. Implementierung eines
globalen Behandlungsbudgets in der Psychiatrie – Welche Anreize, Voraussetzungen und
Herausforderungen gibt es? Psychiatrische Praxis 2022;49(2):71–9.
24. Kliemt R, Häckl D. Anreize und Weiterentwicklungsperspektiven der Vergütung von Psychiatrie und
Psychosomatik unter der Berücksichtigung von Modellvorhaben. In: Klauber J, Geraedts M, Friedrich
J, Wasem J, Beivers A, editors. Krankenhaus-Report 2020. Berlin, Heidelberg: Springer Berlin
Heidelberg; 2020. p. 263–279.
25. Soltmann B, Neumann A, March S, Weinhold I, Häckl D, Kliemt R et al. Multiperspective and
Multimethod Evaluation of Flexible and Integrative Psychiatric Care Models in Germany: Study
Page 26/30
Protocol of a Prospective, Controlled Multicenter Observational Study (PsychCare). Frontiers in
psychiatry 2021;12:659773.
2. Neumann A, Swart E, Häckl D, Kliemt R, March S, Küster D et al. The inuence of cross-sectoral
treatment models on patients with mental disorders in Germany: study protocol of a nationwide long-
term evaluation study (EVA64). BMC Psychiatry 2018;18(1):139.
27. Swart E, Gothe H, Geyer S, Jaunzeme J, Maier B, Grobe TG et al. Gute Praxis Sekundärdatenanalyse
(GPS): Leitlinien und Empfehlungen. Gesundheitswesen (Bundesverband der Arzte des Offentlichen
Gesundheitsdienstes (Germany)) 2015;77(2):120–6.
2. Swart E, Schmitt J. STandardized Reporting Of Secondary data Analyses (STROSA)—Vorschlag für
ein Berichtsformat für Sekundärdatenanalysen. Zeitschrift fur Evidenz, Fortbildung und Qualitat im
Gesundheitswesen 2014;108(8-9):511–6.
29. March S, Andrich S, Drepper J, Horenkamp-Sonntag D, Icks A, Ihle P et al. Gute Praxis Datenlinkage
(GPD). Gesundheitswesen (Bundesverband der Arzte des Offentlichen Gesundheitsdienstes
(Germany)) 2019;81(8-09):636–50.
30. R Core Team (2021). R: A language and environment for statistical computing. R Foundation for
Statistical Computing, Vienna, Austria. URL https://www.R-project.org/. Ein BibTeX-Eintrag für LaTeX-
Benutzer ist.
31. Max Kuhn (2021). caret: Classication and Regression Training. R package version 6.0-90.
https://CRAN.R-project.org/package=caret.
32. Chen G, Faris P, Hemmelgarn B, Walker RL, Quan H. Measuring agreement of administrative data with
chart data using prevalence unadjusted and adjusted kappa. BMC medical research methodology
2009;9:5.
33. Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. Journal of clinical epidemiology
1993;46(5):423–9.
34. Perez-Jaume S, Skaltsa K, Pallarès N, Carrasco JL (2017). “ThresholdROC: Optimum Threshold
Estimation Tools for Continuous Diagnostic Tests in R.” _Journal of Statistical Software,*82*(4), 1-
21. doi: 10.18637/jss.v082.i04 (URL: https://doi.org/10.18637/jss.v082.i04).
35. Matthias Gamer, Jim Lemon and Ian Fellows Puspendra Singh <puspendra.pusp22@gmail.com>
(2019). irr: Various Coecients of Interrater Reliability and Agreement. R package version 0.84.1.
https://CRAN.R-project.org/package=irr.
3. J. Richard Landis and Gary G. Koch. The Measurement of Observer Agreement for Categorical Data.
International Biometric Society 1977;33:159–74.
37. Bhandari A, Wagner T. Self-reported utilization of health care services: improving measurement and
accuracy. Medical care research and review MCRR 2006;63(2):217–35.
3. Noble N, Paul C, Walsh J, Wyndham K, Wilson S, Stewart J. Concordance between self-report and
medical records of preventive healthcare delivery among a sample of disadvantaged patients from
four aboriginal community controlled health services. BMC health services research 2019;19(1):111.
Page 27/30
39. Waaseth M, Havelkova M, Forsdahl G, Lund E, Log T. Use of Selective Serotonin Reuptake Inhibitors -
Validity of Self-Report versus Plasma Concentrations and Pharmacy Dispensations - A Cross-
Sectional Analysis of the Norwegian Women and Cancer Study. Clinical epidemiology 2022;14:815–
26.
40. Cheung AH, Dewa CS, Wasylenki D. Economic grand rounds: impact on cost estimates of differences
in reports of service use among clients, caseworkers, and hospital records. Psychiatric services
(Washington, D.C.) 2003;54(10):1328–30.
41. Mistry H, Buxton M, Longworth L, Chatwin J, Peveler R. Comparison of general practitioner records
and patient self-report questionnaires for estimation of costs. The European journal of health
economics HEPAC health economics in prevention and care 2005;6(3):261–6.
42. Somers JM, Moniruzzaman A, Currie L, Rezansoff SN, Russolillo A, Parpouchi M. Accuracy of
reported service use in a cohort of people who are chronically homeless and seriously mentally ill.
BMC psychiatry 2016;16:41.
43. Baum F, Schoffer O, Neumann A, Seifert M, Kliemt R, March S et al. Effectiveness of Global Treatment
Budgets for Patients With Mental Disorders-Claims Data Based Meta-Analysis of 13 Controlled
Studies From Germany. Frontiers in psychiatry 2020;11:131.
44. Kashner T, Suppes T, Rush A, Altshuler K. Measuring use of outpatient care among mentally ill
individuals: a comparison of self reports and provider records. Evaluation and Program Planning
1999;22(1):31–40.
45. Palmer L, Johnston SS, Rousculp MD, Chu B-C, Nichol KL, Mahadevia PJ. Agreement between
Internet-based self- and proxy-reported health care resource utilization and administrative health care
claims. Value in health the journal of the International Society for Pharmacoeconomics and
Outcomes Research 2012;15(3):458–65.
4. Peersman W, Pasteels I, Cambier D, Maeseneer J de, Willems S. Validity of self-reported utilization of
physician services: a population study. European journal of public health 2014;24(1):91–7.
47. van der Heyden J, Charafeddine R, Bacquer D de, Tafforeau J, van Herck K. Regional differences in
the validity of self-reported use of health care in Belgium: selection versus reporting bias. BMC
medical research methodology 2016;16(1):98.
4. Drapeau A, Boyer R, Diallo FB. Discrepancies between survey and administrative data on the use of
mental health services in the general population: ndings from a study conducted in Québec. BMC
public health 2011;11:837.
49. Wu C-S, Lai M-S, Gau SS-F, Wang S-C, Tsai H-J. Concordance between patient self-reports and claims
data on clinical diagnoses, medication use, and health system utilization in Taiwan. PloS one
2014;9(12):e112257.
50. Savitz ST, Stearns SC, Zhou L, Thudium E, Alburikan KA, Tran R et al. A Comparison of Self-reported
Medication Adherence to Concordance Between Part D Claims and Medication Possession. Medical
care 2017;55(5):500–5.
Page 28/30
51. Haapea M, Miettunen J, Lindeman S, Joukamaa M, Koponen H. Agreement between self-reported
and pharmacy data on medication use in the Northern Finland 1966 Birth Cohort. International
journal of methods in psychiatric research 2010;19(2):88–96.
52. Richardson K, Kenny RA, Peklar J, Bennett K. Agreement between patient interview data on
prescription medication use and pharmacy records in those aged older than 50 years varied by
therapeutic group and reporting of indicated health conditions. Journal of clinical epidemiology
2013;66(11):1308–16.
53. Hafferty JD, Campbell AI, Navrady LB, Adams MJ, MacIntyre D, Lawrie SM et al. Self-reported
medication use validated through record linkage to national prescribing data. Journal of clinical
epidemiology 2018;94:132–42.
54. Haukka J, Suvisaari J, Tuulio-Henriksson A, Lönnqvist J. High concordance between self-reported
medication and ocial prescription database information. European journal of clinical
pharmacology 2007;63(11):1069–74.
55. Gordon LG, Patrao T, Hawkes AL. Can colorectal cancer survivors recall their medications and doctor
visits reliably? BMC health services research 2012;12:440.
5. Jiang L, Zhang B, Smith ML, Lorden AL, Radcliff TA, Lorig K et al. Concordance between Self-Reports
and Medicare Claims among Participants in a National Study of Chronic Disease Self-Management
Program. Frontiers in public health 2015;3:222.
57. Booth MJ, Clauw D, Janevic MR, Kobayashi LC, Piette JD. Validation of Self-Reported Rheumatoid
Arthritis Using Medicare Claims: A Nationally Representative Longitudinal Study of Older Adults. ACR
open rheumatology 2021;3(4):239–49.
5. van Geffen ECG, Gardarsdottir H, van Hulten R, van Dijk L, Egberts ACG, Heerdink ER. Initiation of
antidepressant therapy: do patients follow the GP's prescription? The British journal of general
practice the journal of the Royal College of General Practitioners 2009;59(559):81–7.
59. Semahegn A, Torpey K, Manu A, Assefa N, Tesfaye G, Ankomah A. Psychotropic medication non-
adherence and its associated factors among patients with major psychiatric disorders: a systematic
review and meta-analysis. Systematic reviews 2020;9(1):17.
0. Bouwmans C, Jong K de, Timman R, Zijlstra-Vlasveld M, van der Feltz-Cornelis C, Tan Swan S et al.
Feasibility, reliability and validity of a questionnaire on healthcare consumption and productivity loss
in patients with a psychiatric disorder (TiC-P). BMC health services research 2013;13:217.
1. Lairson DR, Basu R, Begley CE, Reynolds T. Concordance of survey and billing data in a study of
outpatient healthcare cost and utilization among epilepsy patients. Epilepsy research
2009;87(1):59–69.
2. Palin JL, Goldner EM, Koehoorn M, Hertzman C. Prevalence and frequency of mental health care
provided by general practitioners: differences between 2 national data sources for the same
population. Canadian journal of psychiatry. Revue canadienne de psychiatrie 2012;57(6):366–74.
3. Bellón JA, Lardelli P, Luna JD, Delgado A. Validity of self reported utilisation of primary health care
services in an urban population in Spain. Journal of epidemiology and community health
Page 29/30
2000;54(7):544–51.
4. Hunger M, Schwarzkopf L, Heier M, Peters A, Holle R. Ocial statistics and claims data records
indicate non-response and recall bias within survey-based estimates of health care utilization in the
older population. BMC health services research 2013;13:1.
5. DiMatteo MR. Social support and patient adherence to medical treatment: a meta-analysis. Health
psychology ocial journal of the Division of Health Psychology, American Psychological Association
2004;23(2):207–18.
Figures
Page 30/30
Figure 1
Agreement between self-reported data and administrative records for the number of events in different
health care settings for the same period. Upper panels: Bland-Altman Plots for the number of days (a, b)
or number of contacts (c) found in both data sources. The average of the two measurements is plotted
along the horizontal axis and the difference between the two methods is plotted along the vertical axis.
Lower panels: scatterplots for the correlations between data sources in the inpatient (d), day-care (e) and
outpatient (f) settings
Supplementary Files
This is a list of supplementary les associated with this preprint. Click to download.
Additionalle1.docx
