Interpreting treatment-effect estimates with heterogeneity and choice: simulation model results.

College of Pharmacy, University of Iowa, Iowa City, Iowa 52242, USA.
Clinical Therapeutics (Impact Factor: 2.59). 05/2009; 31(4):902-19. DOI: 10.1016/j.clinthera.2009.04.007
Source: PubMed

ABSTRACT Researchers using observational data in health-services research use various treatment-effect estimators to reduce the bias associated with unmeasured confounding variables and have focused on estimate differences to indicate the relative ability of these estimators to mitigate bias. However, available estimators may identify different treatment-effect concepts; if treatment effects are heterogeneous across patients and treatment choice reflects "sorting on the gain," then treatment-effect estimates should differ regardless of confounding. Risk-adjustment approaches yield estimates of the average treatment effect on the treated (ATT), whereas instrumental variable approaches yield estimates of a local average treatment effect (LATE).
The goal of this article was to use simulation methods to illustrate the treatment-effect concepts that are identified using observational data with various estimators.
We simulated patient treatment choices based on expected treatment valuation to observe estimates of both ATT and LATE. Different model scenarios were run to isolate the effects of both treatment-effect heterogeneity and unmeasured confounding on treatment-effect concept estimation. Models were estimated using standard linear and nonlinear estimation methods.
We show that the true values of the underlying treatment concepts differ if patients (with the help of their health care providers) make treatment choices based on expected gains, and that distinct estimators produce estimates of distinct concepts. In scenarios without unmeasured confounding, both linear and nonlinear estimation models produced estimates close to the true value of the concept identified by each estimator. However, nonlinear models suggested additional treatment-effect heterogeneity that does not exist in these scenarios.
Our results suggest that, to ensure clarity and correctness of treatment-effect estimate interpretation, it is important for researchers to state the treatment-effect concept that they are trying to identify before beginning estimation. In addition, theoretical models of treatment choice are needed to provide the foundation linking treatment-effect estimates to treatment-effect concepts and to justify instrument selection.

  • Clinical Therapeutics 08/2009; 31(8):1858-1858. DOI:10.1016/j.clinthera.2009.08.013 · 2.59 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: OBJECTIVE: To assess the covariate balancing properties of propensity score-based algorithms in which covariates affecting treatment choice are both measured and unmeasured. DATA SOURCES/STUDY SETTING: A simulation model of treatment choice and outcome. STUDY DESIGN: Simulation. DATA COLLECTION/EXTRACTION METHODS: Eight simulation scenarios varied with the values placed on measured and unmeasured covariates and the strength of the relationships between the measured and unmeasured covariates. The balance of both measured and unmeasured covariates was compared across patients either grouped or reweighted by propensity scores methods. PRINCIPAL FINDINGS: Propensity score algorithms require unmeasured covariate variation that is unrelated to measured covariates, and they exacerbate the imbalance in this variation between treated and untreated patients relative to the full unweighted sample. CONCLUSIONS: The balance of measured covariates between treated and untreated patients has opposite implications for unmeasured covariates in randomized and observational studies. Measured covariate balance between treated and untreated patients in randomized studies reinforces the notion that all covariates are balanced. In contrast, forced balance of measured covariates using propensity score methods in observational studies exacerbates the imbalance in the independent portion of the variation in the unmeasured covariates, which can be likened to squeezing a balloon. If the unmeasured covariates affecting treatment choice are confounders, propensity score methods can exacerbate the bias in treatment effect estimates.
    Health Services Research 12/2012; 48(4). DOI:10.1111/1475-6773.12020 · 2.49 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Discuss the tradeoffs inherent in choosing a local area size when using a measure of local area practice style as an instrument in instrumental variable estimation when assessing treatment effectiveness. Assess the effectiveness of angiotensin converting-enzyme inhibitors and angiotensin receptor blockers on survival after acute myocardial infarction for Medicare beneficiaries using practice style instruments based on different-sized local areas around patients. We contrasted treatment effect estimates using different local area sizes in terms of the strength of the relationship between local area practice styles and individual patient treatment choices; and indirect assessments of the assumption violations. Using smaller local areas to measure practice styles exploits more treatment variation and results in smaller standard errors. However, if treatment effects are heterogeneous, the use of smaller local areas may increase the risk that local practice style measures are dominated by differences in average treatment effectiveness across areas and bias results toward greater effectiveness. Local area practice style measures can be useful instruments in instrumental variable analysis, but the use of smaller local area sizes to generate greater treatment variation may result in treatment effect estimates that are biased toward higher effectiveness. Assessment of whether ecological bias can be mitigated by changing local area size requires the use of outside data sources.
    Journal of clinical epidemiology 08/2013; 66(8 Suppl):S69-83. DOI:10.1016/j.jclinepi.2013.04.008 · 5.48 Impact Factor