A novel framework for validating and applying standardized small area measurement strategies

Institute for Health Metrics and Evaluation, University of Washington, 2301 5th Ave, Suite 600, Seattle, WA 98121, USA. .
Population Health Metrics (Impact Factor: 2.11). 09/2010; 8:26. DOI: 10.1186/1478-7954-8-26
Source: PubMed

ABSTRACT Local measurements of health behaviors, diseases, and use of health services are critical inputs into local, state, and national decision-making. Small area measurement methods can deliver more precise and accurate local-level information than direct estimates from surveys or administrative records, where sample sizes are often too small to yield acceptable standard errors. However, small area measurement requires careful validation using approaches other than conventional statistical methods such as in-sample or cross-validation methods because they do not solve the problem of validating estimates in data-sparse domains.
A new general framework for small area estimation and validation is developed and applied to estimate Type 2 diabetes prevalence in US counties using data from the Behavioral Risk Factor Surveillance System (BRFSS). The framework combines the three conventional approaches to small area measurement: (1) pooling data across time by combining multiple survey years; (2) exploiting spatial correlation by including a spatial component; and (3) utilizing structured relationships between the outcome variable and domain-specific covariates to define four increasingly complex model types - coined the Naive, Geospatial, Covariate, and Full models. The validation framework uses direct estimates of prevalence in large domains as the gold standard and compares model estimates against it using (i) all available observations for the large domains and (ii) systematically reduced sample sizes obtained through random sampling with replacement. At each sampling level, the model is rerun repeatedly, and the validity of the model estimates from the four model types is then determined by calculating the (average) concordance correlation coefficient (CCC) and (average) root mean squared error (RMSE) against the gold standard. The CCC is closely related to the intraclass correlation coefficient and can be used when the units are organized in groups and when it is of interest to measure the agreement between units in the same group (e.g., counties). The RMSE is often used to measure the differences between values predicted by a model or an estimator and the actually observed values. It is a useful measure to capture the precision of the model or estimator.
All model types have substantially higher CCC and lower RMSE than the direct, single-year BRFSS estimates. In addition, the inclusion of relevant domain-specific covariates generally improves predictive validity, especially at small sample sizes, and their leverage can be equivalent to a five- to tenfold increase in sample size.
Small area estimation of important health outcomes and risk factors can be improved using a systematic modeling and validation framework, which consistently outperformed single-year direct survey estimates and demonstrated the potential leverage of including relevant domain-specific covariates compared to pure measurement models. The proposed validation strategy can be applied to other disease outcomes and risk factors in the US as well as to resource-scarce situations, including low-income countries. These estimates are needed by public health officials to identify at-risk groups, to design targeted prevention and intervention programs, and to monitor and evaluate results over time.

Download full-text


Available from: Ali Mokdad, Aug 11, 2015
1 Follower
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: For 18 months in 2009-2010, the Rockefeller Foundation provided support to establish the Roundtable on Urban Living Environment Research (RULER). Composed of leading experts in population health measurement from a variety of disciplines, sectors, and continents, RULER met for the purpose of reviewing existing methods of measurement for urban health in the context of recent reports from UN agencies on health inequities in urban settings. The audience for this report was identified as international, national, and local governing bodies; civil society; and donor agencies. The goal of the report was to identify gaps in measurement that must be filled in order to assess and evaluate population health in urban settings, especially in informal settlements (or slums) in low- and middle-income countries. Care must be taken to integrate recommendations with existing platforms (e.g., Health Metrics Network, the Institute for Health Metrics and Evaluation) that could incorporate, mature, and sustain efforts to address these gaps and promote effective data for healthy urban management. RULER noted that these existing platforms focus primarily on health outcomes and systems, mainly at the national level. Although substantial reviews of health outcomes and health service measures had been conducted elsewhere, such reviews covered these in an aggregate and perhaps misleading way. For example, some spatial aspects of health inequities, such as those pointed to in the 2008 report from the WHO's Commission on the Social Determinants of Health, received limited attention. If RULER were to focus on health inequities in the urban environment, access to disaggregated data was a priority. RULER observed that some urban health metrics were already available, if not always appreciated and utilized in ongoing efforts (e.g., census data with granular data on households, water, and sanitation but with little attention paid to the spatial dimensions of these data). Other less obvious elements had not exploited the gains realized in spatial measurement technology and techniques (e.g., defining geographic and social urban informal settlement boundaries, classification of population-based amenities and hazards, and innovative spatial measurement of local governance for health). In summary, the RULER team identified three major areas for enhancing measurement to motivate action for urban health-namely, disaggregation of geographic areas for intra-urban risk assessment and action, measures for both social environment and governance, and measures for a better understanding of the implications of the physical (e.g., climate) and built environment for health. The challenge of addressing these elements in resource-poor settings was acknowledged, as was the intensely political nature of urban health metrics. The RULER team went further to identify existing global health metrics structures that could serve as platforms for more granular metrics specific for urban settings.
    Journal of Urban Health 09/2011; 88(5):793-857. DOI:10.1007/s11524-011-9613-2 · 1.94 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: Existing data suggest that influenza vaccination rates among adults in the United States fall far short of the Healthy People 2010 goals and the updated Healthy People 2020 targets. We identified characteristics associated with influenza vaccination that might inform strategies for increasing coverage. We used data from the 2009 Behavioral Risk Factor Surveillance System to estimate adjusted prevalence ratios for receiving the influenza vaccine in the past 12 months. Among 134,101 adults aged ≥65 years, the influenza vaccination coverage level was 68.9%. Among 286,867 younger adults aged 18-64 years, coverage was markedly lower: 31.8%. Having health care coverage was the strongest predictor of vaccination in both age groups, after accounting for other sociodemographic characteristics, health behaviors, and health status. Those reporting older age, white race, higher education, non-smoking status, being physically active, or having poor physical health or a personal history of various chronic conditions were also more likely to report having received the influenza vaccine. Our results show clearly that vaccine uptake in the United States is related to social position as well as other health behaviors. These findings call for renewed attention to vaccination strategies to meet the updated Healthy People 2020 goals.
    Preventive Medicine 03/2012; 54(5):358-62. DOI:10.1016/j.ypmed.2012.03.008 · 2.93 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Between 1990 and 2010, the U.S ranking in neonatal mortality slipped from 29(th) to 45(th) among countries globally. Substantial subnational variation in newborn mortality also exists. Our objective is to measure the extent to which trends and subnational variation in early neonatal mortality reflect differences in the prevalence of risk factors (gestational age and birth weight) compared to differences in clinical care. Observational study using linked birth and death data for all births in the United States between 1996 and 2006. We examined health service area (HSA) level variation in the expected early neonatal mortality rate, based on gestational age (GA) and birth-weight (BW), and GA-BW adjusted mortality as a proxy for clinical care. We analyzed the relationship between selected health system indicators and GA-BW-adjusted mortality. The early neonatal death (ENND) rate declined 12% between 1996 and 2006 (2.39 to 2.10 per 1000 live births). This occurred despite increases in risk factor prevalence. There was significant HSA-level variation in the expected ENND rate (Rate Ratio: 0.73-1.47) and the GA-BW adjusted rate (Rate ratio: 0.63-1.68). Accounting for preterm volume (defined as <34 weeks), the number of neonatologist and NICU beds, 25.2% and 58.7% of the HSA-level variance in outcomes was explained among all births and very low birth weight babies, respectively. Improvements in mortality could be realized through the expansion or reallocation of clinical neonatal resources, particularly in HSAs with a high volume of preterm births; however, prevention of preterm births and low-birth weight babies has a greater potential to improve newborn survival in the United States.
    PLoS ONE 11/2012; 7(11):e49399. DOI:10.1371/journal.pone.0049399 · 3.53 Impact Factor
Show more