Content uploaded by Frank Potter
Author content
All content in this area was uploaded by Frank Potter on Sep 02, 2014
Content may be subject to copyright.
A preview of the PDF is not available
A study of procedures to identify and trim extreme sampling weights
... where V w represents the weighted estimate's variance, V μ the unweighted and s 2 the variance of the weights, assumed to be scaled to average unity [3,15]. To lower the amount of additional variance, the weights are often modified using trimming [16], collapsing weight strata [17,18] or shrinking [19,20]. The amount of increase in variance is often expressed as the efficiency of a weighting procedure, with higher efficiency standing for a lower increase in variance after weighting. ...
... A common disadvantage of weighting is an increase in the variance of the estimator [14] and in line with this an increase in the standard errors in conducted analyses [11]. We trimmed the weights at the 0.5 and 99.5 percentiles to reduce this variance [16] and observed an increase in estimator variance (unweighted vs. longitudinal weights) between 5% (age) and 35% (BMI) and a final weighting efficiency of 41.7%. At 41.7%, our final weighting efficiency is only modest but acceptable. ...
Background
The bias caused by drop-out is an important factor in large population-based epidemiological studies. Many studies account for it by weighting their longitudinal data, but to date there is no detailed final approach for how to conduct these weights.
Methods
In this study we describe the observed longitudinal bias and a three-step longitudinal weighting approach used for the longitudinal data in the MoMo baseline (N = 4528, 4–17 years) and wave 1 study with 2807 (62%) participants between 2003 and 2012.
Results
The most meaningful drop-out predictors were socioeconomic status of the household, socioeconomic characteristics of the mother and daily TV usage. Weighting reduced the bias between the longitudinal participants and the baseline sample, and also increased variance by 5% to 35% with a final weighting efficiency of 41.67%.
Conclusions
We conclude that a weighting procedure is important to reduce longitudinal bias in health-oriented epidemiological studies and suggest identifying the most influencing variables in the first step, then use logistic regression modeling to calculate the inverse of the probability of participation in the second step, and finally trim and standardize the weights in the third step.
... Weighting is an important step in preparing survey data for analysis. Most papers that discuss weighting focus on rules concerning the number of observations in a stratum and when and how to aggregate weighting cells (Potter, 1988, Botman et al., 2000, Karlton and Flores-Cervantes, 2003, and how to deal with extreme weights (Chowdhury et al., 2007, Battaglia et al., 2006, Battaglia et al., 2009, usually by trimming, due to the effect of extreme weights on the mean square error (MSE) of estimates (Potter, 1990). Few papers discuss how decisions on weighting affect the survey estimates and associated MSEs. ...
Introduction: The calculation and application of weights is an important step in producing estimates from a sample survey. Practical application of survey weighting requires a number of decisions on how to handle issues, such as extreme weights and small cell sample sizes. Generally, discussion centres around the extreme weights themselves, rather than the impact of the extreme weights on results. This paper aims to identify diagnostics that may assist in making decisions about aggregation of small cells and trimming of extreme weights, partly by assessing impacts on results. It uses a case study where there is high variability in the survey weights due to the survey design. The 2017-18 New South Wales (NSW) Emergency Department survey covers 82 hospitals, ranging from tertiary teaching hospitals to small local hospitals. Unbiased estimates are required for each hospital, as well as for 16 administrative areas (Local Health Districts (LHD)) and for NSW. Quarterly data are used to report key performance indicators at the LHD level. Method: Several options for modifying weights were applied to data from the New South Wales (NSW) Emergency Department (ED) survey. We introduce a novel diagnostic we call the deviation sum of squares-the sum of squares of deviations of the sum of weights vs the population. As well as looking at the deviation sum of squares and other information regarding the distribution of the weights, we considered the impact of the different weights on the estimates of the proportion in the most positive response option of six questions, and on the score-based performance indices reported quarterly to the NSW Health system. Results: Although weights need to be applied at the level of the stratification, the actual mechanism used to create the weight had a minimal effect on results for key variables at the three reporting levels (hospital, LHD, NSW) and performance indices at LHD level. The deviation sum of squares, together with the maximum weight can assist in informing decisions about aggregation of cells and trimming of weights. Conclusion: Simple diagnostics of the weight distribution, maximum weight and how closely the weight totals agree with the population by stratum can be used to assist in the decision-making about how to modify weights. Results for the NSW ED survey appear reasonably robust to the method used to create the weights, provided the weights are created at the stratum level, and have a low deviation sum of squares.
... As is often the case with weighting, the resulting weights inflated the variance in the data, as measured by the ratio of the highest weight to the lowest weight and by the design effect due to weighting. 12 This is commonly addressed by weight trimming, 13 a method in which extreme weights, on the low and high ends, 14 are identified and the weight distribution is truncated to reduce error in the estimates stemming from the increase in variance. ...
Objective:
To describe survey methods used to examine reported experiences of discrimination against African Americans, Latinos, Asian Americans, Native Americans, women, and LGBTQ (lesbian, gay, bisexual, transgender, and queer) adults.
Data source and study design:
Data came from a nationally representative, probability-based telephone survey of 3453 US adults, conducted January-April 2017.
Methods:
We examined the survey instrument, sampling design, and weighting of the survey, and present selected survey findings.
Principal findings:
Examining reported discrimination experienced by multiple groups in a telephone survey requires attention to details of sampling and weighting. In health care settings, 32 percent of African Americans reported discrimination, as did 23 percent of Native Americans, 20 percent of Latinos, 18 percent of women, 16 percent of LGBTQ adults, and 13 percent of Asian Americans. Also, 51 percent of LGBTQ adults, 42 percent of African Americans, and 38 percent of Native Americans reported identity-based violence against themselves or family members; 57 percent of African Americans and 41 percent of women reported discrimination in pay or promotions; 50 percent of African Americans, 29 percent of Native Americans, and 27 percent of Latinos reported being discriminated against in interactions with police.
Conclusions:
Even the small selection of results presented in this article as examples of survey measures show a pattern of substantial reported discrimination against all six groups studied.
... (2) an adjustment to account for unit nonresponse; (3) a poststratification adjustment to account for incomplete coverage of the target population (e.g., Brick and Kalton 1996) using the most recent, reliable estimates of the number of residential households available from the American Community Survey (U.S. Census Bureau 2015) as population controls; and (4) use of an established procedure for trimming the estimated mean square error (see Potter 1990) to minimize the effects of extreme weights on the sampling variance. The variance of the fishing effort estimates was calculated using Taylor series linearization (Dienes 1957;SAS Institute 2016). ...
Many fisheries monitoring programs use self‐administered surveys to collect data, which are subject to recall error. Recall error occurs when respondents inaccurately remember past events due to telescoping (remembering events more recently or further back in time than they occurred) or omission error (forgetting events altogether). Previous research on the effects of variable reference periods in fisheries surveys has been inconclusive due to difficulty in disentangling method effects from recall error and in determining whether estimates from shorter recall periods are less biased, or more subject to telescoping. The National Marine Fisheries Service has developed a new household mail survey, the Fishing Effort Survey, where anglers are asked to recall cumulative fishing effort over the past two months from which estimates of saltwater fishing effort are produced. Here, we examined how the length of the reference period may affect the Fishing Effort Survey in four U.S. states by comparing effort estimates to two feasible alternatives; 1) a survey administered monthly with both a one‐ and two‐month reference period (where respondents were asked to recall fishing effort for each of the past two months individually), and 2) a survey administered monthly with a one‐month reference period. To further explore bias in the designs, we compared total effort, fishing prevalence and mean trips per household estimates derived from the two experimental surveys. We found no significant differences between the Fishing Effort Survey and experimental survey estimates. However, we found evidence that multiple reference periods in a single survey may reduce bias for one‐month estimates. Increased understanding of techniques that can reduce recall bias, and of the trade‐offs of shorter or longer reference periods will ultimately help fisheries survey designers more accurately weigh bias against survey costs, and improve the quality of data used to inform management decisions.
... information-systems/wesvarw-support), the calibrate() function in the R survey package (Lumley 2010), the ipfraking user-written package in Stata (Kolenikov 2014), the sreweight user-written command in Stata (Pacifico 2014), and the CALMAR 2 software developed by Le Guennec and Sautory (2002) are also capable of computing calibration adjustments to design weights based on the methods described above, given population information on the chosen auxiliary variables (see http:// vesselinov.com/CalmarEngDoc.pdf for more details on the various calibration options in the CALMAR 2 software). The final calibrated weights may then be trimmed to minimize the impact of weight variance on the precision of weighted survey estimates (Potter 1990;Elliott and Little 2000;Kalton and Flores-Cervantes 2003;Beaumont 2008; see also Asparouhov and Muthén 2007 for optimal weight trimming approaches using the Mplus software). The weights that result from this process then need to be input by analysts into software procedures enabling design-unbiased point estimation of population parameters (see Subsection 2.4). ...
In this article, we review current state-of-the art software enabling statisticians to apply design-based, model-based, and so-called “hybrid” approaches to the analysis of complex sample survey data. We present brief overviews of the similarities and differences between these alternative approaches, and then focus on software tools that are presently available for implementing each approach. We conclude with a summary of directions for future software development in this area.
... It is unclear whether and how to incorporate auxiliary information (Groves and Couper, 1995). Discussion of smoothing and trimming in the survey weighting literature (e.g.Potter, 1988Potter, , 1990Elliott and Little, 2000;Elliott, 2007;Xia and Elliott, 2016) has focused on estimating the finite population total or mean, with less attention to subdomain estimates.Beaumont (2008)proposes to smooth weights by predicting and regressing these on the survey variables, where the direction is inspiring but tangential to the inference objective. Borrowing information on survey outcomes potentially increases efficiency and calls for a general framework. ...
We combine Bayesian prediction and weighted inference as a unified approach to survey inference. The general principles of Bayesian analysis imply that models for survey outcomes should be conditional on all variables that affect the probability of inclusion. We incorporate the weighting variables under the framework of multilevel regression and poststratification, as a byproduct generating model-based weights after smoothing. We investigate deep interactions and introduce structured prior distributions for smoothing and stability of estimates. The computation is done via Stan and implemented in the open source R package "rstanarm" ready for public use. Simulation studies illustrate that model-based prediction and weighting inference outperform classical weighting. We apply the proposal to the New York Longitudinal Study of Wellbeing. The new approach generates robust weights and increases efficiency for finite population inference, especially for subsets of the population.
One of the important features of survey data is the availability of auxiliary information from the survey population. Auxiliary population information may be available from census enumerations or administrative records. It could also be obtained from unusual sources such as satellite images for natural resource inventory surveys or a previous survey from the same population. Auxiliary information can be utilized at the survey design stage to facilitate frame construction, stratification, clustering and unequal probability selection of units. Calibration methods address issues related to the use of auxiliary information at the estimation stage of survey data analysis.
The last step in weighting, which is extremely important in many surveys, is to use auxiliary data to correct coverage problems and to reduce standard errors. By auxiliary data, we mean information that is available for the entire frame or target population, either for each individual population unit or in aggregate form. This chapter describes the general method of calibration estimation, including poststratification, raking, and general regression estimation. The software packages used for examples are the R survey package, SUDAAN, and in Stata. The steps for computing base weights, nonresponse adjustments, and calibration may result in weights whose sizes vary quite a bit. Quadratic programming and weight-trimming methods are also covered.
Depression is the most common metal disorder linked to media use. Theoretically, the relationship between depression and media use has been conceptualized as a linear function. However, depressive symptoms vary from dysphoric moods to severely depressed states with major social impairment, thus providing a strong alternative rationale for a non-linear relationship. This paper reports on findings from a representative telephone survey of the general German population (N = 2002) including both the respondents’ motivation behind spending time using traditional media and a measure to screen for depression in the general population. The curve-fitting methodology revealed that the associations between depression and media use are described by a cubic growth function for newspapers, the radio, magazines, and books; associations with television use were positive, but more complex. The relationship between depression and media use should be modeled as a polynomial function for more accurate estimations in the future.
Probability weights are used in many areas of research including complex survey designs, missing data analysis, and adjustment for confounding factors. They are useful analytic tools but can lead to statistical inefficiencies when they contain outlying values. This issue is frequently tackled by replacing large weights with smaller ones or by normalizing them through smoothing functions. While these approaches are practical, they are also prone to yield biased inferences. This paper introduces a method for obtaining optimal weights, defined as those with smallest Euclidean distance from target weights among all sets of weights that satisfy a constraint on the variance of the resulting weighted estimator. The optimal weights yield minimum-bias estimators among all estimators with specified precision. The method is based on solving a constrained nonlinear optimization problem whose Lagrange multipliers and objective function can help assess the trade-off between bias and precision of the resulting weighted estimator. The finite-sample performance of the optimally weighted estimator is assessed in a simulation study, and its applicability is illustrated through an analysis of heterogeneity over age of the effect of the timing of treatment-initiation on long-term treatment efficacy in patient infected by human immunodeficiency virus in Sweden.
The Current Population Survey Design and Report 40
- R H Hanson
Hanson, R.H. (1978), The Current Population Survey Design and Report 40, Washington D.C. Bureau of the Census
Procedures in Implementing the New Design, in The NAEP 1983-1984 Technical Report Survey of Procedures to Control Extreme Sampling Weights
- E G Johnson
Johnson, E.G. et al. Procedures in Implementing the New Design, in The NAEP 1983-1984 Technical Report, ed. A. E. Beaton, Princeton, N.J.: of Educational Progress, pp. 493-504. Potter, F. (1988), Survey of Procedures to Control Extreme Sampling Weights, Proceedings of the Section on Survey Research Methods, American Statistical Association, pp. 453-458. U.S. Department of Health and Human Services (1987), The Area Resource File (ARF) System, NTIS Accession No. HRP-0907022, Washington, D.C. : Health Resources Administration