ArticlePublisher preview available

Quantifying the Likelihood of False Positives: Using Sensitivity Analysis to Bound Statistical Inference

DOI:10.1007/s10940-018-9385-x
Authors:
Kyle J. Thomas
Kyle J. Thomas
Kyle J. Thomas
  • This person is not on ResearchGate, or hasn't claimed this research yet.
Jean Marie McGloin
Jean Marie McGloin
Jean Marie McGloin
  • This person is not on ResearchGate, or hasn't claimed this research yet.
Christopher J. Sullivan at University of Cincinnati
Christopher J. Sullivan
Read publisher preview
Download citation
To read the full-text of this research, you can request a copy directly from the authors.

Abstract and Figures

Objective Criminologists have long questioned how fragile our statistical inferences are to unobserved bias when testing criminological theories. This study demonstrates that sensitivity analyses offer a statistical approach to help assess such concerns with two empirical examples—delinquent peer influence and school commitment. Methods Data from the Gang Resistance Education and Training are used with models that: (1) account for theoretically-relevant controls; (2) incorporate lagged dependent variables and; (3) account for fixed-effects. We use generalized sensitivity analysis (Harada in ISA: Stata module to perform Imbens’ (2003) sensitivity analysis, 2012; Imbens in Am Econ Rev 93(2):126–132, 2003) to estimate the size of unobserved heterogeneity necessary to render delinquent peer influence and school commitment statistically non-significant and substantively weak and compare these estimates to covariates in order to gauge the likely existence of such bias. ResultsUnobserved bias would need to be unreasonably large to render the peer effect statistically non-significant for violence and substance use, though less so to reduce it to a weak effect. The observed effect of school commitment on delinquency is much more fragile to unobserved heterogeneity. Conclusion Questions over the sensitivity of inferences plague criminology. This paper demonstrates the utility of sensitivity analysis for criminological theory testing in determining the robustness of estimated effects.
Figures - available from: Journal of Quantitative Criminology
This content is subject to copyright. Terms and conditions apply.
  • A preview of this full-text is provided by Springer Nature.
  • Learn more
Preview content only
Content available from Journal of Quantitative Criminology
This content is subject to copyright. Terms and conditions apply.
Vol.:(0123456789)
Journal of Quantitative Criminology (2019) 35:631–662
https://doi.org/10.1007/s10940-018-9385-x
1 3
ORIGINAL PAPER
Quantifying theLikelihood ofFalse Positives: Using
Sensitivity Analysis toBound Statistical Inference
KyleJ.Thomas1· JeanMarieMcGloin2· ChristopherJ.Sullivan3
Published online: 22 June 2018
© Springer Science+Business Media, LLC, part of Springer Nature 2018
Abstract
Objective Criminologists have long questioned how fragile our statistical inferences are to
unobserved bias when testing criminological theories. This study demonstrates that sensi-
tivity analyses offer a statistical approach to help assess such concerns with two empirical
examples—delinquent peer influence and school commitment.
Methods Data from the Gang Resistance Education and Training are used with models
that: (1) account for theoretically-relevant controls; (2) incorporate lagged dependent vari-
ables and; (3) account for fixed-effects. We use generalized sensitivity analysis (Harada in
ISA: Stata module to perform Imbens’ (2003) sensitivity analysis, 2012; Imbens in Am
Econ Rev 93(2):126–132, 2003) to estimate the size of unobserved heterogeneity neces-
sary to render delinquent peer influence and school commitment statistically non-signifi-
cant and substantively weak and compare these estimates to covariates in order to gauge
the likely existence of such bias.
Results Unobserved bias would need to be unreasonably large to render the peer effect
statistically non-significant for violence and substance use, though less so to reduce it to a
weak effect. The observed effect of school commitment on delinquency is much more frag-
ile to unobserved heterogeneity.
Conclusion Questions over the sensitivity of inferences plague criminology. This paper
demonstrates the utility of sensitivity analysis for criminological theory testing in deter-
mining the robustness of estimated effects.
Keywords Sensitivity analysis· False positives· Unobserved bias· Theory testing
“A fragile inference is not worth taking seriously.” Leamer (1985)
* Kyle J. Thomas
thomaskj@umsl.edu
1 University ofMissouri-St. Louis, 1 University Blvd, 331 Lucas Hall, St.Louis, MO63121, USA
2 University ofMaryland, CollegePark, MD, USA
3 University ofCincinnati, Cincinnati, OH, USA
Content courtesy of Springer Nature, terms of use apply. Rights reserved.
... Low self-control Gottfredson and Hirschi (1990) view propensity for delinquent conduct as a direct function of one's self-control. The G.R.E.A.T. data contains eight items that are a subset of the Grasmick, Tittle, Bursik, and Arneklev (1993) self-control scale (see also Thomas, McGloin, & Sullivan, 2018). On a Likert-scale ranging from 1 ¼ strongly disagree to 5 ¼ strongly agree, respondents are asked how much they agree with the following statements: "I often act on the spur of the moment," "I do what brings me pleasure here and now," "I don't devote a lot of thought and effort preparing for the future," "I am more concerned with what happens to me in the short run than in the long run," "I like to test myself every now and then by doing something a little risky," "Sometimes I will take a risk just for the fun of it," "I sometimes find it exciting to do things for which I might get in trouble," and "excitement and adventure are more important to me than security." ...
... Individuals were told to think about their mother or mother-figure and asked to rate the following statements: "can talk about anything," "always trusts me," "knows all my friends," "always understands me," "always ask her for advice," and "always praises me when I do well." Responses ranged from 1 to 7, and higher values are indicative of higher levels of attachment (see Thomas et al., 2018). The maternal attachment items have high interitem reliability across all waves (Wave 1: a ¼ .826; ...
... We measure school commitment using four Likert-scale items that ask respondents how much they agree with the statements: "I try hard in school," "Education is so important that it's worth it to put up with things about school that I don't like," "In general, I like school," and "I usually finish my homework." Response options ranged from 1 ¼ strongly disagree to 5 ¼ strongly agree, with higher values reflecting higher levels of commitment to school (see Thomas et al., 2018). The items demonstrate high inter-item reliability across all waves (Wave 1: a ¼ .735; ...
Affinity, Affiliation, and Guilt: Examining Between- and Within-Person Variability in Delinquent Peer Influence
Article
  • Jul 2019
A longstanding debate in criminology concerns whether peers influence delinquency or if they are of no significance because criminal propensity develops independent of associations. Matza (1969 Matza, D. (1969). Becoming deviant. New Brunswick: Transaction. [Google Scholar]) criticized these competing perspectives, suggesting instead that the peer–delinquency relationship is nuanced, such that there is both between- and within-person variability in the influence of friends. Drawing on his choice-based perspective, we hypothesize that peers have a stronger influence on the delinquent tendencies of individuals low in criminal propensity. Further, we predict that the influence of peers varies within-individuals across crime types, such that delinquent friends are less influential for behaviors that individuals anticipate more guilt. Using data from the Gang Resistance Education and Training (G.R.E.A.T.) evaluation, we find support for these hypotheses. Perceived peer delinquency is a stronger predictor of offending for those lower in criminal propensity. Further, individuals are less susceptible to delinquent peers for acts in which they anticipate higher levels of guilt.
... Fixes to such violations are well-known and some more advanced fixes are presented in this issue (e.g. Moody and Marvell 2020;Thomas et al. 2019). Some less well-known model assumptions are equally important. ...
Standard Errors in Quantitative Criminology: Taking Stock and Looking Forward
Article
  • May 2020
... Table 4 provides a series of suggestive steps that can be applied during the process of defining peer influence and in determining the best analytic approach to use to address these questions (for additional reading on the importance of definitions of peer influence see Kindermann & Skinner, 2019). We also suggest that scholars carefully consider recent work indicating that conventional statistical tools do not overestimate influence effects compared with network approaches (Ragan et al., 2019) and influence effects attained from cross-lagged models are unlikely to be false positives derived from unobserved variable biases (Thomas et al., 2019). ...
A Comparison of Dyadic and Social Network Assessments of Peer Influence
Article
Full-text available
  • Mar 2021
The present study compares two methods for assessing peer influence: the longitudinal actor–partner interdependence model (L-APIM) and the longitudinal social network analysis (L-SNA) Model. The data were drawn from 1,995 (49% girls and 51% boys) third grade students ( M age = 9.68 years). From this sample, L-APIM ( n = 206 indistinguishable dyads and n = 187 distinguishable dyads) and L-SNA ( n = 1,024 total network members) subsamples were created. Students completed peer nominations and objective assessments of mathematical reasoning in the spring of the third and fourth grades. Patterns of statistical significance differed across analyses. Stable distinguishable and indistinguishable L-APIM dyadic analyses identified reciprocated friend influence such that friends with similar levels of mathematical reasoning influenced one another and friends with higher math reasoning influenced friends with lower math reasoning. L-SNA models with an influence parameter (i.e., average reciprocated alter) comparable to that assessed in L-APIM analyses failed to detect influence effects. Influence effects did emerge, however, with the addition of another, different social network influence parameter (i.e., average alter influence effect). The diverging results may be attributed to differences in the sensitivity of the analyses, their ability to account for structural confounds with selection and influence, the samples included in the analyses, and the relative strength of influence in reciprocated best as opposed to other friendships.
Reflected appraisals across multiple reference groups: Discrepancies in self and individual delinquency
Article
We explore the implications of multiple reference groups for symbolic interaction explanations of delinquency. Specifically, we test the idea that dissimilarity in reflected appraisals across reference groups weakens the effect of the appraisals of a single group on behavior. We also assess whether the relative influence of different reflected appraisals is moderated by differences in the amount of time spent in and attachment to the roles taken among specific reference groups. Using data from the National Youth Survey, our findings mostly support symbolic interactionist predictions: Dissimilarity in reference groups' reflected appraisals weakens the influence of a single group's appraisals on behavior. We also found some evidence that, relatively, the more time an individual spends and the more attached they are to a role in a particular reference group, the more strongly correlated the respective appraisals are with delinquent behavior. Contrary to our predictions, we found that relatively strong attachments to one's role with their parents weakens the influence of parental reflected appraisals as a rule violator.
Peer Influence and Delinquency
Article
  • Jan 2019
Peer influence occupies an intriguing place in criminology. On the one hand, there is a long line of theorizing and empirical work highlighting it as a key causal process for delinquency. On the other, there is a group of theoretical skeptics who view it as one of the most notorious examples of a spurious link. After discussing these perspectives, this review takes stock of our intellectual advancements in understanding peer influence over decades' worth of research toward this endeavor. We conclude that although there have been important gains, essential questions and gaps remain. Toward this aim, we offer some lines of future work that we believe offer pathways to yielding the greatest added value to the discipline.
Working with Misspecified Regression Models
Article
Full-text available
Objectives Conventional statistical modeling in criminology assumes proper model specification. Very strong and unrebutted criticisms have existed for decades. Some respond that although the criticisms are correct, there is for observational data no alternative. In this paper, we provide an alternative. Methods We draw on work in econometrics and statistics from several decades ago, updated with the most recent thinking to provide a way to properly work with misspecified models. ResultsWe show how asymptotically, unbiased regression estimates can be obtained along with valid standard errors. Conventional statistical inference can follow. Conclusions If one is prepared to work with explicit approximations of a “true” model, defensible analyses can be obtained. The alternative is working with models about which all of the usual criticisms hold.
Institutional Placement and Illegal Earnings: Examining the Crime School Hypothesis
Article
Full-text available
Objectives The objective of the current study is to examine the hypothesis that correctional environments can facilitate the accumulation of “criminal capital” and might actually encourage offending by serving as a school of crime. Methods We use panel data from a sample of 615 serious juvenile offenders who reported illegal earnings and information regarding institutional stays over a 7 year period. There are two separate measures that can contribute to criminal capital within institutions: the prevalence of friends in the institution who have committed income generating crimes and the length of institutional stays as a cumulative dosage. We account for unobserved heterogeneity using fixed effects estimation. Results Both institutional measures, total number of days incarcerated and exposure to deviant peers, predict a positive increase in an individual’s daily illegal wage rate, even after eliminating fixed unobserved heterogeneity and controlling for important time-varying confounders. For total number of days institutionalized, there were positive though marginally declining returns to length of stay. Conclusions For juveniles, placement in institutions can have important negative consequences. Our findings extend and complement findings showing that institutionalization might help increase opportunities in the illegal market. This not only undermines prison as a crime-prevention strategy but could also promote longer and more successful criminal careers.
Social Learning and Social Structure: A General Theory of Crime and Deviance
Book
  • Jul 2017
Case studies and theory development in the social sciences
Book
  • Jan 2005
INCENTIVES FOR COLLECTIVE DEVIANCE: GROUP SIZE AND CHANGES IN PERCEIVED RISK, COST, AND REWARD*: INCENTIVES FOR COLLECTIVE DEVIANCE
Article
Research has demonstrated that the presence of others shifts decision-making about risky/deviant behavior. One reason for this shift could be changes in the anticipated experience of formal sanctions, informal costs, and rewards. To investigate this possibility, this study conducted two randomized controlled trials with hypothetical vignettes, in which a range of how many other people were also involved in the criminal act defined the treatment conditions. Across two samples of university students (Ns = 396 and 263), the results revealed that as the size of the involved group increased, the anticipated experience of sanction risk and several informal social costs associated with engaging in the act decreased, and the anticipated experience of two rewards increased. Additional analyses suggest that, with one exception in each data set, these changes are not only tied to the solo/group distinction.
Considering the Elements that Inform Perceived Peer Deviance
Article
Objectives To consider, at a conceptual level, the factors that inform perceptions of peer deviance and subsequently, at an empirical level, the extent to which survey information from high school students confirms whether these elements shape perceptions of friends’ drinking. This study also offers an alternative way to document projection bias. Methods 249 public high school students completed a survey about what factors inform their perceptions of friends’ drinking behavior. Subjects also responded to several vignettes in order to assess their general tendency to engage in projection. Results Subjects rely on both observed behavior and various forms of communication when forming perceptions of friends’ drinking, though there is notable variation across these elements. When using hypothetical vignettes, results suggest projection bias is significantly diminished as subjects are provided with more information about a hypothetical peer. Conclusions Adolescents appear to rely on a wide range of information when forming perceptions about friends’ drinking behavior. Although we did document a tendency to engage in projection when subjects had minimal information about a peer, the fragility of this tendency questions whether perceptual measures are inherently contaminated.
The ASA's Statement on p-Values: Context, Process, and Purpose
Article
  • Mar 2016
Social mechanisms of community influences on crime and pathways in criminality
Article
  • Jan 2003
Can rational choice be considered a general theory of crime? Evidence from individual-level panel data
Article
In the last few decades, rational choice theory has emerged as a bedrock theory in the fields of economics, sociology, psychology, and political science. Although rational choice theory has been available to criminologists for many years now, the field has not embraced it as other disciplines have. Moreover, rational choice scholars have fueled this skepticism of the theory's generality by modeling offender decision making that is one-sided—large on the costs of crime (sanction threats), short on the benefits of crime. In this article, we directly assess the generality of rational choice theory by examining a fully specified model in a population that is often presumed to be less rational—adolescents from lower socioeconomic families who commit both instrumental (property) and expressive crimes (violence/drugs). By using a panel of N = 1,354 individuals, we find that offending behavior is consistent with rational responses to changes in the perceived costs and benefits of crime even after eliminating fixed unobserved heterogeneity and other time-varying confounders, and these results are robust across different subgroups. The findings support our argument that rational choice theory is a general theory of crime.
Experimental tests of criminological theory
Article
  • Jan 2012
Though randomized, controlled, experimental trials hardly dominate the landscape of research in criminology, we nonetheless find ourselves in the midst of what many scholars are calling the era of experimental criminology (Clear 2010; Sampson 2010). As Angrist (2006: 23) observes, “criminologists do not appear to have… ‘sciencephobia’…Criminology is the only one [social science] to show a marked increase in the use of randomized trials since the mid-sixties.” Not only has the reliance on randomized controlled trials (RCTs) to determine “what works” grown in previous years – though the number of such inquiries arguably remains tied to the productivity of a handful of scholars (Farrington 2003) – but the prominence and endorsement of this method has also grown markedly, both inside and outside of academic circles. To be clear, the view that randomized, controlled experimental methods provide important and rigorous methodological insight into empirical questions of interest to social scientists is hardly new. However, undeniably, the relevance and importance of this method for criminology specifically has gained increased momentum in recent years. The past decade and a half has witnessed the establishment of the Academy of Experimental Criminology, the Jerry Lee Centre of Experimental Criminology, the American Society of Criminology’s Division of Experimental Criminology, the Campbell Collaboration Crime and Justice Group, the Center for Evidence-Based Crime Policy, and the Journal of Experimental Criminology. These organizations (and the journal) have a clear agenda of focusing attention on the importance of experiments for social science generally and criminology in particular; they argue that successfully implemented RCTs are the “gold standard” by which we should assess the effectiveness of policy and interventions.