Jon Krosnick

Stanford University | SU

Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of methods or whether presumptively optimal methods are not, in fact, optimal. This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experimental f...

This chapter identifies useful directions for research addressing potentially problematic practices, the causes of such behaviors, and possible solutions. Problems in science in recent years are not limited to studies not replicating. Other pervasive problems that decrease the accuracy of scientific findings include, but are not limited to, errors...

The primary goal of science is to “get it right,” meaning that scientists seek to accurately document the world as it is. While erroneous conclusions and flawed theories can and do occur, they can only be tolerated as long as reliable mechanisms of self-correction exist. Unfortunately, in recent years, an array of evidence has emerged suggesting th...

According to many natural scientists, global warming has worsened wildfires and floods in the United States, causing increasingly devastating damage, and will do so in the future. Whether government takes action to better mitigate the risks and costs of these natural disasters may depend on public opinion. This study explored Americans’ preferences...

How do people form their attitudes toward complex policy issues? Although there has long been an assumption that people consider the various components of those issues and come to an overall assessment, a growing body of recent work has instead suggested that people may reach summary judgments as a function of heuristic cues and goal-oriented ratio...

In the context of the current “replication crisis” across the sciences, failures to reproduce a finding are often viewed as discrediting it. This paper shows how such a conclusion can be incorrect. In 1981, Schuman and Presser showed that including the word “freedom” in a survey question significantly increased approval of allowing a speech against...

Policymakers may wish to take into account public opinion on climate change as they craft legislation, but if public opinion changes substantially in response to seemingly trivial changes in survey questionnaire design, perhaps such reliance would be unwise. This paper examines 110 experiments implemented in surveys of truly random samples of Ameri...

Research in a few U.S. states has shown that candidates listed first on ballots gain extra votes as a result. This study explored name order effects for the first time in New Hampshire, where such effects might be weak or entirely absent because of high political engagement and the use of party column ballots. In general elections (in 2012 and 2016...

Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of optimal methods or whether presumptively optimal methods are not, in fact, optimal. This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experi...

Questionnaire design is routinely guided by classic experiments on question form, wording, and context conducted decades ago. This article explores whether two question order effects (one due to the norm of evenhandedness and the other due to subtraction or perceptual contrast) appear in surveys of probability samples in the United States and 11 ot...

There is an ongoing debate in the survey research literature about whether and when probability and nonprobability sample surveys produce accurate estimates of a larger population. Statistical theory provides a justification for confidence in probability sampling as a function of the survey design, whereas inferences based on nonprobability samplin...

Survey researchers today can choose between relatively higher-cost probability sample telephone surveys and lower-cost surveys of nonprobability samples of potential respondents who complete questionnaires via the internet. Previous studies generally indicated that the former yield more accurate distributions of variables, but little work to date h...

Predictions about the effects of climate change cannot be made with complete certainty, so acknowledging uncertainty may increase trust in scientists and public acceptance of their messages. Here we show that this is true regarding expressions of uncertainty, unless they are also accompanied by acknowledgements of irreducible uncertainty. A represe...

In the last 60 years, the proportion of white Americans expressing anti-black prejudice in face-to-face survey interviews has declined dramatically. To test whether social desirability pressures affect the expression of anti-black prejudice, we analyzed a within-subjects experiment in the 2008 American National Election Study in which white respond...

A crescendo of incidents have raised concerns about whether scientific practices in psychology may be suboptimal, sometimes leading to the publication, dissemination, and application of unreliable or misinterpreted findings. Psychology has been a leader in identifying possibly suboptimal practices and proposing reforms that might enhance the effici...

For decades, social psychologists have collected data primarily from college undergraduates and, recently, from haphazard samples of adults. Yet researchers have routinely presumed that thusly observed treatment effects characterize “people” in general. Tests of seven highly-cited social psychological phenomena (two involving opinion change resulti...

Questionnaires routinely measure unipolar and bipolar constructs using rating scales. Such rating scales can offer odd numbers of points, meaning that they have explicit middle alternatives, or they can offer even numbers of points, omitting the middle alternative. By examining four types of questions in six national or regional telephone surveys,...

Race of interviewer effects are presumed to occur in surveys because respondents answer questions differently depending on interviewer race. This article explored an alternative explanation: differential respondent recruitment. Data from telephone interviews conducted during the 2008 U.S. Presidential election campaign by major survey organizations...

People tend to perceive their own opinions to be especially prevalent among others, especially among members of their ingroups. This paper proposes that this process operates differently in groups with prominent figures who take public stances on issues (opinion cues) that suggest what group members do or should think when some members of such grou...

[This corrects the article DOI: 10.1371/journal.pone.0182063.].

Many studies in various countries have found that telephone and internet surveys of probability samples yielded data that were more accurate than internet surveys of nonprobability samples, but some authors have challenged this conclusion. This paper describes a replication and an expanded comparison of data collected in the United States, using a...

This chapter explains the design of the multinational study of questionnaire design (MSQD) and the challenges faced when implementing the project across countries. It then elaborates on the sampling and online implementation of the questionnaire, as well as the questionnaire design experiments selected for the study. The aim of the MSQD was to cond...

Postelection surveys regularly overestimate voter turnout by 10 points or more. This article provides the first comprehensive documentation of the turnout gap in three major ongoing surveys (the General Social Survey, Current Population Survey, and American National Election Studies), evaluates explanations for it, interprets its significance, and...

This handbook is a comprehensive reference guide for researchers, funding agencies and organizations engaged in survey research. Drawing on research from a world-class team of experts, this collection addresses the challenges facing survey-based data collection today as well as the potential opportunities presented by new approaches to survey resea...

Few topics in survey methodology have received more attention than questionnaire design and there is a large and growing body of research findings that inform best practices. But for every questionnaire design topic for which there are accepted best practices there are several topics that still generate a large amount of controversy, and these cont...

Survey research is facing some really serious challenges. There are reasons to be concerned about the methods that are being used in many cases and the potential for improving them. One could take that as a downer, but before I get into the downer part I thought it would be nice to look at an upper part. LinChiat Chang and I have been attempting to...

Assessing survey accuracy is important to the development of valid and reliable methodologies for the practice of survey research. However, despite this importance, surprisingly little research has been done gauging survey measurement accuracy, largely due to the difficulty with identifying and collecting objective measures to use as accuracy bench...

We know a lot about best practices with regard to questionnaire design but there is also a lot we don’t know and lots of future research that is needed. Many of the other contributors to this volume are experts in all of these issues and those who know this literature in detail know that everything I will include in this chapter has controversy beh...

Most Americans recognize that smoking causes serious diseases, yet many Americans continue to smoke. One possible explanation for this paradox is that perhaps Americans do not accurately perceive the extent to which smoking increases the probability of adverse health outcomes. This paper examines the accuracy of Americans’ perceptions of the absolu...

Exploring responses of 500. (PDF)

References for supporting information. (PDF)

Generalized Additive Models predicting the probability of being a current smoker vs. former smoker: FFRISP (n = 471). (PDF)

Generalized Additive Models predicting the probability of being a current smoker vs. never smoker: FFRISP (n = 714). (PDF)

Literature on the relation of health risk perceptions with quitting smoking. (PDF)

Generalized Additive Models predicting the probability of being a current smoker: SRBI Survey (n = 456). (PDF)

Survey methodology. (PDF)

Proportions of Americans who failed to assert that smoking is dangerous to human health: Gallup Organization Surveys. (PDF)

Generalized Additive Models predicting the probability of being a current smoker: Harris Interactive Survey (n = 795). (PDF)

Demographics of current and former smokers in the SRBI Survey, current and former smokers in the Harris Interactive Survey, all individuals in the FFRISP Survey, and the nation’s population. (PDF)

In much psychology research, mediators are measured, not manipulated. Therefore, the paths from mediators to outcomes—the so-called b paths—can be confounded by omitted variable bias, as in any other correlational analysis. The present research builds on the logic of falsification tests in econometrics and sensitivity analysis in statistics to prop...

In much psychology research, mediators are measured, not manipulated. Therefore, the paths from mediators to outcomes—the so-called b paths—can be confounded by omitted variable bias, as in any other correlational analysis. The present research builds on the logic of falsification tests in econometrics and sensitivity analysis in statistics to prop...

When large-scale accidents cause catastrophic damage to natural or cultural resources, government and industry are faced with the challenge of assessing the extent of damages and the magnitude of restoration that is warranted. Although market transactions for privately owned assets provide information about how valuable they are to the people invol...

This study (1) tested the effectiveness of a new survey tool to collect ego-centered network data and (2) assessed the impact of giving people feedback about their network on subsequent responses. The new tool, GENSI (Graphical Ego-centered Network Survey Interface), allows respondents to describe all network contacts at once via a graphical repres...

Attitude strength has been the focus of a huge volume of research in psychology and related sciences for decades. The insights offered by this literature have tremendous value for understanding attitude functioning and structure and for the effective application of the attitude concept in applied settings. This is the first Annual Review of Psychol...

In post-election surveys, the proportion of respondents who claim to have voted is routinely greater than officially reported turnout rates, which has often been attributed to respondent lying. Such concerns have led researchers to prefer gauging respondents’ turnout by using government records of their behavior (called “validated turnout measures”...

Desrvousges et al. (2012) investigate criteria for judging the adequacy of scope test differences in contingent valuation studies. They focus particular attention on our study (Chapman et al. 2009), arguing that, while it demonstrated a statistically significant scope effect, the effect is too small. Unfortunately, DMT misinterpreted Chapman et al....

We thank Robert Unsworth, Sarah Malloy, Mark Hansen, Sidney Verba, Morris Fiorina, Catherine Heaney, Russell Hardin, Robert Mitchell, Robert Mendelson, Daniel Kahneman, David Schkade, Kathleen Carr, Anne Smith, Paul Beck, Richard Timpone, Dean Lacy, Marilynn Brewer, Robert Arkin, Roger Tourangeau, Nora Cate Schaeffer, Seymour Sudman, and the member...

Past research demonstrated that racial prejudice played a significant role in the 2008 presidential election, but relatively less is known about the relationship between prejudice and public opinion throughout the Obama administration. In the present research, we examined not only whether racial attitudes were associated with evaluations of Mr. Oba...

According to some recent research, Americans hold a great deal of misinformation about important political issues. However, such investigations treat incorrect answers to quiz questions measuring knowledge as evidence of misinformation. This study instead defines misperceptions as incorrect answers that respondents are confident are correct. Two su...

Experiments embedded in surveys of nationally representative samples of American adults assessed whether attitudes toward preparation for the possible effects of global warming varied depending on who endorsed such efforts, the stated purpose of preparation, the consequences of global warming targeted in a preparation message, and the words used to...

Despite the release of his birth certificate, some Americans express continued skepticism over whether Barack Obama was born in the United States. This study examined two possible causes of birther beliefs: that Republicans and conservatives, whose ideological beliefs and policy preferences led to disapproval of the president, might be particularly...

Much published research indicates that voting behavior in the 2008 presidential election and evaluations of Barack Obama were importantly influenced by anti-Black sentiment. Various psychological theories made opposing predictions as to whether exposure to the first Black president during his first term would strengthen or weaken the alignment betw...

Although some past studies suggest that candidates may receive more votes when their names are listed first among their competitors than when they are listed later, two recent studies challenged this conclusion with regard to major-party candidates running in statewide races and raised questions about the impact of analytic methods on the conclusio...

Provides new insights into the accuracy and value of online panels for completing surveys Over the last decade, there has been a major global shift in survey and market research towards data collection, using samples selected from online panels. Yet despite their widespread use, remarkably little is known about the quality of the resulting data. Th...

This chapter provides an overview of studies comparing the quality of data collected by online survey panels by looking at three criteria: (1) comparisons of point estimates from online panels to high-quality, established population benchmarks; (2) comparisons of the relationship among variables; and (3) the reproducibility of results for online su...

In this introductory chapter, written by the six editors of this volume, we introduce and attempt to systematize the key concepts used when discussing online panels. The connection between Internet penetration and the evolution of panels is discussed as are the different types of online panels, their composition, and how they are built. Most online...

Past studies of elections have shown that candidates whose names were listed at the beginning of a list on a ballot often received more votes by virtue of their position. This article tests speculations about the cognitive mechanisms that might be responsible for producing the effect. In an experiment embedded in a large national Internet survey, p...

In 2010 and 2011, highly visible national surveys documented frequent failure among the public, especially among Republicans, to acknowledge that Barack Obama was born in the United States. However, different questions yielded strikingly different results. The highest rate of partisan division was generated by a CBS/ New York Times closed-ended que...

Paralleling the distinction between mindful and mindless behavior, survey methodologists have proposed a distinction between two ways that a respondent might answer questions in a questionnaire: optimizing and satisficing. To help psychologists and other social and behavioral scientists understand this distinction and to enhance the quality of thei...

Although agree-disagree (AD) rating scales suffer from acquiescence response bias, entail enhanced cognitive burden, and yield data of lower quality, these scales remain popular with researchers due to practical considerations (e.g., ease of item preparation, speed of administration, and reduced administration costs). This article shows that if res...

Despite efforts by some congressional legislators to pass laws to limit greenhouse gas emissions and reduce the use of fossil fuels, no such laws have yet been adopted. Is this failure to pass new laws attributable to a lack of public desire for such legislation? Data from national surveys support two answers to this question. First, large majoriti...

Survey respondents typically report having voted at a rate higher than the nation in fact turned out on election day. This may be the result of errors people make when trying to remember whether they voted and of motivated misreporting due to social desirability bias. This paper explores whether a new sequence of questions designed to reduce both t...

Prior to the Nixon administration, environmental policy in the United States was rudimentary at best. Since then, it has evolved into one of the primary concerns of governmental policy from the federal to the local level. As scientific expertise on the environment rapidly developed, Americans became more aware of the growing environmental crisis th...

Studies have shown that allowing people to answer questionnaires completely anonymously yields more reports of socially inappropriate attitudes, beliefs, and behaviors, and researchers have often assumed that this is evidence of increased honesty. But such evidence does not demonstrate that reports gathered under completely anonymous conditions are...

Survey researchers often administer batteries of questions to measure respondents' abilities, but these batteries are not always designed in keeping with the principles of optimal test construction. This paper illustrates one instance in which following these principles can improve a measurement tool used widely in the social and behavioral science...

Introduction Evaluating Data Quality Number of Scale Points Labeling Scale Points No-Opinion Filters Epilogue

Researchers often measure attitudes and beliefs using “some/other” questions (“Some people think that … but other people think that …”) instead of asking simpler “direct” questions. This article reports studies testing the hypothesis that the some/other question form yields improved response validity. Eight experiments embedded in national surveys...

The American National Election Studies (ANES) is the premier social science survey program devoted to voting and elections. Conducted during the presidential election years and midterm Congressional elections, the survey is based on interviews with voters and delves into why they make certain choices. This book brings together a group of leading so...

A great deal of developmental research has relied on self-reports solicited using the "some/other" question format ("Some students think that… but other students think that…"). This article reports tests of the assumptions underlying its use: that it conveys to adolescents that socially undesirable attitudes, beliefs, or behaviors are not uncommon...

For decades, numerous surveys have asked Americans the "Most Important Problem" (MIP) question: "What do you think is the most important problem facing this country today?" Global warming and the environment have rarely been cited by more than a small number of respondents in these surveys in recent years, which might seem to suggest that these hav...

For decades, numerous surveys have asked Americans the "Most Important Problem" (MIP) question: "What do you think is the most important problem facing this country today?" Global warming and the environment have rarely been cited by more than a small number of respondents in these surveys in recent years, which might seem to suggest that these hav...

Does “climate change” seem like a less serious problem than “global warming” to Americans and Europeans? Does describing the costs of climate change mitigation in terms of “higher taxes” instead of “higher prices” reduce public support for such efforts? In an experiment embedded in an American national survey, respondents were randomly assigned to...

Many researchers rely on high-quality face-to-face national surveys conducted by the federal government to estimate the prevalence of nicotine product use, but some scholars have suggested that adults' self-reports in such surveys are intentionally distorted by social desirability response bias, thus raising questions about the validity of those da...

Overview of Political Psychology2Requirements of Democracy3Causes of Behavior4Causes of Political Attitudes and Beliefs5How do People Decide with Which Political Party to Affiliate?6Causes of Other Political Judgments7Causes of Political Knowledge8Summary

Surveys usually yield reported rates of voting in elections that are higher than official turnout figures, a phenomenon often attributed to intentional misrepresentation by respondents who did not vote and would be embarrassed to admit that. The experiments reported here tested a procedure for reducing social desirability response bias by allowing...

Although Agree/Disagree (A/D) rating scales are hugely popular in the social sciences, a large body of research conducted during more than five decades has documented the bias that results from acquiescence in responses to these items. This may be a reason to prefer questions with Item Specific (IS) response options, but remarkably little research...

According to psychological theories of cognitive consistency, voters are likely to overestimate the degree to which political candidates they like agree with their own policy preferences, and to overestimate the degree to which political candidates they oppose disagree with their own policy preferences. This paper reviews and critiques the literatu...

When the United States began its overt military conflict with Iraq in January 1991, the American news media focused attention powerfully and seemingly unceasingly on the Gulf crisis for a period of almost three months. Through analysis of data collected in a panel survey conducted before and after the war, we found that paralleling this surge in me...

A previous field experiment conducted via national surveys showed that data collected via the Internet manifested higher concurrent and predictive validity and less random and systematic measurement error than data collected via telephone interviewing. To ascertain the extent to which these differences were attributable to mode per se, a laboratory...