Figure 3 - uploaded by Jens Hainmueller
Content may be subject to copyright.
The partial R 2 values for our core attributes for the pooled MT data as function of the number of completed tasks.
Source publication
In recent years, political and social scientists have made increasing use of conjoint survey designs to study decision-making. Here, we study a consequential question which researchers confront when implementing conjoint designs: how many choice tasks can respondents perform before survey satisficing degrades response quality? To answer the questio...
Context in source publication
Context 1
... result for the partial R 2 values, presented in Figure 3, confirms the stability of conjoint responses across the 30 tasks for our MT respondents. The partial R 2 for the two core attributes is about 0.104 in the respondents' first task, with a 95% block-bootstrapped confidence interval of [0.091, 0.118]. ...
Citations
... In the fast-growing literature on conjoint analysis, we are the first to consider the problem of finding optimal profiles. Although there is a large body of related work in the sequential decision-making context (e.g., the multiarm bandit setting of Audibert et al. (2010)), in the non-sequential conjoint setting, most existing works have focused upon the estimation of various causal effects (de la Cuesta et al., 2019;Egami and Imai, 2019;Goplerud et al., 2022;Hainmueller et al., 2014), hypothesis testing (Ham et al., 2022;Liu and Shiraito, 2023), interpretation of causal estimands (Abramson, Koçak, et al., 2022), and experimental designs (Bansak, Hainmueller, Hopkins, and Yamamoto, 2018). ...
Conjoint analysis, an application of factorial experimental design, is a popular tool in social science research for studying multidimensional preferences. In such experiments in the political analysis context, respondents are asked to choose between two hypothetical political candidates with randomly selected features, which can include partisanship, policy positions, gender and race. We consider the problem of identifying optimal candidate profiles. Because the number of unique feature combinations far exceeds the total number of observations in a typical conjoint experiment, it is impossible to determine the optimal profile exactly. To address this identification challenge, we derive an optimal stochastic intervention that represents a probability distribution of various attributes aimed at achieving the most favorable average outcome. We first consider an environment where one political party optimizes their candidate selection. We then move to the more realistic case where two political parties optimize their own candidate selection simultaneously and in opposition to each other. We apply the proposed methodology to an existing candidate choice conjoint experiment concerning vote choice for US president. We find that, in contrast to the non-adversarial approach, expected outcomes in the adversarial regime fall within range of historical electoral outcomes, with optimal strategies suggested by the method more likely to match the actual observed candidates compared to strategies derived from a non-adversarial approach. These findings indicate that incorporating adversarial dynamics into conjoint analysis may yield unique insight into social science data from experiments.
... 2 Given the degree to which inattentiveness can bias experimental studies toward substantively smaller and non-significant effect sizes, this finding reaffirms that conjoint experimentalists may not yet have sufficient strategies for addressing inattentiveness. 3 1 While some research has found only limited degradation of response quality as the number of attributes or tasks increases (Bansak et al., 2018;, this does not address the fact that respondent inattentiveness may nevertheless be high, regardless of whether it changes over the course of an experiment. 2 These articles were published in eight high-ranking journals in political science (American Political Science Review, American Journal of Political Science, Journal of Politics, British Journal of Political Science, Political Behavior, Public Opinion Quarterly, Political Science Research and Methods, and Journal of Experimental Political Science). 3 Notably, recent work by Clayton et al. (2023) investigates measurement reliability in conjoint experiments. ...
... There has been some concern about whether respondents experience information overload (see Bansak et al., 2018;. In the studies presented here, approximately two-thirds of respondents (on average) correctly answered at least two out of three factual questions about a conjoint task. ...
In typical survey experiments--i.e., experiments involving a limited amount of manipulated content--respondent inattentiveness tends to bias treatment effect estimates toward zero. This same bias likely exists in conjoint experiments, which require respondents to attend to an even larger amount of manipulated content. Yet, little research has investigated strategies to account for inattentiveness in conjoint experiments specifically. In this study, we explore ways to both measure--and account for--inattentiveness when estimating causal effects in single- and two-profile conjoint designs. Replicating published conjoint experiments with large national samples, our study offers researchers a simple strategy that relies upon pre-treatment measures of attentiveness. Further, we propose a novel method--``conjoint attention checks'' (CACs)-- to both measure respondents' attentiveness to conjoint profiles and provide for more robust tests of hypotheses in conjoint experiments. Lastly, we provide researchers with importable survey templates to facilitate the use of CACs in their own experiments.
... I systematically test the explanations from the literature by using a conjoint survey experiment run on 1953 respondents in the United States. Conjoint experiments allow for simultaneous testing of multiple factors, showing which factors affect the respondent's decision the most (Bansak, et al. 2018). As such, they also reflect real-life decision-making, in which people must balance different attributes. ...
... Zero indicates a neutral feeling toward a religious group. 6 Research shows that survey response quality does not substantially decline even when respondents go through dozens of tasks (Bansak, et al. 2018). 7 Researchers usually provide average marginal component effects (AMCEs) or marginal means when evaluating conjoint experiment results. ...
How do residents evaluate zoning relief applications for new houses of worship? Do they decide based on the facility’s expected level of nuisance, the religion of the house of worship, or the attitudes of neighbors and local officials? Using a conjoint survey experiment, this paper shows that religion is the most important predictor of resistance. People are more likely to resist new mosques than Christian churches, irrespective of other facility properties. Furthermore, this paper highlights the significant role of partisanship in residents’ evaluation of zoning relief applications. Republican respondents were more likely to reject minority houses of worship and support Christian churches than Democrats, moderating the influence of religion. Such bias has important implications for the zoning relief application process. Local officials should evaluate residents’ opposition differently when the application concerns minority groups.
... There has been some concern about whether respondents experience information overload (see Bansak et al., 2018;. In the studies presented here, approximately two-thirds of respondents (on average) got at least two out of three factual questions about a conjoint task correct. ...
In typical survey experiments–i.e., experiments that involve a relatively limited amount of manipulated content–respondent inattentiveness tends to bias treatment effect estimates toward zero. Such bias may be even more pronounced in conjoint experiments, which require respondents to attend to an even larger amount of manipulated content. And yet, little research has investigated strategies to account for inattentiveness in conjoint experiments specifically. In this study, we explore potential ways to both measure–and account for–respondent inattentiveness when estimating causal effects in both single- and two-profile conjoint experiments. Replicating published conjoint experiments with large national samples, we demonstrate how researchers can implement a simple strategy using pre-treatment measures of attentiveness. Toward this end, we propose a novel method– “conjoint attention checks” (CACs)–to both measure respondents’ attentiveness to conjoint profiles and provide for more robust tests of hypotheses in conjoint experiments.
... There has been some concern about whether respondents experience information overload (see Bansak et al., 2018;. In the studies presented here, approximately two-thirds of respondents (on average) got at least two out of three factual questions about a conjoint task correct. ...
In typical survey experiments–i.e., experiments that involve a relatively limited amount of manipulated content–respondent inattentiveness tends to biase treatment effect estimates toward zero. Such bias may be even more pronounced in conjoint experiments, which require respondents to attend to an even larger amount of manipulated content. And yet, little research has investigated strategies of accounting for inattentiveness in conjoint experiments specifically. In this study, we explore potential ways to both measure–and account for–respondent inattentiveness when estimating causal effects in both single- and two-profile conjoint experiments. Replicating published conjoint experiments with large national samples, we demonstrate how researchers can implement a simple strategy using pre-treatment measures of attentiveness. Toward this end, we propose a novel method– “conjoint attention checks” (CACs)–to both measure respondents’ attentiveness to conjoint profiles and provide for more robust tests of hypotheses in conjoint experiments.
... We also ask a standard battery of questions regarding ideology, partisanship, race/ethnicity, age, income, regional location, feelings toward Donald Trump, and whether the participant or an immediate family member has ever served in the armed forces or been a police officer. Research indicates that this number of decision tasks does not overwhelm participants or undermine response quality and response rate (Bansak et al. 2018;Jenke et al. 2021). ...
Under what conditions does the US public support the domestic use of different institutions of coercive state power? We theorize how the type of situation, the type of actor, the mission, and the type of intervention influence public support for such missions. We use a preregistered conjoint survey experiment to test our hypotheses and find that participants (i) are less supportive of interventions in response to protests than to natural disasters or terrorism, (ii) generally prefer the police or the National Guard to the military, (iii) mistrust order maintenance interventions, and (iv) prefer intervening actors be unarmed. Preferences (ii)–(iv) are strongly conditioned on the type of event. We also find that Republicans are more accepting of military actors, order maintenance interventions, armed interventions, and policing responses to protests. We note implications for public trust in the military, the militarization of policing, and the domestic use of federal forces.
... 51 Compared to ours, conjoint designs could have unclear and often arbitrary reference categories, which may seriously impact data interpretation and inference (Leeper, Hobolt, and Tilley 2020). Conjoint designs are cognitively demanding for the respondents who, as a result, may satisfice by relying on heuristics which can degrade response quality (Bansak et al. 2018). The independent randomization of many candidate traits in conjoint designs may also produce unrealistic profiles, and, separately, the simultaneous assessment of numerous hypotheses may encourage data mining (Bansak et al. 2019). ...
Why do voters shun some business tycoons yet elect others into power? As structural conditions facilitate the entry of super-wealthy actors into politics, the differential electoral support across business elites suggests a puzzle. We conceptualize four mechanisms behind the popular support for “tycoon candidates”: competence signaling, framing, fame, and clientelism. To test their relative efficacy, we leverage an experiment embedded in a nationally representative survey in South Africa, an important developing democracy where certain tycoons are successfully running for office. We find that, across distinct electoral appeals by tycoon candidates, clientelism is particularly effective, especially for mobilizing support from the less affluent voters. Racial framing significantly decreases support among white voters. Meanwhile, tycoons’ competence signaling or fame do not help them at the ballot box. By identifying the micro-level underpinnings of voter support across tycoon candidates, our study contributes to the literatures on business and politics, voting behavior, and clientelism.
... MTurk samples are typically more representative than other convenience samples, such as student samples (Berinsky, Huber and Lenz 2012), and scholars have shown that experiments conducted on MTurk replicate in other samples (Coppock 2019).The experimental design is a multifactor or "conjoint" vignette experiment, where each respondent is shown brief descriptions of five health care scenarios. 3 With 507 respondents × 5 3Bansak et al. (2018) shows that response quality does not substantially decrease when respondents are shown several tasks or scenarios in conjoint experiments. The study used MTurk as one of its participant pools. ...
Context:
Nearly half of adults in the United States have received an unexpected medical bill in recent years. While government, provider, and insurance policies related to unexpected medical expenses are the subject of attention in media, this study focuses on variation in public support.
Methods:
The study employs two multi-factor survey vignette experiments to detect how different features of common health care scenarios that result in costly medical expenses influence the public's sympathy for patient, perceived fairness of the medical costs, and demand for government action.
Findings:
The results point to the cost, severity of the treatment, and the patient's insurance situation as important for public opinion. The public is significantly more supportive of government action when the costs are high and out of the patient's control; in contrast, respondents are generally less sympathetic toward patients described as uninsured or who seek out more costly providers.
Conclusions:
The findings underscore the sensitivity of health care attitudes to framing effects, which may occur when media choose how to cover health care costs. The results also point to a potential mismatch in legislation addressing "surprise billing," narrowly, with public support for government to address disproportionate costs across a broader range of scenarios.
... Second, we simply run an OLS regression of the binary dependent variable on each of the two independent variables and their interaction, where each independent variable runs from 1 to 9 and is treated as continuous. 16 One may worry about question fatigue when doing the same conjoint task 7 times, but existing evidence suggests that respondents exhibit similar behavior when completing up to 30 such tasks (Bansak et al., 2018). ...
Security concerns about immigration are on the rise. Many countries respond by fortifying their borders. Yet little is known about the influence of border security measures on perceived threat from immigration. Borders might facilitate group identities and spread fear of outsiders. In contrast, they might enhance citizens' sense of security and control over immigration. We test these claims using survey experiments run on a nationally representative sample of over 1,000 Americans. The findings show that allocating more government resources to border security increases desired levels of immigration. This effect is likely driven by a sense of control over immigration, induced by border security measures even when the number or characteristics of immigrants remain unchanged. Our findings suggest that border controls, which are widely considered as symbols of closure and isolation, can promote openness to immigration.
... In this respect, there is no agreement on the ideal number of attributes to consider, nor on the number of tasks to be implemented. Researchers need to balance among the theoretical aspects to investigate, the respondents' fatigue, as well as sample size and statistical power (Bansak et al., 2018(Bansak et al., , 2021. A second criticism is that conjoint analysis may lead scholars to less formalised and more inductive forms of research, posing less restrictions with respect to the number of factors under observation. ...
The article offers an overview of the use of survey experiments in political research by relying on available examples, bibliographic data and a content analysis of experimental manuscripts published in leading academic journals over the last two decades. After a short primer to the experimental approach, we discuss the development, applications and potential problems to internal and external validity in survey experimentation. The article also provides original examples, contrasting a traditional factorial and a more innovative conjoint design, to show how survey experiments can be used to test theory on relevant political topics. The main challenges and possibilities encountered in envisaging, planning and implementing survey experiments are examined. The article outlines the merits, limits and implications of the use of the experimental method in political research.
