No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Does Conjoint Analysis Mitigate Social Desirability Bias?
Abstract
How can we elicit honest responses in surveys? Conjoint analysis has become a popular tool to address social desirability bias (SDB), or systematic survey misreporting on sensitive topics. However, there has been no direct evidence showing its suitability for this purpose. We propose a novel experimental design to identify conjoint analysis’s ability to mitigate SDB. Specifically, we compare a standard, fully randomized conjoint design against a partially randomized design where only the sensitive attribute is varied between the two profiles in each task. We also include a control condition to remove confounding due to the increased attention to the varying attribute under the partially randomized design. We implement this empirical strategy in two studies on attitudes about environmental conservation and preferences about congressional candidates. In both studies, our estimates indicate that the fully randomized conjoint design could reduce SDB for the average marginal component effect (AMCE) of the sensitive attribute by about two-thirds of the AMCE itself. Although encouraging, we caution that our results are exploratory and exhibit some sensitivity to alternative model specifications, suggesting the need for additional confirmatory evidence based on the proposed design.
... Respondents pointing to restaurant features rather than to the criminal record does not necessarily contradict the conjoint experiment findings; restaurant characteristics were also meaningful in the conjoint experiment, and these factors may have simply outweighed the criminal record when asked to select the most important factor. Also, the conjoint analysis may be detecting criminal record bias that might not be revealed through other methods; conjoint experiments can mitigate social desirability bias because multiple observed profile attributes exist for any decision (Horiuchi et al., 2022). Respondents may be unaware they are making a socially undesirable choice, or they might justify bias in their decision-making because other components should be considered within a profile (Horiuchi et al., 2022). ...
... Also, the conjoint analysis may be detecting criminal record bias that might not be revealed through other methods; conjoint experiments can mitigate social desirability bias because multiple observed profile attributes exist for any decision (Horiuchi et al., 2022). Respondents may be unaware they are making a socially undesirable choice, or they might justify bias in their decision-making because other components should be considered within a profile (Horiuchi et al., 2022). Taken together, the conjoint analysis enables us to detect aversion to this hiring initiative, whereas the open-ended responses provide insight into the reported top reason underlying these decisions and the rationale for criminal record aversion among respondents prioritizing that component in their decisions. ...
A common concern in hiring individuals with criminal convictions is the stigma associated with the criminal record, which can include negative consumer reactions. We provide two novel tests of courtesy stigma, or the idea of transferring negative traits from one entity to another, through a nationwide survey. Using a conjoint experiment and a follow-up open-ended question, we first establish whether the public is less likely to select a restaurant if the business has a hiring initiative for people with conviction records. Using a vignette experiment , we then test whether the same factors driving personal stigma apply to courtesy stigma and whether hiring messaging frames influence courtesy stigma. We find evidence of criminal conviction courtesy stigma in the conjoint experiment. Respondents, however, typically reported the characteristics of the business itself as influential, and when the criminal record mattered, the underlying rationale was mainly instrumental: Avoiding a criminal record-friendly business was often due to safety concerns. We find similar instrumental results in the vignette experiment; the quality of service, rather This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made.
... Furthermore, conjoint experiments guard against social desirability bias and expressive responding (Horiuchi, Markovich, and Yamamoto, 2021;Theodoridis et al. 2023), which could greatly affect reported judgments of political violence (Kalmoe and Mason 2022a; Theodoridis et al. 2023). Because conjoints vary several attributes, it is harder to consciously avoid norm-violating bias towards certain social groups. ...
... However, the fact that we delivered information about perpetrator race via text normally amplifies rather than mutes the effect of profile race in conjoints (Abrajano et al. 2018). Furthermore, conjoint experiments tend to guard against social desirability bias (Horiuchi et al. 2021). Therefore, while it is possible the artificial nature of each scenario affects the role of perpetrator race, other aspects of the experiments do not display that issue. ...
What factors do Americans find most important when evaluating acts of political violence? Normatively, details regarding the violent act (e.g., the target and violence severity) should determine the punishment for political violence. However, recent work on polarization and identity suggests evaluations of political violence may depend on the perpetrator’s characteristics. In two pre-registered conjoint experiments, we vary both perpetrator characteristics and features of the violent act to discern the relative weight of act-centric and perpetrator-centric considerations. We find that even though the perpetrator’s characteristics (e.g., partisanship) do influence people’s punishment of political violence, the features of the act matter much more for citizen evaluations of political violence, on average. Though these findings can be interpreted as normatively negative given the perpetrator’s identities do influence punishment, the disproportionate effect of the violent act’s target and severity are normatively encouraging.
... These favorable characteristics have contributed to the high popularity of conjoint experiments, as illustrated by their increased usage over time, which in turn has been accompanied by methodological discussions, as can be observed from recent trends in publication of discussions on conjoint experiment methodologies in outlets such as Political Analysis and Political Science Research and Methods. While some recent studies have thereby corroborated conjoint experiments beneficial properties, such as the potential to reduce social desirability bias (Horiuchi et al., 2022), other studies point to potential problems and challenges. Most prominently, analytical problems related to the "quantity of interest" have been discussed (Ganter, 2023;Bansak et al., 2023;Abramson et al., 2022Abramson et al., , 2023, especially when analyzing subgroups (Leeper et al., 2020), as well as the challenges related to testing multiple hypotheses (Liu and Shiraito, 2023) or bias induced through low intra-respondent reliability (Clayton et al., 2023). ...
... Satisficing through speeding can be controlled for relatively easily by excluding respondents that did not spend a minimum time on the survey. Not revealing true preferences could also happen at any other moment in the survey, i.e., is not limited to-or even relatively less important (Horiuchi et al., 2022) in-conjoint experiments. Moreover, given that respondents typically participate in surveys on a voluntary basis, many individuals who do not want to reveal their preferences likely will just not take part in the survey at all, especially when survey finishers do not get (substantial) rewards. ...
Conjoint experiments have gained popularity in political science, accompanied by methodological discussions. We contribute to these by investigating the impact of (randomly) offering an abstention option alongside the usual forced choice dependent variable. We explore the usage of abstention employing data from a novel experiment with 4466 respondents. The findings reveal that a notable proportion of respondents abstain due to non-acceptance and unwillingness or inability to choose. Abstention from binary choice also impacts the subsequent rating tasks, suggesting binary choice may overestimate favorability deduced from conjoint experiments. Mirroring the real-world decision process, which often includes abstention options, improves our understanding of preferences and prevents biases in the analysis, highlighting the importance of considering offering an abstention option in conjoint experiments.
... First, conjoint experiments can potentially reduce social desirability bias and "satisficing". This enhances their external validity (Hainmueller, Hangartner, andYamamoto 2015 andHoriuchi, Markovich, andYamamoto 2022). Second, when presented in their traditional tabular format, these experiments more effectively manage ordering effects. ...
... First, conjoint experiments can potentially reduce social desirability bias and "satisficing". This enhances their external validity (Hainmueller, Hangartner, andYamamoto 2015 andHoriuchi, Markovich, andYamamoto 2022). Second, when presented in their traditional tabular format, these experiments more effectively manage ordering effects. ...
Związek między preferencjami dot. programów społecznych a ideologiami politycznymi stał się kluczowym obszarem we współczesnej debacie na temat reformy opieki społecznej i polityki społecznej. Przy użyciu eksperymentu conjoint dotyczącego najszerszego w Polsce programu ochrony socjalnej w zakresie zasiłków na dzieci, badamy, w jakim stopniu preferencje dotyczące adekwatności i efektywności programy i stabilności fiskalnej różnią się w zależności od poglądów politycznych. Na postawie wyników stwierdzamy, że w Polsce konserwatywni wyborcy opowiadają się za wyższą adekwatnością wydatków socjalnych, podczas gdy postępowi wyborcy wolą bardziej wydajne i zrównoważone fiskalnie wydatki socjalne. Postępowi wyborcy sprzeciwiają się krajowej dyskryminacji w zakresie świadczeń socjalnych, podczas gdy konserwatywni wyborcy są obojętni na krajową dyskryminację. Artykuł dostarcza cennych spostrzeżeń na temat polskich preferencji dotyczących programów ochrony socjalnej i ilustruje, w jaki sposób ideologie polityczne są powiązane z poparciem społecznym dla polityki socjalnej.
... For example, if SDB was at work and respondents wanted to appear as non-discriminating or even favouring minority groups, we assume that the language survey question-probably the item that might have triggered greater suspicion-would not have generated non-discrimination from candidates and yet discrimination from citizens. Finally, while other experimental designs such as conjoint experiments could have maybe decreased SDB (Horiuchi et al., 2022), it would have jeopardized our objective of comparing the citizenscandidacy-offices results. ...
This research note presents the results of audit studies that were conducted with the constituency offices of provincial and federal elected representatives across Canada. We investigate whether individuals from ethnic minority groups, the LGBTQ+ community and French or English speakers are discriminated against when contacting their constituency office for administrative services. Survey experiments administered to both candidates of the 2021 Canadian election and a representative sample of Canadian citizens complement these studies. Our results indicate the absence of discrimination towards constituents from an ethnic minority or who identify with the LGBTQ+ community. We found, however, that emails sent in French were less likely to be answered by Members of Parliament (MPs) than those sent in English. Constituency offices of anglophone MPs and those representing ridings with a small proportion of francophones were significantly less likely to respond to French emails. A similar pattern, albeit more moderate, is observed among constituency offices of francophone MPs in response to English emails. The survey experiments show similar discrimination from citizens but less so from candidates.
... Most importantly, due to the fact that respondents in conjoint experiments are not asked to 'express' or present their attitudes or preferences, but their preferences are only inferred from their choices between profiles, these experiments are expected to reduce social desirability bias. Indeed, Horiuchi, Markovich, and Yamamoto (2022) recently provided evidence that conjoint experiments can substantially reduce social desirability biases. ...
... Scholars have employed conjoint analysis in a range of policy issues (Christensen and Rapeli 2021) such as environmental and climate change policies (e.g., Bechtel and Scheve 2013), Eurozone bailout policies (Bechtel, Hainmueller, and Margalit 2017), health policies (e.g., Bridges et al. 2011), and foreign policies (e.g., Clary and Siddiqui 2021). Moreover, existing studies have suggested that conjoint analysis mitigate social desirability biases since respondents are not directly asked for opinions about socially sensitive attributes in the conjoint set-up (e.g., Carey et al. 2020;Horiuchi, Markovich and Yamamoto 2021). In particular, prior studies have suggested that conjoint analysis is a very effective method to measure economic preferences such as willingness to pay since it is easier for respondents to apply preference ordering and rating when competing alternatives are presented than when one option is given (e.g., Breidert, Hahsler, and Reutterer 2006;Schmidt and Bijmolt 2020;Miller et al. 2011). ...
To improve health coverage and revenue collection, several African countries consider tax-for-health-services programs where informal workers pay income tax for health insurance. We examine these programs in Nigeria, investigating whether informal workers support such initiatives, what parameters improve program perception, and what drives preferences about these parameters. Using a conjoint survey experiment with 12,000 informal workers across 12 Nigerian states, we find citizens more likely to support earmarked tax allocation programs, with tax level being most important. Informal workers with less healthcare experience and need, and those feeling distant from government, prioritize tax level and starting date more than others. Our study shows that informal workers in Nigeria generally support earmarked tax-for-health-services programs, but specific design parameters matter. Preferences vary based on healthcare experience and government trust. These findings inform the design of earmarked tax programs to improve healthcare coverage and revenue collection.
... China [33]), preference falsification in surveys, and coordinated inauthentic activity. These biases may be mitigated by using list experiments or conjoint experiments [34]. Additionally, preference falsification may be counteracted using "anonymity sampling" [35] which improves survey accuracy while protecting respondent anonymity. ...
With a folk understanding that political polarization refers to socio-political divisions within a society, many have proclaimed that we are more divided than ever. In this account, polarization has been blamed for populism, the erosion of social cohesion, the loss of trust in the institutions of democracy, legislative dysfunction, and the collective failure to address existential risks such as Covid-19 or climate change. However, at a global scale there is surprisingly little academic literature which conclusively supports these claims, with half of all studies being U.S.-focused. Here, we provide an overview of the global state of research on polarization, highlighting insights that are robust across countries, those unique to specific contexts, and key gaps in the literature. We argue that addressing these gaps is urgent, but has been hindered thus far by systemic and cultural barriers, such as regionally stratified restrictions on data access and misaligned research incentives. If continued cross-disciplinary inertia means that these disparities are left unaddressed, we see a substantial risk that countries will adopt policies to tackle polarization based on inappropriate evidence, risking flawed decision-making and the weakening of democratic institutions.
... The tasks included a "none" option in our choice tasks to better simulate consumer choice in the actual marketplace. The full randomization of sensitive and non-sensitive attributes also helps to mitigate concerns about social desirability bias (Horiuchi, Markovich, and Yamamoto 2022). The exception was the good news/bad news experiment, which had to presented before the other three because its complexity made it impossible to move in a block. ...
Consumers play an important role in regulating labor rights in global supply chains, either by punishing companies that violate labor rights or rewarding those that market fair labor practices. There is, however, currently limited understanding of how consumer demand can be effectively harnessed to protect freedom of association and collective bargaining (FACB) rights in garment-exporting countries. Through a series of conjoint experiments, we test the strength of consumer demand for FACB rights relative to other labor and environmental standards, and manipulate price and information frames to analyze the extent to which a business case exists to promote FACB rights. We find that consumers display willingness to pay premiums for various ethical labels around labor protections, indicating a business case for promoting ethical labor standards in supply chains. However, we also find that consumer demand for certain labor rights—including FACB rights and payment of a living wage—can diminish considerably in the context of price increases, thus limiting the profits firms might accrue by marketing labor rights protections. Our results open up the black box of consumer demand for different labor standards and evaluate the different modes through which consumers can influence labor protections in the global economy.
... The estimates of preferences obtained from conjoint experiments have been validated against both aggregate and individual behavioral benchmarks (Hainmueller, Hangartner, and Yamamoto 2015;Jenke et al. 2021). Conjoint tasks also are resilient to some of the common problems of survey data collection such as satisficing and socialdesirability bias (Bansak et al. 2018;Horiuchi, Markovich, and Yamamoto 2022). ...
Social construction theory postulates that policy outcomes depend on whether target groups are imagined by the public as deserving or undeserving. However, recent evidence demonstrates that the constructions in question are contentious rather than uniformly shared. This article applies the conjoint-experimental method to measure the social construction of immigrant (il)legality and to assess its political implications. We demonstrate that it is multidimensional because the absence of legal status is associated with receipt of government benefits, Hispanic origin, police record, poor English fluency, and less education. We also show that whereas the receipt of government benefits is not associated with the absence of legal status among most respondents, individuals who hold this association support stricter immigration-enforcement policies. Our findings corroborate the social construction approach but also indicate that researchers may want to measure multiple dimensions of target-group constructions in addition to deservingness.
... This way we can assess the importance of specific author's rights. Such methodology has also the advantage of reducing satisficing and social desirability bias, as compared to the traditional experiments (Bansak et al. 2021;Horiuchi et al. 2022). Two last powerful advantages of the conjoint experiment, especially in its tabular and paired profile format, is that it allows to alleviate the ordering effects and makes respondents to consider the trade-offs across attributes more carefully. ...
Policymakers around the world are increasingly regulating creators’ (copy)rights in their work. This includes economic rights and moral rights. While the former type of rights is recognized and protected in most jurisdictions, the approach to the latter – moral rights – differs. How allocation and protection of copyrights affects creators’ choices depends on their preferences. Yet, creators’ preferences are almost not researched empirically. This paper uses a conjoint experiment, applied for the first time in this context, on representative samples in the UK (general population and professionals) and the USA (professionals) designed to reveal people’s preferences with respect to different rights derived from copyrights laws. We find that moral rights are valued more than economic rights, yet participants were willing to trade this right. Such findings might suggest reconsidering existing regulations in particularly with regards to the question of whether the right of attribution can and should be “traded”.
... There are multiple considerations when researchers are deciding whether or not to allow odd attribute-level combinations within their conjoint designs. Previous research has highlighted how the inclusion or exclusion of particular attribute-level combinations can affect the estimation of key quantities of interest from theoretical and statistical perspectives (Abramson, Koçak, and Magazinnik 2022;Bansak et al. 2021;De la Cuesta, Egami, and Imai 2022;Ganter 2023;Hainmueller, Hopkins, and Yamamoto 2014;Horiuchi, Markovich, and Yamamoto 2022). For instance, one consideration is external validity (De la Cuesta et al. 2022). ...
Scholars often face a choice when designing conjoint experiments: to allow for or to exclude “odd” combinations of attribute levels in the randomized conjoint profiles shown to respondents (such as a profile of a Democratic candidate who does not support abortion rights or an individual who is a medical doctor but does not have a graduate degree). While previous work has studied the statistical and theoretical implications of this decision, there has been little effort to analyze how it impacts the behavior of survey respondents. Utilizing eye-tracking, this study considers how respondents’ attention, information search behavior, and choice patterns respond to odd combinations of attributes included in conjoint profiles. We find that the impact of odd attribute-level combinations is minimal. They do not impact attention, search, or choice behavior substantially or consistently. Our conclusion is that scholars should prioritize other considerations—such as statistical, theoretical, and substantive considerations—when designing conjoint experiments.
... The ability of conjoint experiments to capture such multidimensionality makes them better able to mirror real-life situations. Conjoint experiment design further lessens the risk of social desirability bias because respondents can implicitly trade off one attribute against another [85]. ...
Lower-income countries account for a small share of accumulated greenhouse gas emissions but are highly vulnerable to climate-induced events. In response, industrialized higher-income countries, the major contributors to greenhouse gas stock, have pledged policy packages to support developing countries to adapt to climate change. Foreign aid and international migration often figure prominently in such packages. We employ a survey-embedded conjoint experiment to assess public support in Switzerland for international climate assistance packages which consist of six attributes: (1) the country receiving the package (Algeria, Kenya, Bangladesh, and the Philippines); (2) the volume of Swiss bilateral climate aid to this country; (3) the number of climate migrants from this country in Switzerland; (4) types of extreme weather event this country faces; (5) Swiss trade with this country; and (6) the country’s record of voting with Switzerland in the United Nations Security Council. We find that while Swiss respondents are indifferent to aid volume, their support for the policy package diminishes as the number of migrants increases. Respondents support policy packages for countries that trade with and vote alongside Switzerland in the Security Council. Respondents also have country-specific preferences: they support assistance to the Philippines, disfavor Algeria, and are indifferent to Kenya and Bangladesh. Ideology, cultural beliefs, and benchmarking with peer countries of Global North or past Swiss aid and immigration records do not change support for the policy package.
... It allows us to manipulate country of origin and religion as distinct variables, as well as compare their effects to those of other attributes such as skill. In addition, conjoint analysis may help to alleviate social desirability bias (Horiuchi et al., 2022), which can be particularly important for a sensitive topic like immigration. Finally, conjoint experiments have been validated as a measure of preferences on both the aggregate and individual levels Jenke et al., 2021). ...
Does an immigrant’s country of origin shape Americans’ immigration preferences? If so, are some attributes of origin countries likely to provoke particularly strong opposition over others? We answer these questions using three conjoint experimental studies that focus on one of these potential attributes: religion. In doing so we find consistent evidence of strong opposition to immigration from Muslim-majority countries. Furthermore, individual Muslim immigrants face stronger opposition than non-Muslims, independently of their origin countries. Aversion to Muslim immigration is found across the partisan divide, even though it is lower among Democrats than among Republicans. Our findings suggest that exclusionary immigration policies aimed at Muslims, like President Donald Trump’s travel ban, can have non-trivial support amongst the American public. Methodologically, we demonstrate the limitations of relying on country of origin as a catch-all attribute in conjoint experiments and suggest that, instead, researchers should directly manipulate the relevant characteristics of potential immigrants.
... Regarding limitations, while preferable to standard survey approaches to elicit more rigorous assessments of support (18) , conjoint experiments are still subject to a degree of social desirability bias (48) . We tried to reduce this by including estimated government spending or revenue to help respondents consider trade-offs. ...
Objective
This study examines public support—and its drivers—for comprehensive policy packages (i.e., bundles of coherent policy measures introduced together) aimed at improving food environments.
Design
Participants completed an online survey with a choice-based conjoint experiment, where they evaluated pairs of policy packages comprising up to seven distinct food environment measures. After choosing a preferred package or opting for a single policy, participants designed their ideal policy package. Based on their choices, respondents were categorized as resistant, inclined, or supportive towards policy packaging according to their frequency of opting out for single measures and the number of policies they included in their ideal package.
Setting
The study was conducted in Germany via an online survey.
Participants
The sample included 1,200 eligible German voters, recruited based on age, gender, and income quotas.
Results
Based on both opt-out frequency (44.7%) and ideal policy packaging (72.8%) outcomes, most respondents were inclined towards policy packages. The inclusion of fiscal incentives and school-based measures in packages enhanced support, while fiscal disincentives reduced it. Key drivers of support included beliefs about the importance of diet-related issues and the role of government in regulation, while socio-demographic factors, political leaning, and personal experience with diet-related disease had minimal impact.
Conclusions
The results reveal public appetite for policy packages to address unhealthy food environments, contingent on package design and beliefs about the issue’s severity and legitimacy of intervention. Public health advocates should design and promote policy packages aligned with public preferences, especially given anticipated opposition from commercial interests.
... 12 We note that one of the benefits of using conjoint experiments to investigate gender biases is that they aid in minimizing social desirability bias-a potential problem in studies of sensitive issues, for example, concerning gender. As Horiuchi, Markovich, and Yamamoto (2022) show, social desirability bias is minimized when conjoint designs contain fully randomized attributes (as we do here), that is, when respondents have the possibility of seeing, for example, attacks on two women politicians, two men politicians, or one of each. In the study and experiment, we also note that no cues related to gender were given to respondents, who were told only that the study concerned disrespectful social media behavior. ...
Politicians frequently face toxic behaviors. We argue that these behaviors impose a double burden on women, who may not only face higher exposure to toxicity, but experience attacks that they and others understand to be motivated by prejudice and designed to push them out of office. Using large-scale image-based conjoint experiments in the United States, Denmark, Belgium, and Chile, we demonstrate that both politicians themselves and citizens regard messages targeting women politicians as more toxic than otherwise equivalent messages targeting men. This perception intensifies when messages mention gender or come from perpetrators who are men. A second experiment to investigate the mechanisms shows that hostile behaviors toward women are more frequently understood as driven by prejudice and attempts to remove women from politics. These findings highlight the importance of understanding how perceptions of perpetrators’ motives affect the severity of political toxicity, and provide insights into the gendered effects of political hostility.
... The main advantage of this method is that it allows to analyze a number of attributes as they vary across respondents. Compared to other stated preference methods, discrete choice models appear to be less vulnerable to known biases (hypothetical bias, social desirability bias) and yield more robust results (Brand and Rausch 2021;Bryan et al. 2000;Horiuchi, Markovich, and Yamamoto 2022;Murphy et al. 2005). ...
Mobility accounts for the largest share of CO2 emissions generated by tourism industry. The extent of CO2 resulting from tourists' long-distance travel depends strongly on the transport mode used. To reduce these emissions, destinations should engage in promoting a shift from flying and driving to traveling on rail. This study hypothesizes that the entire mobility chain, that is, long-distance travel and mobility at destination, represents an interconnected bundle of services. Using data from a Discrete Choice Experiment conducted with visitors to a tourist destination in Austrian Alps, we estimated the effects of attributes of long-distance travel by personal vehicle and rail (travel time, travel costs, number of transfers) and effects of attributes of local mobility services offered at the destination (transit frequency, carsharing, mobility hub). The outcomes indicate that local mobility services are highly relevant for transport mode choice of tourists and can increase the market share of rail significantly.
... A related reason to employ such tools is their power to estimate multiple causal effects simultaneously, which mirrors the way decisions are made in the real-world with the inherent trade-offs and the interplay of different characteristics of the decision-making environment (Hainmueller et al., 2014). Finally, these designs limit the effect of social desirability bias since they allow respondents to justify their decisions using multiple reasons (Hainmueller and Hopkins 2015;Horiuchi et al., 2021). ...
Current research demonstrated that many factors affect public satisfaction of government services. One central challenge for scholars is that citizen satisfaction is a multi-dimensional concept, yet most empirical tests focus on a small number of factors, and do not account for the simultaneous effects of the many factors. In this study, we present a comprehensive approach that accounts for this multi-dimensionality, and views satisfaction as dependent on two categories of factors: service-specific attributes, and performance comparisons. We assess the multi-dimensional nature of citizen satisfaction with a conjoint experiment, a methodological tool that allows us to test the causal effect of multiple factors operating at the same time. We use the conjoint design to investigate citizen satisfaction from every-day government services. The findings suggest that citizen satisfaction is driven by both groups of variables. Also, we find evidence for a negativity bias in public evaluations when service-specific attributes are involved: individuals' reported satisfaction is more sensitive to negative information like higher costs or unprofessional behavior by service providers compared to positive description of these factors. At the same time, we find that factors in the performance comparisons category create more balanced effects on satisfaction with respect to the information valence. The analysis reveals the drivers of public satisfaction of government services and the importance of key factors. The use of a conjoint design provides a more appropriate tool to tackle the multi-dimensional nature of a central public administration concept of citizen satisfaction.
... Second, factorial experiments can exhibit social desirability bias by emphasizing obtrusive attributes (in our case, e.g., support for autocracies for monetary gain) (Mutz 2011). However, recent research (Horiuchi et al. 2022;Auspurg et al. 2014) shows that social desirability bias is likely of low concern in factorial experiments, at least compared to standard survey items, and even for groups most likely to exhibit these biases. We might be most worried that the vignette experiment could suffer from such bias given it exhibits only two prominent and salient dimensions. ...
The 2022 Russian invasion of Ukraine raised for many parties the question of how to position themselves in view of urgently requested arms deliveries. Since, the topic of arms trade , which has hitherto rarely been addressed, has become a heavily politicized and divisive issue and partly even polarized public opinion. A major prerequisite for parties’ position-taking is to anticipate how voters react to such arms transfers and, more specifically, whether their respective attitudes are structured along the predominant left-right axis. Based on a large-scale survey experiment with French and German voters ( ) in the year before the Russian invasion, we are able to focus on the relationship between ideological predispositions, vote intentions, and issue attitudes in a non-politicized period. Using both vignette and conjoint experiments, we demonstrate that voters’ attitudes on military transfers can be subsumed remarkably well under the left-right scale. Differentiating the impact of normative and economic considerations, the former is stronger among the left, while the latter also affects the attitudes of rightist citizens. However, normative considerations are the most important concern along the whole political spectrum. The turn of the German Green Party in 2022 to assist countries that are being aggressively attacked (because of the Responsibility to Protect), was not reflected in our data.
Data estimates suggest that up to half of all migrants return to the country of origin within 5 years of leaving. Return migration is known to be a boon for the local economy and a catalyst for political reform. However, these effects are conditional on successful reintegration, which is dependent on the preferences of nonmigrants. What causes negative attitudes towards return migration, given its significant potential economic benefits? I argue that nonmigrants are concerned about both the economic and political competition of returnees. Nonmigrants prefer to welcome back migrants who can bring financial capital and employment back home, but will oppose competitors on the job market when unemployment is high. Furthermore, nonmigrants are concerned about the potential role of return migrants as norm entrepreneurs. I test my hypotheses with a conjoint survey experiment conducted in Colombia, as well as an analysis of the 2016 peace referendum.
The term “woke” has evolved from its roots in the 1960s civil rights movement to a mainstream political term, most frequently used disparagingly by Republicans. Despite its frequent use, it’s unclear precisely what Americans perceive of as woke. This study investigates perceptions of woke using a conjoint survey experiment. Respondents in the conjoint are tasked with selecting the more woke list. The lists contain various attributes, identities, and policies that have, over the past few years, been branded as woke. Our findings indicate that there’s generally broad agreement among Democrats and Republicans about what is woke. However, Democratic respondents are more likely associate lower salience racially progressive and gender-progressive items as woke, while Republican respondents identify higher salience items clearly aligned with the Democratic Party as woke. Independents, as a whole, have much less consistent views, tracking more closely with Democrats when it comes to some considerations while more closely with Republicans on others.
Free speech is a core principle in most democratic societies. However, the proliferation of prejudiced speech on social media has prompted intense debates about censorship. As a corollary in the workplace, and against the backdrop of online boundary blurring, we recruited a U.S. nationally representative sample to document that coworkers’ prejudiced social media posts concern Democrats whereas employer censorship of such posts concerns Republicans. To investigate this tension, we integrated boundary research into the psychological safety literature to theorize that organizational censorship on social media differentially affects liberals’ and conservatives’ individual-level psychological safety at work. We also theorized that psychological safety may be fostered—for all employees—by exploiting the tension between safeguards and free expression in organizational social media policies. Taking a multimethod approach, we first explored reactions to organizational censorship of prejudiced speech using open-ended responses (Study 1). Next, we demonstrated in preregistered observational and experimental surveys that such censorship increases Democrats’ psychological safety but decreases Republicans’ psychological safety (Studies 2–3). Finally, in a preregistered, conjoint survey experiment (Study 4), we examined moderators that could help manage this tension. We found that alignment between organizational values and censorship practices, as well as censorship of anti-Black speech, threatening speech, and references to one’s organization, can improve psychological safety for Democrats without undermining psychological safety for Republicans. As political polarization permeates societies worldwide, maintaining the social-relational fabric of organizations will require policies that balance ideological demands. We conclude by discussing implications for psychological safety, “free” speech, and organizational censorship.
Funding: This project was funded by the University of Notre Dame’s Mendoza College of Business and Rooney Center for the Study of American Democracy.
Supplemental Material: The online appendix is available at https://doi.org/10.1287/orsc.2024.19506 .
We seek to better understand the demand side of vote buying: the conditions under which voters participate in, eschew, tolerate, or punish the exchange of targeted material benefits for votes. We ask whether voters perceive vote buying as substituting for local public goods provision in office, or whether they think that candidates who buy votes will excel at securing local public goods. Voters who place great value on future public goods may opt out of vote buying if they believe they are substitutes and punish vote‐buying candidates at the polls. We explore these issues in a nationwide survey in Nepal. Multiple survey experiments provide evidence that Nepali voters perceive vote buying and local public goods provision as substitutes. Voters who hold this view also express a preference for candidates who do not engage in vote buying, implying they prioritize public goods provision, although this latter result is not causally identified.
Members of the ethnic majority tend to view immigrants and ethnic minorities as less willing to contribute to the collective. Why is this the case? I argue that in Europe, ethnic attributes signal citizens’ socioeconomic resources, cultural values, and norm compliance and that these factors, rather than ethnic identities per se, explain why citizens are expected (not) to contribute. Through a novel conjoint experimental design in Denmark that manipulated respondents’ access to information about these different mechanisms, the argument finds support. First, in information-sparse environments, ethnic majority members expect that minority members contribute substantially less to the provision of public goods than majority members. Second, this ethnic bias is reduced by each of the three mechanisms and explained away once information on all three is available. This demonstrates that negative expectations toward minorities operate through multiple, complementary channels and that stereotype-countering information can reduce the majority-minority expectation gap.
We investigate how to design sanction regimes that enhance public support for sanctions in the sending countries. To this end, we conduct a conjoint experiment on quota samples in Poland and Germany, using the context of sanctions on Russia. We find that support for sanctions decreases as domestic costs increase. However, support can be bolstered through aid programs that mitigate these costs and through beneficial policy alternatives. Furthermore, we show that support for sanctions rises when the costs imposed on the target state increase and when the sanctioning coalition expands.
The International Monetary Fund (IMF) has recently expanded its policy scope to include a broader set of policies to promote green, inclusive, and resilient growth. How does this expansion affect the support for the IMF and its loans among the populations of borrowing countries? We conducted a pre-registered survey experiment with 2,694 respondents from three borrower countries—Argentina, Kenya, and Pakistan. We show that support for IMF programs increases by approximately 24 percent compared to traditional programs when the IMF includes good governance, anti-poverty, climate change, and gender equality measures in its programs. Our results imply that people do not uniformly reject the imposition of policies of global governance institutions but have well-defined preferences over policy measures. Our findings contribute to debates on the backlash against international institutions by highlighting that citizens are willing to accept sovereignty intrusion when they push for policy goals aligned with their policy preferences.
Perceptions of school quality affect many aspects of education policy and politics. Using two randomized survey experiments, we studied the factors that individuals use to evaluate school quality. Our surveys included two novel components: text analysis of open-ended responses and videos highlighting different perspectives on assessing school quality. While we found that individuals respond most strongly to test scores, we also found that a video explaining how standardized testing imperfectly reflects student learning led attentive respondents to reduce reliance on achievement status and to increase attention to growth. The results suggest that informational interventions may affect how individuals interpret school performance data. Yet, we also found evidence of political polarization and preferences for school characteristics not in standard reporting.
Across the world, women continue to be underrepresented in parliaments. As gatekeepers to candidate lists, party leaders are in a pivotal position to promote gender balance. But do party elites consider women’s underrepresentation when deciding who to nominate? Leveraging a large-scale conjoint experiment with 1,389 party elites in Austria, Germany, and Switzerland, we find that the more underrepresented women are in candidate lists, the better the chances of women aspirants. Awareness of women’s underrepresentation influences selectors for whom promoting gender equality may be a less crystallized priority: centrists and men. Women’s underrepresentation also reinforces preferences for women aspirants among those for whom gender equality may be a core value (left-wing and women selectors) but does not affect those for whom opposing gender equality may bring electoral advantages (right-wing party elites). Our findings shed light on the potential role that signalling underrepresentation may have on party elites’ selection of women aspirants.
Political elites express their ideological positions on contentious issues across various arenas in the public sphere. Social science research often relies on data extracted from various media or political and administrative sources, as well as surveys that are administered directly with the political actor. Although some studies compare ideology across different sources, few systematically analyze how political actors adjust their ideological messaging to the audiences in the respective communication arenas and how such changes are associated with systematic bias in data sources. This paper uses a unique dataset combining climate policy belief observations from three arenas—social media, Congressional testimony, and surveys—on identical ideological variables and during the same time period. We apply item response theory to understand how responses differ by arena and find that ideological communication on Twitter is most left-leaning, Congressional testimony is most right-leaning, and surveys, the data source with the smallest potential arena effect, is in the middle. We also find that actors with strong ideological leaning moderate their positions on social media and in Congress. These findings enhance our understanding of strategic communication depending on audience context and inform social research on biases when analyzing specific data sources.
What determines the legitimacy of aspiring rulers? Questions about support of the governed are central to theories of state-building and political order. Millions worldwide live under the influence of competing armed groups, yet we know little about how people in these contexts make comparative assessments of would-be rulers. We theorize how local norms, social networks, and the provision of goods and services influence these comparative judgments. We report results from a conjoint survey experiment in Colombia among nearly 2,400 respondents across 54 municipalities contested by multiple armed groups. Armed groups that take community norms into account and those that involve local leaders in decision-making are judged less negatively. Additionally, providing services and limiting violence both reduce negative evaluations of armed groups. These findings help us understand dimensions of political legitimacy under limited statehood and the effects of governance on civilian attitudes in areas of competition.
We provide evidence that there are substantial racial and gender gaps among lobbyists. These gender and racial differences are also greater among conservative leaning groups. However, we show, these gaps are decreasing over time. Does demand for minority and female lobbyists play a role in these trends? Although previous work has highlighted the relative scarcity of women and minorities in positions leading to the lobbyist profession (supply), we know less about whether interest groups are interested in hiring qualified women and minorities for such positions (demand). Using a conjoint experiment embedded in a survey of individuals involved in hiring lobbyists, we find greater demand for female and minority lobbyists than for their male and White counterparts, especially among ideological liberals. Our work shows that the lobbying industry does not appear to discount the candidacies of potential female and minority lobbyists.
Individuals in developing countries are the ultimate end users of foreign aid. While the international donor community has emphasized the importance of aligning aid with recipient countries’ preferences, the literature on public opinion and foreign aid has remained largely focused on donors. Using an original conjoint experiment conducted in seven developing countries, we examine the determinants of public attitudes toward foreign aid in recipient countries. We find that the characteristics of donor countries and foreign aid projects significantly influence recipient attitudes, often more than the size of the aid packages themselves. Individuals in recipient countries consistently prefer aid from democracies and donors with transparent aid agencies, as well as aid delivered by international organizations rather than directly from donor countries’ aid agencies. These findings underscore the importance of multilateral aid agencies in aligning the preferences of donors and recipients.
What does it mean to be a good citizen, and who can fulfill the obligations of citizenship? There is widespread agreement on how individuals understand how to perform good citizenship, with emphasis on behavioral and liberal democratic norms, but little work on who they think can or should perform it. It also remains unclear how growing polarization challenges this consensus, where divisions are reified along partisan lines. This paper uses a conjoint experiment in the United States, United Kingdom, and Germany to explore attributes and characteristics of good citizenship. It finds behavioral norms and liberal values, from voting to tolerance, remain essential to good citizenship across democracies. Yet, we observe variation by partisanship over who can be a good citizen, where right-wing party voters prefer Christian over Muslims and native-born over naturalized citizens in all three cases. It concludes by discussing the consequences of civic differentiation and hierarchy for democracy.
Can qualifications help candidates from historically marginalized groups win over out-group voters? We help answer this question with an original conjoint experiment fielded on a face-to-face survey conducted in three Afghan provinces between 2016 and 2017. The conjoint asked over 2,400 Afghan respondents to rank and choose between profiles of hypothetical candidates with varying gender, ethnic and educational attributes. We explore whether candidate qualifications – as measured by educational attainment – can amplify respondents’ support for hypothetical candidates from two social groups traditionally underrepresented in Afghan politics: women and Hazaras, a predominately Shi’a ethnic minority group. We find that higher qualifications consistently increase male (non-Hazara) respondents’ ranking and likelihood of choosing profiles with female (Hazara) candidates. These gains, however, do not completely offset male (non-Hazara) respondents’ in-group biases. Qualifications help but are not enough to dismantle voter preferences for in-group candidates.
Research on perceptions of deservingness has so far mainly focused on cash benefits. We correct this imbalance by studying public attitudes towards access to social investment services, specifically, subsidised childcare. Based on the well-established corpus of deservingness research and recent work on varieties of social investment, we expect the traditional deservingness criteria to matter and for results to vary across countries. To test our argument, we rely on an original survey experiment conducted in 2021 in six Western countries (Denmark, Sweden, Germany, Switzerland, the UK and the USA). We find that cross-national patterns of responses broadly reflect the categorisation of three types of social investments, that is, inclusive, stratified and targeted social investment. Further, we find that some of the well-known determinants of deservingness perceptions play an important role in the attribution of priorities of parents who need childcare: most clearly need and identity (operationalised with the parents’ nationality and their length of residency). This is true in all six countries covered. We conclude that patterns of deservingness perceptions to subsidised childcare services are determined by a mix of institutional factors (that differ across welfare regimes) and more fundamental attitudes towards helping those in need.
This study explores public preferences for algorithmic and human decision-makers (DMs) in high-stakes contexts, how these preferences are shaped by performance metrics, and whether public evaluations of performance differ depending on the type of DM. Leveraging a conjoint experimental design, approximately 9,000 respondents chose between pairs of DM profiles in two high-stakes scenarios: pretrial release decisions and bank loan approvals. The profiles varied by type (human vs. algorithm) and three metrics—defendant crime rate/loan default rate, false positive rate (FPR) among white defendants/applicants, and FPR among minority defendants/applicants—as well as an implicit fairness metric defined by the absolute difference between the two FPRs. The results show that efficiency was the most important performance metric in the respondents’ evaluation of DMs, while fairness was the least prioritized. This finding is robust across both scenarios, key subgroups of respondents (e.g. by race and political party), and across the DM type under evaluation. Additionally, even when controlling for performance, we find an average preference for human DMs over algorithmic ones, though this preference varied significantly across respondents. Overall, these findings show that while respondents differ in their preferences over DM type, they are generally consistent in the performance metrics they desire.
Amidst concerns about replicability but also thanks to the professionalisation of labs, the rise of pre-registration, the switch to online experiments, and enhanced computational power, experimental economics is undergoing rapid changes. They all call for efficient designs and data analysis, that is, they require that, given the constraints on participants' time, experiments provide as rich information as possible. In this Element the authors explore some ways in which this goal may be reached.
Why do elites in low‐ and middle‐income countries (LMICs) favor some foreign aid projects and partners over others? Research on the “aid curse” and Chinese development finance suggests elites should prefer aid that can be easily captured, with few conditionalities, regulations, or transparency requirements. We administer a conjoint survey experiment across 141 LMICs to elicit the aid preferences of elites who are uniquely close to development policy debates. Perhaps unsurprisingly, we find that elites favor larger over smaller projects, grants over loans, and transportation infrastructure projects over initiatives focused on civil society or tax collection capacity. But contrary to the aid curse theory, elites also prefer projects with transparent terms and labor, corruption, and environmental regulations, and are at worst indifferent towards good governance conditionalities. These preferences hold even in corrupt and autocratic countries and even among high‐level government officials who might be expected to favor “no‐strings‐attached” aid regimes.
Any peaceful resolution to the Cross-Strait conflict between China and Taiwan is likely to be composed of give-and-take between the two sides, and any agreement will ultimately require the support of Taiwan’s citizens, by whom Taiwan’s leaders are held accountable. Yet little is known about the actual tradeoffs Taiwanese citizens are willing to make in pursuit of an agreement in a time of peace. In conjoint experiments fielded in Taiwan in April 2022, we present Taiwanese respondents with hypothetical Cross-Strait agreements that randomly assign economic, security, and sovereignty-related concessions by the two sides. We find that support for agreements is conditioned by the concessions included, with sovereignty-related or security-related concessions by Taiwan lowering support, and sovereignty or security-related concessions by China increasing support. Additionally, respondents indicate that neither US support nor PRC threats significantly influence support for agreements, and that Taiwan's acceptance of an agreement substantially reduces the perceived likelihood of conflict.
This study explores how race impacts the legitimacy of international organizations (IOs). Specifically, we examine whether the representation of Black people in IO leadership positions influences perceptions of IO legitimacy among Black and white individuals. To do so, we fielded seven survey experiments in two racially diverse countries, South Africa and the United States, and three experiments in one predominantly Black country, Kenya. Our experiments were designed to distinguish the effects of an IO leader's race from their region of origin. We find that Black IO leadership enhances perceptions of institutional legitimacy among Black citizens, but does not strongly influence the legitimacy perceptions of their white counterparts. Our findings suggest that improving the representation of historically marginalized racial groups within IOs can enhance their popular legitimacy.
How do people around the world define democracy? Answering this question is critical as countries face democratic backsliding and authoritarian governments promote alternative notions of democracy. Indeed, some scholars argue that people from different backgrounds understand democracy differently. By contrast, we discovered very consistent views about what constitutes a “democratic” country from conjoint survey experiments conducted in Egypt, India, Italy, Japan, Thailand, and the United States. Across countries ( N = 6150) and diverse subgroups within countries, people similarly emphasized free and fair elections and civil liberties as being the key determinants of democracy. Countries that produce desirable social and economic outcomes are also considered more democratic, but these and other factors exert a smaller and less consistent effect than elections and civil liberties.
While gender quota and parity laws are increasingly popular worldwide, their introduction often causes controversy. Thus far, we lack an understanding of how the framing of these measures affects public opinion. We conducted a survey experiment in the UK and France (combined N = 2677) to identify the causal effect of framing on levels of support for the policy and potential backlash against women candidates. Comparing (1) gender quotas to increase women’s underrepresentation and (2) gender parity laws to achieve gender balance, we find that overall levels of support are greater than opposition in both countries. Parity is more supported than quotas in France, but no such framing effects emerge in the UK. Respondents’ gender also matters, with men less supportive of both measures than women. We find no evidence that either type of positive action measure increases backlash in the form of reduced support for hypothetical women candidates running under such measures.
Eliciting honest answers to sensitive questions is frustrated if subjects withhold the truth for fear that others will judge or punish them. The resulting bias is commonly referred to as social desirability bias, a subset of what we label sensitivity bias. We make three contributions. First, we propose a social reference theory of sensitivity bias to structure expectations about survey responses on sensitive topics. Second, we explore the bias-variance trade-off inherent in the choice between direct and indirect measurement technologies. Third, to estimate the extent of sensitivity bias, we meta-analyze the set of published and unpublished list experiments (a.k.a., the item count technique) conducted to date and compare the results with direct questions. We find that sensitivity biases are typically smaller than 10 percentage points and in some domains are approximately zero.
Conjoint experiments are popular, but there is a paucity of research on respondents’ underlying decision-making processes. We leverage eye-tracking methodology and a series of conjoint experiments, administered to university students and local community members, to examine how respondents process information in conjoint surveys. There are two main findings. First, attribute importance measures inferred from the stated choice data are correlated with attribute importance measures based on eye movement. This validation test supports the interpretation of common conjoint metrics, such as average marginal component effects (AMCEs), as measures of attribute importance. Second, when we experimentally increase the number of attributes and profiles in the conjoint table, respondents view a larger absolute number of cells but a smaller fraction of the total cells displayed. Moving from two to three profiles, respondents search more within-profile, rather than within-attribute, to build summary evaluations. However, respondents’ stated choices remain fairly stable regardless of the number of attributes and profiles in the conjoint table. Together, these patterns speak to the robustness of conjoint experiments and are consistent with a bounded rationality mechanism. Respondents adapt to complexity by selectively incorporating relevant new information to focus on important attributes, while ignoring less relevant information to reduce cognitive processing costs.
Amazon's Mechanical Turk is widely used for data collection; however, data quality may be declining due to the use of virtual private servers to fraudulently gain access to studies. Unfortunately, we know little about the scale and consequence of this fraud, and tools for social scientists to detect and prevent this fraud are underdeveloped. We first analyze 38 studies and show that this fraud is not new, but has increased recently. We then show that these fraudulent respondents provide particularly low-quality data and can weaken treatment effects. Finally, we provide two solutions: an easy-to-use application for identifying fraud in the existing datasets and a method for blocking fraudulent respondents in Qualtrics surveys.
Conjoint experiments are popular, but there is a paucity of research on respondents' underlying decision-making processes. We leverage eye-Tracking methodology and a series of conjoint experiments, administered to university students and local community members, to examine how respondents process information in conjoint surveys. There are two main findings. First, attribute importance measures inferred from the stated choice data are correlated with attribute importance measures based on eye movement. This validation test supports the interpretation of common conjoint metrics, such as average marginal component effects (AMCEs), as measures of attribute importance. Second, when we experimentally increase the number of attributes and profiles in the conjoint table, respondents view a larger absolute number of cells but a smaller fraction of the total cells displayed. Moving from two to three profiles, respondents search more within-profile, rather than within-Attribute, to build summary evaluations. However, respondents' stated choices remain fairly stable regardless of the number of attributes and profiles in the conjoint table. Together, these patterns speak to the robustness of conjoint experiments and are consistent with a bounded rationality mechanism. Respondents adapt to complexity by selectively incorporating relevant new information to focus on important attributes, while ignoring less relevant information to reduce cognitive processing costs.
Mitigating climate change requires countries to provide a global public good. This means that the domestic cleavages underlying mass attitudes toward international climate policy are a central determinant of its provision. We argue that the industry-specific costs of emission abatement and internalized social norms help explain support for climate policy. To evaluate our predictions we develop novel measures of industry-specific interests by cross-referencing individuals’ sectors of employment and objective industry-level pollution data and employing quasi-behavioral measures of social norms in combination with both correlational and conjoint-experimental data. We find that individuals working in pollutive industries are 7 percentage points less likely to support climate co-operation than individuals employed in cleaner sectors. Our results also suggest that reciprocal and altruistic individuals are about 10 percentage points more supportive of global climate policy. These findings indicate that both interests and norms function as complementary explanations that improve our understanding of individual policy preferences.
Significance
Estimating and analyzing heterogeneous treatment effects is timely, yet challenging. We introduce a unifying framework for many conditional average treatment effect estimators, and we propose a metalearner, the X-learner, which can adapt to structural properties, such as the smoothness and sparsity of the underlying treatment effect. We present its favorable properties, using theory and simulations. We apply it, using random forests, to two field experiments in political science, where it is shown to be easy to use and to produce results that are interpretable.
Significance
Little evidence exists on whether preferences about hypothetical choices measured in a survey experiment are driven by the same structural determinants of the actual choices made in the real world. This study answers this question using a natural experiment as a behavioral benchmark. Comparing the results from conjoint and vignette experiments on which attributes of hypothetical immigrants generate support for naturalization with the outcomes of closely corresponding referendums in Switzerland, we find that the effects estimated from the surveys match the effects of the same attributes in the behavioral benchmark remarkably well. We also find that seemingly subtle differences in survey designs can produce significant differences in performance. Overall, the paired conjoint design performs the best.
Effective climate mitigation requires international cooperation, and these global efforts need broad public support to be sustainable over the long run. We provide estimates of public support for different types of climate agreements in France, Germany, the United Kingdom, and the United States. Using data from a large-scale experimental survey, we explore how three key dimensions of global climate cooperation-costs and distribution, participation, and enforcement-affect individuals' willingness to support these international efforts. We find that design features have significant effects on public support. Specifically, our results indicate that support is higher for global climate agreements that involve lower costs, distribute costs according to prominent fairness principles, encompass more countries, and include a small sanction if a country fails to meet its emissions reduction targets. In contrast to well-documented baseline differences in public support for climate mitigation efforts, opinion responds similarly to changes in climate policy design in all four countries. We also find that the effects of institutional design features can bring about decisive changes in the level of public support for a global climate agreement. Moreover, the results appear consistent with the view that the sensitivity of public support to design features reflects underlying norms of reciprocity and individuals' beliefs about the potential effectiveness of specific agreements.
Marketing ethics and social responsibility are inherently controversial, and years of research continue to present conflicts and challenges for marketers on the value of a socially responsible approach to marketing activities. This article examines whether or not consumers care about ethical behaviour, and investigates the effect of good and bad ethical conduct on consumer purchase behaviour. Through focus group discussions it becomes clear that although we are more sophisticated as consumers today, this does not necessarily translate into behaviour which favours ethical companies and punishes unethical firms. The article concludes by some thoughts on how marketers might encourage consumers to engage in positive purchase behaviour in favour of ethical marketing.
Vignette studies use short descriptions of situations or persons (vignettes) that are usually shown to respondents within surveys in order to elicit their judgments about these scenarios. By systematically varying the levels of theoretically important vignette characteristics a large population of different vignettes is typically available – too large to be presented to each respondent. Therefore, each respondent gets only a subset of vignettes. These subsets may either be randomly selected in following the tradition of the factorial survey or systematically selected according to an experimental design. We show that these strategies in selecting vignette sets have strong implications for the analysis and interpretation of vignette data. Random selection strategies result in a random confounding of effects and heavily rely on the assumption of no interaction effects. In contrast, experimental strategies systematically confound interaction effects with main or set effects, thereby preserving a meaningful interpretation of main and important interaction effects. Using a pilot study on attitudes toward immigrants we demonstrate the implementation and analysis of a confounded factorial design.
Research has revealed negative associations between religiosity and alcohol consumption. Given these associations, the aim of the current research was to evaluate whether the order of assessing each construct might affect subsequent reports of the other. The present research provided an experimental evaluation of response biases of self-reported religiosity and alcohol consumption based on order of assessment. Participants (N = 301 undergraduate students) completed an online survey. Based on random assignment, religiosity was assessed either before or after questions regarding recent alcohol consumption. Social desirability bias was also measured. Results revealed a priming effect such that participants who answered questions about their religiosity prior to their alcohol consumption reported fewer drinks on their peak drinking occasions, drinking less on typical occasions, and drinking less frequently, even when controlling for social desirability and for the significant negative associations between their own religiosity and drinking. In contrast, assessment order was not significantly associated with religiosity. Results indicate priming religion results in reporting lower, but potentially more accurate, levels of health risk behaviors and that these effects are not simply the result of socially desirable responding. Results are interpreted utilizing several social-cognitive theories and suggest that retrospective self-reports of drinking may be more malleable than self-descriptions of religiosity. Implications and future directions are discussed. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
I survey the literature post Ledyard (Handbook of Experimental Economics, ed. by J. Kagel, A. Roth, Chap.2, Princeton, Princeton
University Press, 1995) on three related issues in linear public goods experiments: (1) conditional cooperation; (2) the role of costly monetary
punishments in sustaining cooperation and (3) the sustenance of cooperation via means other than such punishments. Many participants
in laboratory public goods experiments are “conditional cooperators” whose contributions to the public good are positively
correlated with their beliefs about the average group contribution. Conditional cooperators are often able to sustain high
contributions to the public good through costly monetary punishment of free-riders but also by other mechanisms such as expressions
of disapproval, advice giving and assortative matching.
KeywordsPublic goods–Conditional cooperation–Monetary punishments–Non-monetary punishments–Moral suasion–Sorting
Psychologists have worried about the distortions introduced into standardized personality measures by social desirability bias. Survey researchers have had similar concerns about the accuracy of survey reports about such topics as illicit drug use, abortion, and sexual behavior. The article reviews the research done by survey methodologists on reporting errors in surveys on sensitive topics, noting parallels and differences from the psychological literature on social desirability. The findings from the survey studies suggest that misreporting about sensitive topics is quite common and that it is largely situational. The extent of misreporting depends on whether the respondent has anything embarrassing to report and on design features of the survey. The survey evidence also indicates that misreporting on sensitive topics is a more or less motivated process in which respondents edit the information they report to avoid embarrassing themselves in the presence of an interviewer or to avoid repercussions from third parties.
Conjoint analysis has become popular among social scientists for measuring multidimensional preferences. When analyzing such experiments, researchers often focus on the average marginal component effect (AMCE), which represents the causal effect of a single profile attribute while averaging over the remaining attributes. What has been overlooked, however, is the fact that the AMCE critically relies upon the distribution of the other attributes used for the averaging. Although most experiments employ the uniform distribution, which equally weights each profile, both the actual distribution of profiles in the real world and the distribution of theoretical interest are often far from uniform. This mismatch can severely compromise the external validity of conjoint analysis. We empirically demonstrate that estimates of the AMCE can be substantially different when averaging over the target profile distribution instead of uniform. We propose new experimental designs and estimation methods that incorporate substantive knowledge about the profile distribution. We illustrate our methodology through two empirical applications, one using a real-world distribution and the other based on a counterfactual distribution motivated by a theoretical consideration. The proposed methodology is implemented through an open-source software package.
Debate persists on whether voters hold politicians accountable for corruption. Numerous experiments have examined whether informing voters about corrupt acts of politicians decreases their vote share. Meta-analysis demonstrates that corrupt candidates are punished by zero percentage points across field experiments, but approximately 32 points in survey experiments. I argue this discrepancy arises due to methodological differences. Small effects in field experiments may stem partially from weak treatments and noncompliance, and large effects in survey experiments are likely from social desirability bias and the lower and hypothetical nature of costs. Conjoint experiments introduce hypothetical costly trade-offs, but it may be best to interpret results in terms of realistic sets of characteristics rather than marginal effects of particular characteristics. These results suggest that survey experiments may provide point estimates that are not representative of real-world voting behavior. However, field experimental estimates may also not recover the “true” effects due to design decisions and limitations.
We propose generalized random forests, a method for nonparametric statistical estimation based on random forests (Breiman [Mach. Learn. 45 (2001) 5–32]) that can be used to fit any quantity of interest identified as the solution to a set of local moment equations. Following the literature on local maximum likelihood estimation, our method considers a weighted set of nearby training examples; however, instead of using classical kernel weighting functions that are prone to a strong curse of dimensionality, we use an adaptive weighting function derived from a forest designed to express heterogeneity in the specified quantity of interest. We propose a flexible, computationally efficient algorithm for growing generalized random forests, develop a large sample theory for our method showing that our estimates are consistent and asymptotically Gaussian and provide an estimator for their asymptotic variance that enables valid confidence intervals. We use our approach to develop new methods for three statistical tasks: nonparametric quantile regression, conditional average partial effect estimation and heterogeneous treatment effect estimation via instrumental variables. A software implementation, grf for R and C++, is available from CRAN.
Many scientific and engineering challenges---ranging from personalized
medicine to customized marketing recommendations---require an understanding of
treatment effect heterogeneity. In this paper, we develop a non-parametric
causal forest for estimating heterogeneous treatment effects that extends
Breiman's widely used random forest algorithm. Given a potential outcomes
framework with unconfoundedness, we show that causal forests are pointwise
consistent for the true treatment effect, and have an asymptotically Gaussian
and centered sampling distribution. We also discuss a practical method for
constructing asymptotic confidence intervals for the true treatment effect that
are centered at the causal forest estimates. Our theoretical results rely on a
generic Gaussian theory for a large family of random forest algorithms that, to
our knowledge, is the first set of results that allows any type of random
forest, including classification and regression forests, to be used for
provably valid statistical inference. In experiments, we find causal forests to
be substantially more powerful than classical methods based on nearest-neighbor
matching, especially as the number of covariates increases.
Although politicians’ personal attributes are an important component of elections and representation, few studies have rigorously investigated which attributes are most relevant in shaping voters’ preferences for politicians, or whether these preferences vary across different electoral system contexts. We investigate these questions with a conjoint survey experiment using the case of Japan’s mixed-member bicameral system. We find that the attributes preferred by voters are not entirely consistent with the observed attributes of actual politicians. Moreover, voters’ preferences do not vary when asked to consider representation under different electoral system contexts, whereas the observed attributes of politicians do vary across these contexts. These findings point to the role of factors beyond voters’ sincere preferences, such as parties’ recruitment strategies, the effect of electoral rules on the salience of the personal vote, and the availability of different types of politicians, in determining the nature of representation.
This paper theorizes three forms of bias that might limit women's representation: outright hostility, double standards, and a double bind whereby desired traits present bigger burdens for women than men. We examine these forms of bias using conjoint experiments derived from several original surveys—a population survey of American voters and two rounds of surveys of American public officials. We find no evidence of outright discrimination or of double standards. All else equal, most groups of respondents prefer female candidates, and evaluate men and women with identical profiles similarly. But on closer inspection, all is not equal. Across the board, elites and voters prefer candidates with traditional household profiles such as being married and having children, resulting in a double bind for many women. So long as social expectations about women's familial commitments cut against the demands of a full-time political career, women are likely to remain underrepresented in politics.
How does spatial scale affect support for public policy? Does supporting housing citywide but “Not In My Back Yard” (NIMBY) help explain why housing has become increasingly difficult to build in once-affordable cities? I use two original surveys to measure how support for new housing varies between the city scale and neighborhood scale. Together, an exit poll of 1,660 voters during the 2015 San Francisco election and a national survey of over 3,000 respondents provide the first experimental measurements of NIMBYism. While homeowners are sensitive to housing’s proximity, renters typically do not express NIMBYism. However, in high-rent cities, renters demonstrate NIMBYism on par with homeowners, despite continuing to support large increases in the housing supply citywide. These scale-dependent preferences not only help explain the deepening affordability crisis, but show how institutions can undersupply even widely supported public goods. When preferences are scale dependent, the scale of decision-making matters.
An increase in the amount of packaging consumed in the U.S. has put pressure on companies to take responsibility for the entire life-cycle of their product. This study uses discrete choice experiments to assess consumer willingness to pay (WTP) for packaging materials and recyclability of a beverage product. A between-subject design was used to analyze the effectiveness of indirect questioning in addressing issues of social desirability bias as well as the effects of information on consumer behavior. Consumer WTP for packaging material was highest for plastic packaging, followed by glass, carton and aluminum. Our empirical analysis reveals that indirect questioning results in WTP values for packaging recyclability that are 60% lower than those obtained from direct questioning. We find that information from a video treatment had a significant and positive effect on consumer preferences and demand for packaging recyclability. Our results suggest that more scrutiny should be placed on studies that do not address social desirability bias when evaluating recycling behavior.
In this paper we propose methods for estimating heterogeneity in causal effects in experimental and observational studies and for conducting hypothesis tests about the magnitude of differences in treatment effects across subsets of the population. We provide a data-driven approach to partition the data into subpopulations that differ in the magnitude of their treatment effects. The approach enables the construction of valid confidence intervals for treatment effects, even with many covariates relative to the sample size, and without "sparsity" assumptions. We propose an "honest" approach to estimation, whereby one sample is used to construct the partition and another to estimate treatment effects for each subpopulation. Our approach builds on regression tree methods, modified to optimize for goodness of fit in treatment effects and to account for honest estimation. Our model selection criterion anticipates that bias will be eliminated by honest estimation and also accounts for the effect of making additional splits on the variance of treatment effect estimates within each subpopulation. We address the challenge that the "ground truth" for a causal effect is not observed for any individual unit, so that standard approaches to cross-validation must be modified. Through a simulation study, we show that for our preferred method honest estimation results in nominal coverage for 90% confidence intervals, whereas coverage ranges between 74% and 84% for nonhonest approaches. Honest estimation requires estimating the model with a smaller sample size; the cost in terms of mean squared error of treatment effects for our preferred method ranges between 7-22%.
Survey experiments have become a central methodology across the social sciences. Researchers can combine experiments’ causal power with the generalizability of population-based samples. Yet, due to the expense of population-based samples, much research relies on convenience samples (e.g. students, online opt-in samples). The emergence of affordable, but non-representative online samples has reinvigorated debates about the external validity of experiments. We conduct two studies of how experimental treatment effects obtained from convenience samples compare to effects produced by population samples. In Study 1, we compare effect estimates from four different types of convenience samples and a population-based sample. In Study 2, we analyze treatment effects obtained from 20 experiments implemented on a population-based sample and Amazon's Mechanical Turk (MTurk). The results reveal considerable similarity between many treatment effects obtained from convenience and nationally representative population-based samples. While the results thus bolster confidence in the utility of convenience samples, we conclude with guidance for the use of a multitude of samples for advancing scientific knowledge.
About a half century ago, in 1965, Warner proposed the randomized response method as a survey technique to reduce potential bias due to nonresponse and social desirability when asking questions about sensitive behaviors and beliefs. This method asks respondents to use a randomization device, such as a coin flip, whose outcome is unobserved by the interviewer. By introducing random noise, the method conceals individual responses and protects respondent privacy. While numerous methodological advances have been made, we find surprisingly few applications of this promising survey technique. In this article, we address this gap by (1) reviewing standard designs available to applied researchers, (2) developing various multivariate regression techniques for substantive analyses, (3) proposing power analyses to help improve research designs, (4) presenting new robust designs that are based on less stringent assumptions than those of the standard designs, and (5) making all described methods available through open-source software. We illustrate some of these methods with an original survey about militant groups in Nigeria.
We examine the trade-offs associated with using Amazon.com's Mechanical Turk (MTurk) interface for subject recruitment. We first describe MTurk and its promise as a vehicle for performing low-cost and easy-to-field experiments. We then assess the internal and external validity of experiments performed using MTurk, employing a framework that can be used to evaluate other subject pools. We first investigate the characteristics of samples drawn from the MTurk population. We show that respondents recruited in this manner are often more representative of the U.S. population than in-person convenience samples-the modal sample in published experimental political science-but less representative than subjects in Internet-based panels or national probability samples. Finally, we replicate important published experimental work using MTurk samples. © The Author 2012. Published by Oxford University Press on behalf of the Society for Political Methodology. All rights reserved.
Much of what we know about the responses of voters to Black candidates and female candidates comes from experimental research. Yet the accuracy of experimental data can be threatened by the possibility that social desirability pressures contaminate self-reporting. We address this threat in a project that considers psychological approaches to reducing social desirability pressures. Offering participants the opportunity to explain their decisions about sensitive subjects, such as voting for a Black or female candidate, can lessen social desirability pressures. We analyze this approach across three commonly used samples: undergraduate, adult convenience, and adult national. Our results suggest that existing experimental research overestimates voter support for Black and female candidates, but these issues can be mitigated with the simple innovation presented here.
Survey questions asking about taboo topics such as sexual activities, illegal behaviour such as social fraud, or unsocial attitudes such as racism, often generate inaccurate survey estimates which are distorted by social desirability bias. Due to self-presentation concerns, survey respondents underreport socially undesirable activities and overreport socially desirable ones. This article reviews theoretical explanations of socially motivated misreporting in sensitive surveys and provides an overview of the empirical evidence on the effectiveness of specific survey methods designed to encourage the respondents to answer more honestly. Besides psychological aspects, like a stable need for social approval and the preference for not getting involved into embarrassing social interactions, aspects of the survey design, the interviewer’s characteristics and the survey situation determine the occurrence and the degree of social desirability bias. The review shows that survey designers could generate more valid data by selecting appropriate data collection strategies that reduce respondents’ discomfort when answering to a sensitive question.
The governance of natural resources used by many individuals in common is an issue of increasing concern to policy analysts. Both state control and privatization of resources have been advocated, but neither the state nor the market have been uniformly successful in solving common pool resource problems. After critiquing the foundations of policy analysis as applied to natural resources, Elinor Ostrom here provides a unique body of empirical data to explore conditions under which common pool resource problems have been satisfactorily or unsatisfactorily solved. Dr Ostrom uses institutional analysis to explore different ways - both successful and unsuccessful - of governing the commons. In contrast to the proposition of the 'tragedy of the commons' argument, common pool problems sometimes are solved by voluntary organizations rather than by a coercive state. Among the cases considered are communal tenure in meadows and forests, irrigation communities and other water rights, and fisheries.
Microarray studies yield data sets consisting of a large number of candidate predictors (genes) on a small number of observations (samples). When interest lies in predicting phenotypic class using gene expression data, often the goals are both to produce an accurate classifier and to uncover the predictive structure of the problem. Most machine learning methods, such as k-nearest neighbors, support vector machines, and neural networks, are useful for classification. However, these methods provide no insight regarding the covariates that best contribute to the predictive structure. Other methods, such as linear discriminant analysis, require the predictor space be substantially reduced prior to deriving the classifier. A recently developed method, random forests (RF), does not require reduction of the predictor space prior to classification. Additionally, RF yield variable importance measures for each candidate predictor. This study examined the effectiveness of RF variable importance measures in identifying the true predictor among a large number of candidate predictors. An extensive simulation study was conducted using 20 levels of correlation among the predictor variables and 7 levels of association between the true predictor and the dichotomous response. We conclude that the RF methodology is attractive for use in classification problems when the goals of the study are to produce an accurate classifier and to provide insight regarding the discriminative ability of individual predictor variables. Such goals are common among microarray studies, and therefore application of the RF methodology for the purpose of obtaining variable importance measures is demonstrated on a microarray data set.
The aim of the factorial survey approach is to uncover the social and individual structures of human judgements of social objects. By having respondents evaluate samples of vignettes (fictive descriptions), in which several factors describing the object of interest are simultaneously manipulated, this approach has a number of advantages over traditional social survey research. The aim of the present article is to provide an easy-to-follow overview of the various ways in which the approach has been applied within sociology between 1982 and 2006. The review, which is based on 106 articles published in central sociology journals, is organised into three different sections: “factorial survey applications”, “factorial survey designs” and “factorial survey analyses”.
Opinion research is beset by two major types of "artifactual" variance: huge amounts of overtime response instability and the common tendency for seemingly trivial changes in questionnaire form to affect the expression of attitudes. We propose a simple model that converts this anomalous "error variance" into sources of substantive insight into the nature of public opinion. The model abandons the conventional but implausible notion that most people possess opinions at the level of specificity of typical survey items--and instead assumes that most people are internally conflicted over most political issues--and that most respond to survey questions on the basis of whatever ideas are at the top of their heads at the moment of answering. Numerous empirical regularities are shown to be consistent with these assumptions.
The Hidden American Immigration Consensus: A Conjoint Analysis of Attitudes Toward Immigrants
- Hainmueller
Causal Inference in Conjoint Analysis: Understanding Multidimensional Choices via Stated Preference Experiments
- Hainmueller
Statistical Analysis of List Experiments
- Blair
Package grf: Generalized Random Forests.” Version 1.2.0, available at the Comprehensive R Archive Network
- J Tibshirani
Identifying voter Preferences for Politicians' Personal Attributes: A Conjoint Experiment in Japan
- T Incerti
"Identifying voter Preferences for Politicians' Personal
Attributes: A Conjoint Experiment in Japan." Political Science Research and Method ( ): -.
Incerti, T.
. "Corruption Information and Vote Share: A Meta-Analysis and Lessons for Experimental
Design." American Political Science Review ( ): -.