Teppei Yamamoto’s research while affiliated with Harvard University and other places

How can we elicit honest responses in surveys? Conjoint analysis has become a popular tool to address social desirability bias (SDB), or systematic survey misreporting on sensitive topics. However, there has been no direct evidence showing its suitability for this purpose. We propose a novel experimental design to identify conjoint analysis’s ability to mitigate SDB. Specifically, we compare a standard, fully randomized conjoint design against a partially randomized design where only the sensitive attribute is varied between the two profiles in each task. We also include a control condition to remove confounding due to the increased attention to the varying attribute under the partially randomized design. We implement this empirical strategy in two studies on attitudes about environmental conservation and preferences about congressional candidates. In both studies, our estimates indicate that the fully randomized conjoint design could reduce SDB for the average marginal component effect (AMCE) of the sensitive attribute by about two-thirds of the AMCE itself. Although encouraging, we caution that our results are exploratory and exhibit some sensitivity to alternative model specifications, suggesting the need for additional confirmatory evidence based on the proposed design.

One of the strongest findings across the sciences is that publication bias occurs. Of particular note is a “file drawer bias” where statistically significant results are privileged over nonsignificant results. Recognition of this bias, along with increased calls for “open science,” has led to an emphasis on replication studies. Yet, few have explored publication bias and its consequences in replication studies. We offer a model of the publication process involving an initial study and a replication. We use the model to describe three types of publication biases: (1) file drawer bias, (2) a “repeat study” bias against the publication of replication studies, and (3) a “gotcha bias” where replication results that run contrary to a prior study are more likely to be published. We estimate the model’s parameters with a vignette experiment conducted with political science professors teaching at Ph.D. granting institutions in the United States. We find evidence of all three types of bias, although those explicitly involving replication studies are notably smaller. This bodes well for the replication movement. That said, the aggregation of all of the biases increases the number of false positives in a literature. We conclude by discussing a path for future work on publication biases.

Does media choice cause polarization, or merely reflect it? We investigate a critical aspect of this puzzle: how partisan media contribute to attitude polarization among different groups of media consumers. We implement a new experimental design, called the Preference-Incorporating Choice and Assignment (PICA) design, that incorporates both free choice and forced exposure. We estimate jointly the degree of polarization caused by selective exposure and the persuasive effect of partisan media. Our design also enables us to conduct sensitivity analyses accounting for discrepancies between stated preferences and actual choice, a potential source of bias ignored in previous studies using similar designs. We find that partisan media can polarize both its regular consumers and inadvertent audiences who would otherwise not consume it, but ideologically-opposing media potentially also can ameliorate existing polarization between consumers. Taken together, these results deepen our understanding of when and how media polarize individuals.

Social and medical scientists are often concerned that the external validity of experimental results may be compromised because of heterogeneous treatment effects. If a treatment has different effects on those who would choose to take it and those who would not, the average treatment effect estimated in a standard randomized controlled trial (RCT) may give a misleading picture of its impact outside of the study sample. Patient preference trials (PPTs), where participants' preferences over treatment options are incorporated in the study design, provide a possible solution. In this paper, we provide a systematic analysis of PPTs based on the potential outcomes framework of causal inference. We propose a general design for PPTs with multi-valued treatments, where participants state their preferred treatments and are then randomized into either a standard RCT or a self-selection condition. We derive nonparametric sharp bounds on the average causal effects among each choice-based sub-population of participants under the proposed design. We also propose a sensitivity analysis for the violation of the key ignorability assumption sufficient for identifying the target causal quantity. The proposed design and methodology are illustrated with an original study of partisan news media and its behavioral impact. (194 words)

Although politicians’ personal attributes are an important component of elections and representation, few studies have rigorously investigated which attributes are most relevant in shaping voters’ preferences for politicians, or whether these preferences vary across different electoral system contexts. We investigate these questions with a conjoint survey experiment using the case of Japan’s mixed-member bicameral system. We find that the attributes preferred by voters are not entirely consistent with the observed attributes of actual politicians. Moreover, voters’ preferences do not vary when asked to consider representation under different electoral system contexts, whereas the observed attributes of politicians do vary across these contexts. These findings point to the role of factors beyond voters’ sincere preferences, such as parties’ recruitment strategies, the effect of electoral rules on the salience of the personal vote, and the availability of different types of politicians, in determining the nature of representation.

Representative democracy entails the aggregation of multiple policy issues by parties into competing bundles of policies, or “manifestos,” which are then evaluated holistically by voters in elections. This aggregation process obscures the multidimensional policy preferences underlying a voter’s single choice of party or candidate. We address this problem through a conjoint experiment based on the actual party manifestos in Japan’s 2014 House of Representatives election. By juxtaposing sets of issue positions as hypothetical manifestos and asking respondents to choose one, our study identifies the effects of specific positions on the overall assessment of manifestos, heterogeneity in preferences among subgroups of respondents, and the popularity ranking of manifestos. Our analysis uncovers important discrepancies between voter preferences and the portrayal of the election results by politicians and the media as providing a policy mandate to the Liberal Democratic Party, underscoring the potential danger of inferring public opinion from election outcomes alone.

In recent years, political and social scientists have made increasing use of conjoint survey designs to study decision-making. Here, we study a consequential question which researchers confront when implementing conjoint designs: how many choice tasks can respondents perform before survey satisficing degrades response quality? To answer the question, we run a set of experiments where respondents are asked to complete as many as 30 conjoint tasks. Experiments conducted through Amazon's Mechanical Turk and Survey Sampling International demonstrate the surprising robustness of conjoint designs, as there are detectable but quite limited increases in survey satisficing as the number of tasks increases. Our evidence suggests that in similar study contexts researchers can assign dozens of tasks without substantial declines in response quality.