ArticlePDF Available

Beyond the Limits of Survey Experiments: How Conjoint Designs Advance Causal Inference in Political Communication Research

Article

Beyond the Limits of Survey Experiments: How Conjoint Designs Advance Causal Inference in Political Communication Research

Abstract and Figures

This paper calls attention to what is arguably the most notable advancement in survey experiments over the last decade: conjoint designs. The benefit of conjoint design is its capacity to study and compare the causal effects of several dimensions simultaneously. Although survey experiments have long been a preferred method for assessing causal effects, the method falls short when studying multidimensional causal relations. Researchers face a trade-off between a lack of statistical power or a restriction in experimental conditions. Conjoint designs solve this problem by letting the researcher vary an indefinite number of factors in one experiment. This method is quickly gaining ground in social and political science but has yet to be widely practiced in political communication research. This article argues that conjoint designs are ideal for studying political communication effects and highlights the possible benefits of using and innovating conjoint designs in political communication research. We make available sample scripts and demonstrate the value of this methodological technique through empirical examples of trust in news media and selective exposure to political news.
Content may be subject to copyright.
Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=upcp20
Political Communication
ISSN: 1058-4609 (Print) 1091-7675 (Online) Journal homepage: http://www.tandfonline.com/loi/upcp20
Beyond the Limits of Survey Experiments: How
Conjoint Designs Advance Causal Inference in
Political Communication Research
Erik Knudsen & Mikael Poul Johannesson
To cite this article: Erik Knudsen & Mikael Poul Johannesson (2018): Beyond the Limits of
Survey Experiments: How Conjoint Designs Advance Causal Inference in Political Communication
Research, Political Communication, DOI: 10.1080/10584609.2018.1493009
To link to this article: https://doi.org/10.1080/10584609.2018.1493009
View supplementary material
Published online: 13 Sep 2018.
Submit your article to this journal
Article views: 105
View Crossmark data
Beyond the Limits of Survey Experiments: How
Conjoint Designs Advance Causal Inference in
Political Communication Research
ERIK KNUDSEN and MIKAEL POUL JOHANNESSON
This paper calls attention to what is arguably the most notable advancement in survey
experiments over the last decade: conjoint designs. The benefit of conjoint design is its
capacity to study and compare the causal effects of several dimensions simultaneously.
Although survey experiments have long been a preferred method for assessing causal
effects, the method falls short when studying multidimensional causal relations.
Researchers face a trade-off between a lack of statistical power or a restriction in
experimental conditions. Conjoint designs solve this problem by letting the researcher
vary an indefinite number of factors in one experiment. This method is quickly gaining
ground in social and political science but has yet to be widely practiced in political
communication research. This article argues that conjoint designs are ideal for study-
ing political communication effects and highlights the possible benefits of using and
innovating conjoint designs in political communication research. We make available
sample scripts and demonstrate the value of this methodological technique through
empirical examples of trust in news media and selective exposure to political news.
Keywords survey experiment, causal inference, selective exposure, conjoint analysis,
trust in media
Nearly a quarter-century has passed since Bartels (1993, p. 267) claimed that the state of the
research on media effects is one of the most notable embarrassments of modern social science.
The cure, he said, was experimental designs and carefulness in measurement. The field of
communication science has evolved considerably since. Now, researchers no longer ask
whether communications shape opinions, but rather when and how(Druckman & Leeper,
2012, p.875). This evolution is at least partly driven by the steady pace of methodological
innovation.
A prime example is survey experiments (Sniderman, 2011), now a preferred method
for testing causal effects (Arceneaux, 2010). Survey experiments elegantly combine the
Erik Knudsen is a postdoctoral research fellow at the Department of Information Science and
Media Studies, and in the Digital Social Science Core Facility (DIGSSCORE), University of Bergen.
Email: Erik.Knudsen@uib.no.
Mikael P. Johannesson is a PhD candidate the Department of Comparative Politics, and in the
Digital Social Science Core Facility (DIGSSCORE), University of Bergen. Email: Mikael.
Johannesson@uib.no.
Address correspondence to Erik Knudsen, Department of Information Science and Media Studies,
University of Bergen, Fosswinckelsgate 6, 7802, Bergen, Norway. E-mail:erik.knudsen@uib.no
Color versions of one or more of the figures in the article can be found online at www.
tandfonline.com/upcp.
Political Communication, 00:113, 2018
Copyright © 2018 Taylor & Francis Group, LLC
ISSN: 1058-4609 print / 1091-7675 online
DOI: https://doi.org/10.1080/10584609.2018.1493009
1
internal validity of experiments with the external validity of representative surveys.
Standard survey experiments, however, can vary only a small number of factors.
This limitation is pertinent for political communication research, as the study of
political communication is to a large degree a study of multidimensional causal
relations. Understanding when and how messages affect the preferences and choices
of audiences, voters, political actors, and government officials means navigating a
jungle of conditioning and countervailing effects. With standard survey experiments,
the only way to cover ground is with persistence and perseverance, testing one isolated
factor at a time. The problem with this is the high fixed cost of conducting even one
survey, let alone tens or hundreds.
Our aim is to call attention to an alternative approach to this problem: conjoint
designs. Conjoint designs let the researcher vary an indefinite number of factors in one
experiment, so that researchers can include more factors and easily study multidimensional
choices. The method solves key problems researchers face when studying multidimen-
sional preferences with survey experiments: the trade-off between statistical power and the
desire to employ many experimental conditions.
Although conjoint designs have become increasingly popular in the social sciences
in the past several years, the method is (with notable exceptions: Helfer, 2016;
Mummolo, 2016) yet to be widely practiced in political communication research.
This is surprising, as this methodological advancement can help answer foundational
questions in political communication that hinge on the opportunity to study multi-
dimensional causal relations. As the benefits for social science research and general
assumptions of causal inference with conjoint experiments are thoroughly and formally
described elsewhere (Hainmueller, Hopkins, & Yamamoto, 2014), we focus on the
application to political communication and demonstrate how the conjoint technique can
be innovated and tailored to study phenomena that are specific to political commu-
nication research. We show that political communication requires different considera-
tions than other fields, and that such considerations need to be addressed when
designing conjoint analyses. We contribute with empirical demonstrations of the
method and offer sample scripts researchers can use to analyze and innovate conjoint
designs in their own surveys. We proceed by detailing the technique, demonstrating
some of its potential benefits for political communication research, and suggest how
the method can be applied in future political communication studies.
The Renaissance of Conjoint Design
Conjoint designs, also called vignette analysis or factorial surveys, were introduced in the
1970s in the fields of marketing research (Green & Rao, 1971) and sociology (Jasso &
Rossi, 1977) but did not become popular in fields such as political science until recently.
Due to the meticulous and imaginative work of Hainmueller and his colleagues, conjoint
designs are now experiencing a renaissance. They established a solid methodological
footing, including causal inference in conjoint analysis under the Neyman-Rubin model
(Hainmueller et al., 2014; Neyman, 1923; Rubin, 1974), validating conjoint experiments
by thoroughly comparing them with actual decisions made by voters in the real world
(Hainmueller, Hangartner, & Yamamoto, 2015), and even systematically testing the effect
of different design choices that are central to conjoint experiments (Bansak, Hainmueller,
Hopkins, & Yamamoto, 2017,2018; Hainmueller et al., 2014).
2 Erik Knudsen and Mikael Poul Johannesson
The Added Benefits of the Conjoint Design
Compared to the traditional survey experiment, conjoint designs strengths lie in its
capacity to include more factors and to study multidimensional choices. As an example,
consider a study that identifies how certain attributes of a newspaper affect its credibility.
One highly important factor could be the newspapers distribution modethat is, testing
whether people trust offline newspapers more than their online counterparts.
We then have two issues we need to overcome. First, the effect of the distribution
mode is ambiguous. For instance, if we, following the aforementioned example, experi-
mentally manipulate the distribution mode, we cannot know whether we have identified
the effect of the newspaper format or simply that this effect is masking the effect of other
factors, such as the newspapers age or use of entertainment news. After all, newspapers
with a traditional paper format were probably founded a long time ago, and online
newspapersat least in the Norwegian contextmight be more oriented toward entertain-
ment news than printed newspapers. Thus, the effect of the distribution mode may mask
that people trust a source because of its legacy and content, rather than that the format as
such has an effect on trust. To isolate the effect of distribution mode, we need to account
for other factors that might mask the actual effects. One approach is to technically include
these other factors but simply hold them constant at one value (e.g., include only old
newspapers with no entertainment news). This we could do with a conventional survey
experiment. Then we would know the effect of the distribution mode, but only for one
particular case. However, conjoint designs add the possibility of identifying the effect of
the distribution mode more generally (i.e., averaged over all possible combinations of
related factors).
Second, respondentsjudgment of credibility, like most other judgments and deci-
sions, are, conceptually, multidimensional. To evaluate a newspapers credibility, respon-
dents would ideally need information about other relevant factors as well. Even if we
isolate the average effect of the distribution mode on credibility, we cannot know how
important this factor is compared to other relevant factors, such as the newspapers amount
of entertainment news, party affiliation, and ethical violations. Whatever distribution mode
effect we find may seem substantive in isolation but in reality may be potentially insig-
nificant compared to other factors. Conjoint designs solve this issue by enabling the
researcher to identify the effect of the distribution mode and many other factors at the
same time. Thus, we can assess the effect of one factor and compare this effect to the
effects of various other factors.
Empirical Examples of How Conjoint Experiments Can Be Applied in
Political Communication Research
We have explained the added benefits of conjoint designs in general. In this section, we
present two very different applications of conjoint analysis that are specific for political
communication research. We do not go into detailed analysis of the results here but use
these examples to walk the reader through methodological choices that need to be made
and how the results are analyzed. The code for replicating these exemplary analyses can be
retrieved from the online supplemental material.
The first study is an example of the traditional conjoint design and illustrates how the
technique detailed by Hainmueller and colleagues (2014) can be beneficial for studying
political communication phenomena. The second study illustrates how the logic of
BEYOND THE LIMITS OF SURVEY EXPERIMENTS 3
conjoint design can be innovated, extended to, and tailored for studying phenomena
specific to political communication.
Data
We collected the data for both examples through the Norwegian Citizen Panel (NCP), a
probability-based online survey panel in Norway. We fielded the experiments in the eight
(March 6 to April 21, 2017) and ninth (May 11 to June 6, 2017) waves of the NCP. The
respondents were invited through a postal recruitment of 25,000 Norwegian individuals,
randomly sampled from Norways National Registryan official list of all residents of
Norway (for details about response rates or other methodological matters, see Skjervheim
and Høgestøl [2017a,2017b]).
Example 1: The Traditional Choice-Based Conjoint Experiment
The first example illustrates the traditional choice-based conjoint design (Hainmueller
et al., 2014). In these designs, respondents face a choice between two profiles. These
profiles list a range of attributes in a table where the particular levels for each attribute in
each profile are randomly assigned. In this case, the table with the two profiles contains
information about two news publications (see Figure 1 for a screenshot of the design), and
the choice task is to choose which news source is the most trustworthy.
Choosing the Attributes. When designing conjoint experiments, one must choose which,
and how many, attributes to include in the experiment. Bansak and colleagues (2017) test
how far researchers can push these limits in terms of the number of attributes included in
such profiles and show that treatment effects are robust to a large number of tasks and
attributes.
In the present design, we include eight different theoretically relevant attributes that
we assume affect peoples trust in a news source. The full list of attributes and attribute
levels are shown on the Y-axis in Figure 2. This design enables an analysis of the effects of
a publications distribution mode (Kiousis, 2001) and reveals possible masking effects, and
compares the distribution mode effects to the effects of other relevant attributes such as the
amount of entertainment news (Ladd, 2012), and the age of the publication.
The design is a 3 × 2 × 3 × 3 × 3 × 3 × 3 × 10 factorial design, equaling more than
29,000 possible combinations. This means that only a fraction of the possible profile
combinations is ever observed. However, as Hainmueller and colleagues (2014) show, we
do not need to observe all possible combinations to identify the average marginal treat-
ment effects of each component. These effects are identifiable under a set of assumptions
that is likely to hold in a typical conjoint experiment: (a) that the respondent would make
the same choice if presented with exactly the same profiles again, (b) that the ordering of
profiles within a choice task does not affect the response, and (c) that the randomization of
each attribute is either conditionally or completely independent of the other attributes (see
Hainmueller et al., 2014, pp.89,13,16).
The inclusion of several attributes can result in absurd or impossible combinations
(e.g., a 20-year-old medical doctor with 30 years of work experience) (Hainmueller et al.,
2014,p.9). We may choose to keep these highly unlikely combinations, remove and
replace them, or strive for a design that does not produce them. In the present design,
we chose the latter option. If specific combinations are removed, certain measures must be
taken in the analysis (see Hainmueller et al., 2014, p.20).
4 Erik Knudsen and Mikael Poul Johannesson
Choosing the Procedure. In order to achieve the required statistical power, researchers
should aim for a large number of observations. When designing conjoint experiments,
researchers must typically choose whether to field a study with a large sample (e.g.,
representative survey) and few choice tasks or a study with a small sample (e.g., lab
experiment) and many choice tasks. Bansak and colleagues (2018) test how many choice
tasks respondents can rate in a row before survey satisficing degrades response quality and
show that treatment effects are robust to a large number of tasks in a row. Thus, the choice
of procedure is often guided by costin larger time-sharing surveys (e.g., TESS) with
many respondents, it might be cheaper to run one choice task with many respondents and
Figure 1. Experimental design. This figure illustrates the experimental design for the Trust Study
conjoint experiment.
BEYOND THE LIMITS OF SURVEY EXPERIMENTS 5
vice versa in surveys where sample size is more expensive than survey space (e.g.,
Amazon Mechanical Turk).
As we fielded this experiment in a large time-sharing survey, the respondents eval-
uated one comparison between a pair of news publications, as shown in Figure 1. The
study puts a random sample of 1955 participants in the NCP in the position of news
consumers. We ask them to choose between two hypothetical online news publications.
We show respondents a screen with profiles of the two news publications (see Figure 1)
with the following introduction: We are interested in examining what makes people trust
different sources of news. Below, we have created two hypothetical news sources. Please
read the descriptions of both sources carefully and answer the question below.We then
instruct the respondents to indicate which of these two do you think would be the most
reliable source to report the news in a fully accurate and fair manner?
Analyzing the Data. Analyzing a typical conjoint design is straightforward. Following
Hainmueller and colleagues (2014), we wish to estimate the average marginal component
effects (AMCEs): the marginal effect of one attribute averaged over the joint distribution
of the other attributes. For example, the AMCE of readership (few versus many readers)
represents the average effect of readership on the probability that the news publication will
Figure 2. Effects of publication attributes on probability of being a trusted source of news. The dots
represent the point estimates of the effects (AMCEs) of different source attributes on the trust. The
bars show 95% percent confidence intervals.
6 Erik Knudsen and Mikael Poul Johannesson
be chosen as reliablethat is, the average of the effect of readership across all possible
combinations of the remaining attributes, weighted by the probability of getting each
combination (and in this case, all combinations are equally likely). Each attribute level is
compared to a different attribute level within the same attribute. The researcher just
chooses a reference category.
The AMCEs can conveniently be estimated without bias with a linear regression
model (under assumptions a, b, and c) where we include an observation for every
individual profile and regress the dependent variable (i.e., selected a profile or not) on
all levels of each attribute (except the reference level for each attribute) (Hainmueller et al.,
2014, pp.1415). To get unbiased estimates of the variance, because respondents are given
two profiles in each task, and often perform several choice tasks, the standard errors need
to be corrected for with within-respondent clustering (e.g., using clusterin Stata; see
Hainmueller et al., 2014, pp.1617). Available statistical software libraries in R (e.g., the
cjoint package by Strezhnev, Berwick, Hainmueller, Hopkins, and Yamamoto, 2017)or
Stata, for instance, makes estimating and plotting the AMCEs straightforward as displayed
in Figure 2.InFigure 2, dots indicate point estimates, bars illustrate 95% confidence
intervals, and dots without bars are reference categories. Here we see that an offline and
online newspaperis more trustworthy than an online newspaper,thus demonstrating
that the distribution mode effect is substantive in isolation and not masking the effects of
age and entertainment news. We can also compare this effect to the effects of the other
attributes and observe that the effect of online newspapersis statistically indistinguish-
able from the effect of primarily focusing on entertainment news.
Although previous studies isolated the effects of factors such as use of advertising and
comment fields on peoples trust evaluations, this example illustrates that conjoint experi-
ments can provide insights into the relative effects of such factors and reveal the expla-
natory strength of different hypotheses identified by previous research. This means that we
can study whether one or more factors are more important than others and overcome issues
of masking effects.
Example 2: The Automated Sentence Generator in a Choice-Based Design
The second example uses a conjoint experimental design to study selective exposure:
citizens preferring to encounter information that is consistent, rather than at odds, with
their existing political attitudes (Knobloch-Westerwick, Mothes, & Polavin, 2017;
Mummolo, 2016).
In experimental approaches to selective exposure, respondents are often required to
make a choice and select one or more news stories over others. However, selective
exposure in the real world involves multidimensional choices with many factors, such as
the partisan reputation of the source (source cues; Mummolo, 2016), the pro or con
message frame of an issue (message cues; Knobloch-Westerwick et al., 2017), the valence
of the headline (negativity bias; Knobloch-Westerwick et al., 2017), and the political actors
(e.g., a political candidate) mentioned in the headline (party cues; Iyengar, Hahn,
Krosnick, & Walker, 2008). Although researchers have used the traditional choice based
conjoint experiment to examine selective exposure (Mummolo, 2016), researchers have
yet to study all of these cues simultaneously. In order to account for all these factors
simultaneously, we introduce a new conjoint experiment template that is tailored for
political communication research.
BEYOND THE LIMITS OF SURVEY EXPERIMENTS 7
Research Design. In contrast to the profile tables shown in the first example, the stimuli in
this example are presented as a list of headlines. Instead of randomly assigning attributes
for profiles in a table, we randomly assign attributes for profiles in a headline. Using the
logic of conjoint design, one can randomly vary a variety of information in a headline and
subsequently analyze the relative importance of each component.
We use this approach to produce a script that constructs 756 headlines that vary on four
attributes. Each headline has a partisan actor that signals a preference about a topic and is
mentioned with a neutral, negative, or positive valence. We randomize the party of the actor
(nine parties), the message topic (seven topics), the message direction (two preferences), and the
valence of the mentioned actor (three valence categories). The seven message topics each have
two unique recipesfor where and how the remaining information is imputed (the attributes and
attribute levels, except valence, are shown in the Y-axis in Figure 4). For instance, one of the two
recipes for headlines about privatization of public services looks like this:
[party]-politician receives [valence] for a new proposal: want to [preference]
the Norwegian Railroad Service
The following is an example of the headlines:
Labor Party politician receives criticism for a new proposal: want to privatize
the Norwegian Railroad Service
Procedure. We asked 2,071 respondents in the NCP to closely read a selection of four
randomly generated news headlines and decide which two headlines they would most
likely choose to spend their time on, as displayed in Figure 4.
The headlines were introduced with the following vignette: We wish to study peoples
news habits. Below you will find some hypothetical headlines, which we have constructed,
similar to those you may find in Norwegian online newspapers. Please read all of the headlines
carefully and imagine that the headlines are real,We followed this with, You would perhaps
not read any of these articles on a normal day, but lets say that you had to read two of these
articles. Which articles would you prefer to spend your time on?
Analysis. The data include 8,284 observations of selection decisions. Because we force
respondents to make a choice, we have information about which attributes respondents
selected and which they did not. As with the first example, the analysis of the headline
selections is straightforward. Given the assumptions mentioned earlier, we can estimate the
average marginal treatment effect of the components in the headlines.
In the analysis of these headline selections, we focus on two so-called cues that can
guide peoples headline selection: message cues (i.e., peoples preferences for political
messages in line with their attitudes) and party cues (i.e., peoples preference for news
stories that feature a party or candidate they prefer).
Figure 3a displays the AMCEs of all the headline attributes for all respondents on the
probability of selecting a headline. However, we learn little about selective exposure from
these results without matching these attributes with the respondentsattitudes and political
preferences. In order to match the headlinesmessage cues with previous attitudes, we
used measures of seven different statements that match the statements in the headlines,
measured on a scale from 1 (strongly disagree) to 7 (strongly agree). For instance: How
much do you agree or disagree with the following statements?:Commercial private
schools should be allowed.These statements were then coded as attitude consistentand
8 Erik Knudsen and Mikael Poul Johannesson
attitude inconsistent.The party cues were matched with respondentsevaluations of
each party, measured by asking respondents, We would like to ask you to consider how
much you like or dislike the various political parties in Norwayon a scale from 1
(intensely dislike) to 7 (intensely like). These measures were then matched with the
attribute values in the headlines and coded as likes partyor dislikes party.
Figure 3b shows the conditional AMCEs when the attributes of the headlines are matched
with the attitudes of the respondents. We observe that party cues yield a clear effect, while the
effects of message cues do not yield a statistically significant effect, suggesting that the effects
of party cues are stronger than message cues. The fact that the effects of the matched AMCEs
(Figure 3a) are smaller than the AMCEs for message topics (Figure 3b)supportsMummolos
(2016) argument about the importance of topic relevance.
This example highlights the need and opportunity for modifications of conjoint
designs to study issues that are specific to political communication research. Previous
studies in social science optimized the conjoint technique to study peoples political
preferences (Hainmueller et al., 2014); however, this automated sentence generator dis-
plays headlines that are closer to what readers actually meet in their day-to-day media
exposure and is easier to comprehend than the traditional choice-based design, when the
objective is to compare peoples selection of news headlines. Although conjoint experi-
ments are often limited to a choice between two profiles, this approach also enables a
design that more easily can include three or more profiles (i.e., headlines) in a choice task.
This selective exposure design illustrates how political communication requires dif-
ferent considerations than other fields, and that such considerations should be addressed
when applying the method. For instance, Knobloch-Westerwick and colleagues (2017)
show through a lab experiment why we should study the effects of different subtypes (i.e.,
confirmation bias, in-group bias, and negativity bias) of selective exposure simultaneously.
They cannot separate the effects of each subtype because they do not use a conjoint
experiment. Their argument would be strengthened by a tailored conjoint design that could
enable a comparison of the effect of each cue. Crucially, our headline template demon-
strates that party cues have a larger effect than message cues on peoples propensity to
engage in selective exposure.
Figure 3. Experimental Design. This figure illustrates the experimental design for the Selective
Exposure Study conjoint experiment.
BEYOND THE LIMITS OF SURVEY EXPERIMENTS 9
We also argue that this example illustrates that political communication research is an
ideal field for further innovating applications of the method. In the following section, we
suggest a research agenda for how conjoint analysis can improve political communication
research and how political communication can improve the method.
How Can Conjoint Analyses Improve Causal Inference in Political
Communication Research?
There are several research areas where conjoint experiments can further our understanding
of multidimensional political communication effects. As we illustrated with two empirical
Figure 4. Effect on Probability of Selecting a News Headline by (a) Randomized Headline
Attributes and (b) Headline Attributes Matched with Respondent Preferences. The dots represent
the point estimates of the effects (AMCEs) of difference source attributes on the trust. The bars show
95 percent confidence intervals. Note that the figure displays six, not seven, topics, because Reduce
taxesis two topics collapsed as one.
10 Erik Knudsen and Mikael Poul Johannesson
examples, this method can be used to study whether one attribute is noticeably stronger
than another and to solve issues of possible masking effects in causal inference. For that
reason, conjoint experiments can help clarify ongoing debates in the political communica-
tion literature. In addition, we demonstrated that conjoint designs can be tailored and
innovated to address issues that are specific to political communication, such as selective
exposure. We suggest three possible future applications of the method.
First, as illustrated with the first example, traditional conjoint designs can improve
causal inference in research where one is interested in how a range of different character-
istics of a phenomenon affects peoples probability of trusting, selecting, or using another
phenomenon (for instance, how politicianscharacteristics [such as the way they commu-
nicate] shape peoples trust in politicians) in a study that randomly varies certain commu-
nication styles or rhetorical techniques between two hypothetical politicians and asks
respondents to compare and contrast them in terms of who they trust. Future research
should seek to use conjoint experiments in such instances.
Second, as illustrated by the second example, researchers have the opportunity to
innovate conjoint designs tailored to political communication research. For instance,
scholars interested in the effects of different attributes of sentences or headlines can use
the logic of conjoint experiments to gain knowledge about how different parts of a
sentence affect peoples choices or attitudes. For instance, one can employ a similar
approach to study effects of a range of variations of message framing, such as decompos-
ing possible multidimensional relationships of framing being an information effect rather
than an emphasis effect (Leeper & Slothuus, 2017).
Third, as we have not illustrated or detailed here, conjoint designs are well-suited to
study mediation effects and investigate whether the effects of the attributes in a conjoint
design are conditional on specific attributes and whether the result is conditional on what
attributes are included in the conjoint (e.g., Dafoe, Zhang, & Caughey, in press). Acharya,
Blackwell, and Sen (2016) demonstrate and detail an approach for dealing with mediation
in conjoint experiments by testing the effects of randomly including or excluding some
specific attributes on the effects of the other attributes. For instance, we can test whether
selective exposure effects in social media environments are contingent on the attributes of
the person who shares a story with you.
Political communication scholars also have the opportunity to engage in methodolo-
gical discussions and extend our knowledge of the limitations and external validity of the
method. Hainmueller and colleagues (2015) validated conjoint designs in one particular
case but we have yet to learn what the results from conjoint designs on political commu-
nication truly teach us about phenomenon in the real world (e.g., Barabas & Jerit, 2010).
Concluding Remarks
This article has highlighted how conjoint experiments can be used as a fruitful addition to
political communication scholarsarsenal of research approaches. The traditional survey
experiment has well-known restrictions regarding the number of factors we can study at
any one time. Conjoint analysis addresses these issues by separately identifying several
component-specific causal effects.
We believe that conjoint experiments can be employed considerably more than thus
far in political communication research. In studies where researchers aim to study multi-
dimensional causal relations and pit two or more hypotheses against one another, or where
answers to scholarly debates hinge on the opportunity to overcome the survey experi-
ments constraints in number of experimental conditions, the conjoint experiment is a
BEYOND THE LIMITS OF SURVEY EXPERIMENTS 11
superior choice. Political communication scholars also have an opportunity to continue to
innovate, enhance, and tune the conjoint design to better understand how political com-
munication shapes modern political reality.
Acknowledgments
We would like to thank Elisabeth Ivarsflaten and Stefan Dahlberg for their guidance on this
project. We would also like to thank the editors, Yanna Krupnikov, Kathleen Searles, and
Claes de Vreese, as wellas the anonymous reviewers for a helpful and efficient review process.
Supplemental Material
Supplemental data for this article can be accessed on the publishers website at
Disclosure statement
No potential conflict of interest was reported by the authors.
Funding
This work was supported by the Bergens Forskningsstiftelse [BFS2015TMT01]; Rådet for
Anvendt Medieforskning [15/370-2/JEA]
References
Acharya, A., Blackwell, M., & Sen, M. (2016). Analyzing causal mechanisms in survey experiments.
Working Paper. Retrieved from http://scholar.harvard.edu/files/msen/files/directeffects-experi-
ments.pdf
Arceneaux, K. (2010). The benefits of experimental methods for the study of campaign effects.
Political Communication,27(2), 199215. doi:10.1080/10584601003709407
Bansak, K., Hainmueller, J., Hopkins, D. J., & Yamamoto, T. (2017). Beyond the breaking point?
survey satisficing in conjoint experiments (Res. Pap. 17-33). Stanford, CA: Grad. Sch. Bus.,
Stanford Univ. Retrieved from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2959146
Bansak, K., Hainmueller, J., Hopkins, D. J., & Yamamoto, T. (2018). The number of choice tasks
and survey satisficing in conjoint experiments. Political Analysis,26,112119. doi:10.1017/
pan.2017.40
Barabas, J., & Jerit, J. (2010). Are survey experiments externally valid? American Political Science
Review,104(2), 226242. doi:10.1017/S0003055410000092
Bartels, L. M. (1993). Messages received: The political impact of media exposure. American
Political Science Review,87(2), 267285. doi:10.2307/2939040
Dafoe, A., Zhang, B., & Caughey, D. (in press). Information equivalence in survey experiments.
Political Analysis.
Druckman, J. N., & Leeper, T. J. (2012). Learning more from political communication experiments:
Pretreatment and its effects. American Journal of Political Science,56(4), 875896.
doi:10.1111/ajps.2012.56.issue-4
Green, P. E., & Rao, V. R. (1971). Conjoint measurement for quantifying judgmental data. Journal of
Marketing Research,8, 355363. doi:10.2307/3149575
Hainmueller, J., Hangartner, D., & Yamamoto, T. (2015). Validating vignette and conjoint survey
experiments against real-world behavior. Proceedings of the National Academy of Sciences,112
(8), 23952400. doi:10.1073/pnas.1416587112
12 Erik Knudsen and Mikael Poul Johannesson
Hainmueller, J., Hopkins, D. J., & Yamamoto, T. (2014). Causal inference in conjoint analysis:
Understanding multidimensional choices via stated preference experiments. Political Analysis,
22(1), 130. doi:10.1093/pan/mpt024
Helfer, L. (2016). Media effects on politicians: An individual-level political agenda-setting experiment.
The International Journal of Press/Politics,21(2), 233252. doi:10.1177/1940161215627461
Iyengar, S., Hahn, K. S., Krosnick, J. A., & Walker, J. (2008). Selective exposure to campaign
communication: The role of anticipated agreement and issue public membership. The Journal of
Politics,70(1), 186200. doi:10.1017/S0022381607080139
Jasso, G., & Rossi, P. H. (1977). Distributive justice and earned income. American Sociological
Review,42, 639651. doi:10.2307/2094561
Kiousis, S. (2001). Public trust or mistrust? perceptions of media credibility in the information age.
Mass Communication & Society,4(4), 381403. doi:10.1207/S15327825MCS0404_4
Knobloch-Westerwick, S., Mothes, C., & Polavin, N. (2017). Confirmation bias, ingroup bias, and
negativity bias in selective exposure to political information. Communication Research.
doi:10.1177/0093650217719596
Ladd, J. M. (2012). Why americans distrust the news media and how it matters. Princeton, NJ:
Princeton University Press.
Leeper, T. J., & Slothuus, R. (2017). Can citizens be framed? how information more than emphasis
changes opinions. Working Paper. Retrieved from https://s3.us-east-2.amazonaws.com/tjl-shar-
ing/assets/CanCitizensBeFramed.pdf
Mummolo, J. (2016). News from the other side: How topic relevance limits the prevalence of
partisan selective exposure. The Journal of Politics,78(3), 763773. doi:10.1086/685584
Neyman, J. (1923). On the application of probability theory to agricultural experiments. essay on
principles. section 9. Statistical Science, 5, 465472. Translated in 1990, with discussion.
Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized
studies. Journal of Educational Psychology,66(5), 688. doi:10.1037/h0037350
Skjervheim, Ø., & Høgestøl, A. (2017a). Norwegian citizen panel methodology report wave 8 (Tech.
Rep.). Bergen, Norway: Ideas 2 Evidence
Skjervheim, Ø., & Høgestøl, A. (2017b). Norwegian citizen panel methodology report wave 9 (Tech.
Rep.). Bergen, Norway: Ideas 2 Evidence.
Sniderman, P. M. (2011). The logic and design of the survey experiment. In J. N.Druckman, D. P.
Green, J. H.Kuklinski, & A.Lupia (Eds.), Cambridge handbook of experimental political
science (pp. 102114). New York, NY: Cambridge University Press.
Strezhnev, A., Berwick, E., Hainmueller, J., Hopkins, D., & Yamamoto, T. (2017). cjoint: Amce
estimator for conjoint experiments. Retrieved from https://cran.r-project.org/web/packages/
cjoint/
BEYOND THE LIMITS OF SURVEY EXPERIMENTS 13
... To compare the relative importance of the five evaluation criteria for voter evaluations of incumbent performance, a discrete choice experiment was designed-an increasingly used method in political science, which has proven to be useful for studying multi-dimensional political preferences (e.g., Hansen et al. 2015;Knudsen and Johannesson 2019). 4 The experiment was conducted in Sweden in June 2019, in wave 34 of the Laboratory of Opinion Research (LORE). ...
... The use of a discrete choice experiment offered two tangible advantages to other methodologies for answering the questions posed in this study (see Hainmueller et al. 2014;Knudsen and Johannesson 2019). First, the chosen design allowed for simultaneous, isolated estimation of the influence of a larger number of evaluation criteria on voters' evaluations of government performance than a typical survey experiment would have. ...
... With regard to other common concerns in the design of discrete choice experiments, attribute order was randomized per respondent to rule out the influence of ordering and primacy effects; and attribute orthogonality was achieved by design as no possible combinations of attribute values were excluded (see Hainmueller et al. 2014;Franchino and Zucchini 2015;Knudsen and Johannesson 2019). In order to address the recent criticism of using Average Marginal Component Effects (AMCEs) for studying voter preferences (see Abramson et al. 2019), several robustness checks were conducted to further validate the direction of the observed effects, and in particular the relative importance of the competing attributes for respondents' evaluations of government performance. ...
Article
Full-text available
Voters hold governments to account through elections, but which criteria are most important to voter evaluations of incumbent performance? While (economic) outcomes have long been central to studies of retrospective voting, recent studies have considered the influence of policy output—the policies implemented by incumbents to achieve their goals. Building on this promising development, this study identifies three ways in which policy output is expected to affect voter evaluations of incumbent performance—the congruence between implemented policy and (1) individual preferences; (2) public opinion; and (3) election pledges. A discrete choice experiment was designed to assess the relative importance of these three aspects of policy output in comparison to each other; as well as to two important economic indicators. Overall, the findings support the notion that policy output matters to voters even beyond outcomes. The findings also show that voters value congruence between policy and their personal preferences considerably more than policy congruence with public opinion; and election pledge fulfillment. This indicates that voters are egotropic in their evaluation of implemented policy, and more policy-seeking than accounted for in much of the empirical retrospective voting literature. These results inform our understanding of how policy output matters to voters, as well as of how voters hold governments accountable for their performance.
... We rely on a conjoint survey experiment (n = 830) designed to examine how much the temporal aspect affects the policy choices of ordinary citizens, compared to other relevant policy aspects, such as policy area, expected costs and level of decisionmaking. Conjoint analysis makes it possible to disentangle the causal effects of multiple factors on the choices of respondents (Hainmueller et al. 2014;Knudsen and Johannesson 2018). This technique therefore makes it possible to determine how the time horizon affects policy choices and to compare the degree of effect with other relevant aspects of these policies. ...
... Conjoint analysis provides a way to test multidimensional causal relations on a representative sample (Hainmueller et al. 2014(Hainmueller et al. , 2015Knudsen and Johannesson 2018). While some question the approach and in particular the conclusions to be drawn based on the results (Abramson et al. 2019), this approach offers important advantages compared to more traditional analyses of surveys. ...
Article
Full-text available
Previous scholarship has focused primarily on how citizens' form policy preferences and how those preferences are taken into account in democratic decision-making. However, the temporal aspect of policy preferences has received little attention, although many significant societal problems have consequences that extend far into the future. To fill the gap, we examine to what extent citizens are willing to support policies, when rewards can only be expected after several electoral cycles. Using a conjoint survey experiment, we demonstrate that while a slight tendency towards more immediate policy rewards is discernible, citizens are not as impatient as has been widely assumed. In contrast with previous research, political trust does not affect the impact of the time horizon of policy choice. Instead, we find that people with higher education are more likely to choose policies the benefits of which materialize in the distant future. These findings add to the growing evidence which suggests that citizens' short-sightedness is not a very strong driver of democratic myopia.
... We explore the interaction between individual and organizational characteristics using a conjoint experiment conducted with a nationally representative sample in the US. 5 Conjoint analysis allows us to elicit donor preferences without respondents explicitly stating their preferences. This kind of experiment is commonly used in marketing to reveal consumer preferences for products and has become increasingly popular in political science and public policy (Knudsen and Johannesson, 2019). Because we are interested in exploring the complexity behind the decision to donate, we apply conjoint methods to the study of nonprofits and philanthropy, where we can study the simultaneous effects of organizational characteristics and individual attributes on individuals' preferences to donate to an INGO. ...
Article
The phenomenon of closing civic space has adversely impacted international non‐governmental organization (INGO) funding. We argue that individual private donors can be important in sustaining the operations of INGOs working in repressive contexts. Individual donors do not use the same performance‐based metrics as official aid donors. Rather, trust can be an important component of individual donor support for nonprofits working towards difficult goals. How does trust in charitable organizations influence individuals' preferences to donate, especially when these groups face crackdown? Using a simulated market for philanthropic donations based on data from a nationally representative sample of individuals in the United States who regularly donate to charity, we find that trust in INGOs matters substantially in shaping donor preferences. Donor profiles with high levels of social trust are likely to donate to INGOs with friendly relationships with host governments. This support holds steady if INGOs face criticism or crackdown. In contrast, donor profiles with lower levels of social trust prefer to donate to organizations that do not face criticism or crackdown abroad. The global crackdown on NGOs may thus possibly sour NGOs' least trusting individual donors. Our findings have practical implications for INGOs raising funds from individuals amid closing civic space. Research on international giving by individuals, especially in the era of closing civic space, is not meant to find answers that can act as substitutes for strategic policy responses, especially by official aid donors and foundations. However, many INGOs are under immediate threat, and individual‐level philanthropy can help support these organizations.
... On the contrary, ties in co-trust networks may not show up in co-exposure networks. News consumers may resort to various media attributes, such as readership size, journalistic ethics, and content focus, to evaluate the credibility of news sources (Knudsen and Johannesson, 2019). Research also shows that people across the political spectrum rated mainstream outlets as more trustworthy than hyper-partisan ones (Pennycook and Rand, 2019). ...
Article
Full-text available
While partisan selective exposure could drive audience fragmentation, other individual factors might also differentiate news diets. This study applies a method that disentangles the differential contributions of the individual characteristics to audience duplication networks. By analyzing a nationally representative survey about US adults' media use in 2019 (N = 12,043), we demonstrate that news fragmentation is driven by a myriad of individual factors, such as gender, race, and religiosity. Partisanship is still an important driver. We also distinguish between media exposure and media trust, showing that many cross-cutting ties in co-exposure networks disappear when media trust is considered. We conclude that audience fragmentation research should extend beyond ideological selectivity and additionally investigate how and why other individual-level preferences differentially contribute to fragmentation both in news exposure and in news trust.
... I used a conjoint survey experiment to test how signals of cohesion affect attraction to partisan advocacy organizations. This method lets me nonparametrically estimate the effects of multiple organizational attributes at the same time, while controlling for unobserved determinants of attraction to organizations (Hainmueller et al. 2014;Knudsen and Johannesson 2019). 4 Randomizing values of the attributes ensures that different profiles of hypothetical organizations will have the same distribution of observed and unobserved attribute values, on average, besides the value of interest. ...
Article
Full-text available
Advocacy organizations—such as political parties, political action committees, and interest groups—have the potential to shape public policy and opinion. But to realize that potential, they must mobilize participants, sympathy, and funding. This article explores whether advocacy organizations gain more support when they signal greater cohesion. A conjoint experiment tests the effects of member cohesion, along with other attributes including group size and use of violent tactics, on people’s attraction to hypothetical organizations that were founded to elect candidates from their preferred political party (in this study, the Democratic Party). Results show that Americans who identify as Democrats are more likely to join, donate to, and generally feel positive toward cohesive organizations, but will embrace those with heterogeneous goals so long as members do not actively disagree with one another. These findings can inform the strategies of “intersectional” advocacy organizations that assemble people of diverse and sometimes conflicting interests.
Article
Research on misinformation and misperceptions often investigates claims that have already reached a critical mass, resulting in little understanding of why certain claims gain widespread belief while others fall into obscurity. Here we consider how various message features factor into why certain claims are more likely to be believed, perceived as credible, and shared with others. Using a conjoint experiment, we randomly assigned participants (N = 1,489) to receive an experimentally manipulated message describing an allegation of political misconduct. Results suggest that partisan cues play a significant role in influencing both belief and perceived credibility. Furthermore, message specificity, language intensity, and whether other users’ comments on the post refute or endorse the post also influenced belief judgment and credibility assessments. We conclude with a discussion of the theoretical and practical importance of these findings for understanding and combating the threat of misinformation.
Article
Conjoint experiments are a standard tool to measure (stated) preferences in the context of multidimensional choices on issues ranging from immigration to democratic innovations. This case study demonstrates how to design and effectively deploy a (survey embedded) conjoint experiment. Practical recommendations are offered along with the conjoint experiment checklist of steps to accomplish before executing a conjoint survey experiment, as well as steps to follow during the execution of the project itself.
Article
Similar to many developing countries, the government of China has publicly committed itself to the rule of law. As part of the campaign to strengthen the rule of law, the government, media and academics have employed multiple justifications for the value of rule of law, including instrumental justifications such as economic growth and corruption reduction, and intrinsic or normative justifications like increased fairness and neutrality of judicial decisions. While various justifications for the rule of law abound, we know exceedingly little about why the public might value the rule of law. To understand mass attitudes toward the rule of law, we conducted a conjoint survey experiment using a national urban sample of 2,100 Chinese residents. We find instrumental justifications, especially corruption reduction, to have the largest effects, though several intrinsic justifications are also significant. Additionally, intergroup analysis shows that income and experience with the courts condition the type of justification that is most likely to be employed.
Article
Full-text available
While news outlets still play an important role as a source of news, people increasingly receive their political information and news from social networking sites (SNSs). This study extends the literature on exposure and sharing decisions on SNSs by exploring how different attributes shape such decisions, how the two decision types differ, and by disentangling the role played by personal news recommendations and shared news stories on SNSs. As SNSs add a social dimension, exposure and sharing decisions are contingent not only upon the news story itself (news-story attributes) but also upon characteristics of the person who shares the story (recommender attributes). We designed a conjoint experiment to disentangle the effects of recommender and news-story attributes on the decision to recommend and read news and fielded it in a probability-based Norwegian online survey. The results suggest that committing to reading and sharing information are two similar yet distinct phenomena and that selective sharing is a stronger commitment than selective exposure. We also found evidence to suggest that selective sharing of news featuring a favored political party was contingent upon whether one also received information about the recommender attributes.
Article
Survey experiments often manipulate the description of attributes in a hypothetical scenario, with the goal of learning about those attributes’ real-world effects. Such inferences rely on an underappreciated assumption: experimental conditions must be information equivalent (IE) with respect to background features of the scenario. IE is often violated because subjects, when presented with information about one attribute, update their beliefs about others too. Labeling a country “a democracy,” for example, affects subjects’ beliefs about the country’s geographic location. When IE is violated, the effect of the manipulation need not correspond to the quantity of interest (the effect of beliefs about the focal attribute). We formally define the IE assumption, relating it to the exclusion restriction in instrumental-variable analysis. We show how to predict IE violations ex ante and diagnose them ex post with placebo tests. We evaluate three strategies for achieving IE. Abstract encouragement is ineffective. Specifying background details reduces imbalance on the specified details and highly correlated details, but not others. Embedding a natural experiment in the scenario can reduce imbalance on all background beliefs, but raises other issues. We illustrate with four survey experiments, focusing on an extension of a prominent study of the democratic peace.
Article
Researchers investigating causal mechanisms in survey experiments often rely on nonrandomized quantities to isolate the indirect effect of treatment through these variables. Such an approach, however, requires a “selection-on-observables” assumption, which undermines the advantages of a randomized experiment. In this paper, we show what can be learned about casual mechanisms through experimental design alone. We propose a factorial design that provides or withholds information on mediating variables and allows for the identification of the overall average treatment effect and the controlled direct effect of treatment fixing a potential mediator. While this design cannot identify indirect effects on its own, it avoids making the selection-on-observable assumption of the standard mediation approach while providing evidence for a broader understanding of causal mechanisms that encompasses both indirect effects and interactions. We illustrate these approaches via two examples: one on evaluations of US Supreme Court nominees and the other on perceptions of the democratic peace.
Article
In recent years, political and social scientists have made increasing use of conjoint survey designs to study decision-making. Here, we study a consequential question which researchers confront when implementing conjoint designs: How many choice tasks can respondents perform before survey satisficing degrades response quality? To answer the question, we run a set of experiments where respondents are asked to complete as many as 30 conjoint tasks. Experiments conducted through Amazon’s Mechanical Turk and Survey Sampling International demonstrate the surprising robustness of conjoint designs, as there are detectable but quite limited increases in survey satisficing as the number of tasks increases. Our evidence suggests that in similar study contexts researchers can assign dozens of tasks without substantial declines in response quality.
Article
Selective reading of political online information was examined based on cognitive dissonance, social identity, and news values frameworks. Online reports were displayed to 156 Americans while selective exposure was tracked. The news articles that participants chose from were either conservative or liberal and also either positive or negative regarding American political policies. In addition, information processing styles (cognitive reflection and need-for-cognition) were measured. Results revealed confirmation and negativity biases, per cognitive dissonance and news values, but did not corroborate the hypothesis derived from social identity theory. Greater cognitive reflection, greater need-for-cognition, and worse affective state fostered the confirmation bias; stronger social comparison tendency reduced the negativity bias.
Article
Conjoint measurement is a new development in mathematical psychology that can be used to measure the joint effects of a set of independent variables on the ordering of a dependent variable. In this (primarily expository) article, the techniques are applied to illustrative problems in marketing. In addition, a number of possible areas of application to marketing research are discussed, as well as some of the methodology's limitations.
Article
Prior research has demonstrated a preference among partisans for like-minded news outlets, a key mechanism through which the media may be polarizing Americans. But in order for source reputations to cause widespread selective exposure, individuals must prioritize them above other competing attributes of news content. Evaluating the relative in-fluence of various contributors to media choice is therefore critical. This study pits two such factors, source reputation and topic relevance, against one another in conjoint survey experiments offering randomly paired news items to partisans. Making a news source's reputation politically unfriendly lowers the probability that an individual chooses an item, but this negative effect is often eclipsed by the positive effect of making a news topic relevant to the individual. In many popular modern news consumption environments, where consumers encounter a diverse mixture of sources and topics, the ability of source reputations to contribute to polarization via partisan selective exposure is limited. © 2016 by the Southern Political Science Association. All rights reserved.