Efficacy Methods to Evaluate Health Communication and Marketing Campaigns
W. Douglas Evans1
1 The George Washington University
2 RTI International
Running Head: Efficacy Methods to Evaluate Health Communication and
Send all correspondence to:
W. Douglas Evans, Ph.D.
The George Washington University
2175 K Street, NW, Suite 700
Washington, DC 20037
Voice: (202) 416-0496
Fax: (202) 416-0433
Communication and marketing are growing areas of health research, but
relatively few rigorous efficacy studies have been conducted in these fields. In
this paper, we review recent health communication and marketing efficacy
research, present two case studies that illustrate some of the considerations in
making efficacy design choices, and advocate for greater emphasis on rigorous
health communication and marketing efficacy research and the development of a
Much of the outcomes research in health communication and marketing,
especially mass media, utilizes effectiveness designs conducted in real-time, in
the media markets or communities in which messages are delivered. However,
such evaluations may be impractical or impossible, limiting opportunities to
advance the state of health communication and marketing research and the
knowledge base on effective campaign strategies, messages, and channels.
Efficacy and effectiveness studies use similar measures of behavior change.
However, efficacy studies offer greater opportunities for experimental control,
message exposure, and testing of health communication and marketing theory.
By examining the literature and two in-depth case studies, we identify
advantages and limitations to efficacy studies. We also identify considerations for
when to adopt efficacy and effectiveness methods, alone or in combination.
Finally, we outline a research agenda to investigate issues of internal and
external validity, mode of message presentation, differences between marketing
and message strategies, and behavioral outcomes.
Communication and marketing are growing areas of health research, and
there is evidence that communication and marketing campaigns are effective in
changing health behavior (Snyder and Hamilton, 2002; Evans, 2006). Evaluating
the effectiveness of health communication and marketing efforts is common, but
relatively few rigorous efficacy studies have been conducted in these fields.
An efficacy study is designed to evaluate the effects of an intervention
under optimum conditions whereas an effectiveness study evaluates the effects
of an intervention under real world conditions (Flay 1986). In medical research,
efficacy studies are regarded as necessary in the development of a new drug,
treatment, or technology (Flay 1986). As a result, when medical researchers
conduct an effectiveness study, it is only after the intervention has been shown to
be efficacious. In contrast, in health communication and marketing evaluations,
effectiveness studies are often conducted without previously determining whether
the campaign messages are truly efficacious. Rather, the messages are tested
using qualitative, formative research methods, implemented in the campaign, and
then evaluated as part of an effectiveness study. Without first determining the
efficacy of the campaign messages, results of effectiveness studies can be more
difficult to interpret (Flay 1986). For example, if a campaign produces negative
or null results, is it because the campaign was implemented in an ineffective way
or because the campaign messages were not efficacious? If efficacy was not
established prior to the effectiveness study, this question is difficult to answer. A
well-done process evaluation would provide information on implementation
fidelity, but it would not provide a good answer to the question of whether and if
so how the message failed in terms of audience reaction and receptivity. Efficacy
studies help to address this gap in the evidence base.
Efficacy studies permit researchers to assess message effects under
controlled experimental conditions. Key design features of efficacy studies
include the ability to randomly assign participants to treatment and control
conditions and the researcher having control over treatment condition
participants’ exposure to campaign messages (see Table 1). These features lead
to high internal validity facilitating much stronger causal inferences than typically
possible in effectiveness studies. On the other hand, efficacy studies typically
have lower external validity than do effectiveness studies. This is because
efficacy studies have traditionally been implemented in-person, in laboratory
settings with small samples. Also, they do not allow researchers to make
conclusions about the effectiveness of the campaign messages under real world
conditions nor to make population-based estimates of actual awareness of or
exposure to campaign messages that are possible with effectiveness studies.
Effectiveness studies are typically conducted in the field as a campaign is
running. Most often, campaigns that are the subject of effectiveness studies are
not implemented with evaluation designs in mind. For example, campaigns
typically aim to maximize exposure given limited resources rather than utilize
delivery strategies that generate significant variations in exposure (which is
advantageous for evaluation). As a result, effectiveness designs often do not
allow for randomization of participants into treatment and control conditions nor
do they enable the researcher to manipulate participants’ exposure to campaign
messages. These design features lead to lower internal validity but higher
external validity, the ability to derive conclusions about the effectiveness of the
campaign messages under real world conditions, and ability to determine
estimates of campaign awareness and exposure.
Recently, there has been increased interest in conducting efficacy studies
due to the existence of Web-based research. Opportunities for efficacy research
are greater now because online survey panels allow respondents to view
multimedia and provide data from their own homes with relative ease. Table 1
compares efficacy and effectiveness studies on key design criteria.
Table 1: Key Design Features and Benefits of Efficacy Versus Effectiveness
Feature Efficacy Study
Randomize participants Yes
Manipulate intervention Yes
Longitudinal tracking Yes
Internal Validity Higher
Measure dose response Yes
External Validity Lower
Yes, with limitations
Yes, with limitations
While there are important distinctions between efficacy and effectiveness
methods, are not mutually exclusive and overlap in many important respects.
Efficacy and effectiveness research in health communication and marketing, like
other fields of public health, are inter-related and may together form part of an
integrated model to evaluate intervention effects on health behavior, such as the
RE-AIM framework (Glasgow, 2002; Glasgow et al., 2003). At the same time, as
tools to answer specific health communication and marketing research questions
described below, these methods have distinctive advantages and disadvantages.
Blended studies, or real-world campaign effectiveness trials, represent an
overlapping methodology in which participants are randomized to conditions
(e.g., comparison cities receiving and not receiving a campaign) but are naturally
exposed to messages. Although exposure can be controlled in terms of who gets
it (e.g., which cities, dose of ads), it is still in a real-world setting and can be
confounded in various ways. Exposure also must be measured, with potential
error (requires self-report or media market data). By contrast, in efficacy studies
as discussed in this paper, participants rely on exposing participants under
controlled conditions in which dose and mode of exposure is known.
The main purposes of this paper are to outline the parameters of, and
methods used in, recent health communication and marketing efficacy research
studies. We review recent efficacy research and present two case studies to
illustrate design options and decisions. In the discussion section, we advocate for
greater emphasis on rigorous health communication and marketing efficacy
research and the development of a research agenda.
Much of the outcomes research in health communication and marketing,
utilizes effectiveness designs conducted in real-time, in the media markets or
communities in which messages are delivered. For example, the national
evaluation of American Legacy Foundation’s truth® campaign was based on a
quasi-experimental design in which campaign exposure was measured both from
environmental measures (Gross Rating Points, or GRPs, derived from
Designated Market Areas, or DMAs) and self-reported “confirmed awareness” of
campaign ads. Based on a nationally representative sample of students aged
12-18, this study found that smoking prevalence declined from 25.3% to 18.0%,
and that the truth® campaign was associated with approximately 22% of that
decline (Farrelly et al. 2005).
In substance abuse, the Office of National Drug Control Policy (ONDCP)
developed a large-scale, multi-channel, anti-drug campaign delivered nationwide
though print, radio, TV and Internet. The ONDCP evaluation (Hornik and
Yanovitsky, 2003) was also based on a repeated cross-sectional, quasi-
experimental design that collected data from a nationally representative sample
of adults and youth in DMAs across the U.S. In this instance, the campaign
strategy and evaluation design may have contributed to negative findings for its
effects on marijuana and other substance use (Palmgreen et al., 2007).
Decisions about campaign strategy and evaluation methods may be driven by
political considerations in such large-scale initiatives (Rice and Aikin, 2001).
However, in many instances, such evaluations are impractical or
impossible, limiting opportunities to advance the state of health communication
and marketing research. In the case of paid media campaigns, the reasons for
such limitations include limited ability to generate sufficient levels of exposure to
detect campaign effects using population survey methodologies. There may also
be logistical constraints such as campaign implementation timelines and lengthy
Federal Office of Management and Budget (OMB) reviews in the case of
Federally-funded evaluation contracts that preclude collection of pre-campaign
In the case of unpaid media, or public service announcements (PSAs),
there are additional limitations in that campaign exposure in any given DMA is
typically low. For example, television PSAs typically air at times when audiences
are small (e.g., late at night or on mid-week afternoons). Thus relatively few
individuals in a given DMA receive the message during a set period of time, and
it is therefore difficult to detect campaign effects, resulting in low statistical power.
There is very limited published research on the reliability and validity of
health marketing measures and methods tested in a controlled environment (i.e.,
laboratory type setting), testing of health messages, and development of refined
messages and materials in controlled settings—so-called “forced exposure”
tests. Much of this research comes from the commercial product literature and is
often proprietary and not published.
There are reasons why commercial firms devote substantial resources to
efficacy research on advertising campaigns before a single media spot is
purchased. The high cost of implementing the campaign, let alone the cost of
designing the product to be advertised, requires anticipating what factors may
affect sales. While few companies release data that indicate they wasted money
on poor advertising, examples of failed attempts are easy to find. Chevrolet’s
initial marketing of the Nova automobile (meaning “no va” or “no go” in Spanish)
is a classic example of failure to conduct adequate research. Public health,
because it often has far less to spend on health communication and marketing
campaigns, must be just as careful in its investigations.
Health communication and marketing researchers often choose short-term
measures of responses to public health campaigns that are predictive of long-
term behavioral changes, particularly those linked to well-established and time-
tested behavior change theory (e.g., the Theory of Reasoned Action, Fishbein
and Azjen, 1975). These include recall and recognition of messages, appraisals
of ad appeal, ad processing, perceived efficacy, attitudinal correlates of
campaign-targeted behaviors, behavioral intentions, and behavior change
(Southwell, et al., 2002). These measures underlie a social process of diffusion
that occurs as individuals are exposed to health marketing messages and gain
new information, process the information, form attitudes and intentions related to
the information, and finally implement new behaviors promoted in the message
In addition, short-term, immediate ad reactions borrowed from commercial
advertising (e.g., likeability, credibility, perceived message effectiveness) have
been employed as outcome measures (Dillard et al., 2007; Dillard and Peck,
2001; Biener et al., 2004). These measures are limited, however, in that they are
not tied to theory-driven models of persuasion and have not been adequately
validated (at least in social marketing contexts).
Table 2 illustrates typical measures used in health communication and
marketing efficacy and effectiveness evaluation designs.
Table 2. Measures of Efficacy and Effectiveness Used in Health Communication
and Marketing Studies
Recall and Recognition
Yes, but requires self-
Yes Reactions to message targeting, tailoring, and
specific ad executions
Behavioral precursors (e.g., attitudes, beliefs,
intentions) compared across exposed and
Behavior change compared across exposed
and unexposed populations
Paid media campaign implementation (e.g.,
Unpaid media campaign implementation (e.g.,
Moderators of behavior change
Mediators and other constructs from
communication and behavior change theory
Yes Yes, but requires self-
Yes Yes, but requires self-
No Possible, but rarely
Yes, with limitations
Health communication and marketing efficacy and effectiveness studies
use similar measures in terms of message reactions, moderators, and mediators
of behavior change. Major differences between the two design approaches arise
in that efficacy studies assume, and control the conditions of, exposure to
messages. Audience members are shown messages (e.g., anti-smoking
advertisements) in a defined protocol as part of the experiment. In effectiveness
studies, exposure must be measured, either through self-reported recall and
recognition, or through environmental measures such as Gross Rating Points
(GRPs) or Total Rating Points (TRPs) that capture media market estimates of
population exposure to a message (Evans, Blitstein, Hersey, 2008).
As in other fields of health research, health communication and marketing
efficacy studies typically have higher internal validity and lower external validity,
while effectiveness studies have the reverse characteristics. The main
differences in validity stem from the researcher’s ability to: 1) control the
conditions of message exposure, and 2) measure exposure under conditions in
which audiences are likely to actually encounter messages.
The ultimate goal of health communication and marketing campaigns is to
promote positive changes in health behaviors. Theories of behavior change, such
as the Integrated Model of Behavior Change (Fishbein, 2001) predict that
changes in behavior in the context of social marketing campaigns occur through
a process whereby individuals are first exposed to a health message; react to the
message; form attitudes, beliefs, and intentions around that message; and finally
act upon the information provided in the message by changing behavior.
Unfortunately, research on message effectiveness that employs audience
response methodologies says little about the effects of exposure to social
marketing messages on campaign-targeted behaviors.
For example, experimental studies with well-controlled treatment
conditions offer no evidence on whether exposure to smoking cessation ads
generates an increase in quit attempts among current smokers due to their focus
on immediate cognitive outcomes. This gap in the current body of research is
due largely to the absence of follow-up components in studies that use audience
response methodologies with experimental treatment and control conditions. This
gap is particularly glaring in light of the fact that follow-up components have long
been commonplace in commercial advertising research, upon which health
communication and marketing seeks to build (Evans and Hastings, 2008;
In the following case studies, we illustrate the opportunities to fill this gap
in the literature using efficacy research designs. We start by highlighting an older
case that represents how those few efficacy studies in the literature were
historically conducted, and a recent example that shows how the field is evolving.
The literature on health communication and marketing suggests
opportunities to build the knowledge base through more targeted and effective
use of efficacy study designs. As noted earlier, Web-based panel studies offer
greater opportunities for efficacy studies. Examples include Knowledge Networks
(www.knowledgenetworks.com), which offers a probability-based sample. There
are also numerous “opt-in” market research panels such as Harris Interactive
(www.harrisinteractive.com), e-Rewards (www.erewards.com), and Mysurvey
(www.mysurvey.com). Each of these panels provides the opportunity to expose
participants to multimedia and to select a sample based on specific target
audience characteristics. These features facilitate multi-factorial designs (e.g.,
exposure x time) to test the efficacy of messages and specific advertising
To illustrate the range of potential design options and to demonstrate how
the field has progressed, we describe two case studies below. The first example
focused on testing the effects of antismoking and cigarette advertising on youths’
perceptions of a peer who smokes (Pechmann and Ratneshwar 1994). The
experiment was conducted on-site at four middle schools in California. The
second example focused on testing the efficacy of messages from the Parents
Speak Up National Campaign (PSUNC). The entire experiment was conducted
online with a probability based sample of participants from Knowledge Networks’
Case Study Using a Sample of Students from Four Middle Schools In California:
Tobacco Advertising’s Impact On Youth
We begin by highlighting a health communication efficacy study conducted
over 15 years ago as an example of the history of the field. The study design and
process used in this case study are illustrative of the way in which early efficacy
studies were conducted (e.g., in person, on-site, without longitudinal follow-up).
At the time of this study (Pechmann and Ratneshwar, 1994), tobacco
advertising had been banned from radio and television since the early 1970s.
However, a number of legal marketing channels for cigarettes remained,
including billboards, magazines, in-store posters, and promotional activities at
“adult only” establishments such as bars and clubs (Niederdeppe et al., 2003).
This study also predated the 1998 Master Settlement Agreement between five
major tobacco companies and 46 state Attorneys General that established a
number of restrictions on the marketing of cigarettes toward youth.
An accumulation of evidence suggests that youth are highly aware of
tobacco advertising (Arnett and Terhanian, 1998; Fischer et al., 1991) and that
exposure to and liking cigarette ads are related to future smoking initiation and
escalated consumption (Botvin et al., 1993; Feighery et al., 1998; MacFayden,
Hastings, and MacKintosh, 2001). Researchers have speculated that the images
portrayed in cigarette ads help to create and/or perpetuate favorable myths about
smokers and that antismoking ads are needed to counter these myths (Burton et
al. 1989; Davis 1987; DiFranza et al. 1991; Fischer et al. 1991; Gordon 1986).
Youth smoking and intentions to smoke have been associated with believing that
smokers look cool, sexy, exciting, and mature (Barton et al. 1982; Chassin et al.
1981; Collins et al. 1987; Gordon 1986; Grube et al. 1984; McAlister et al. 1984).
However, no causal link has been established between cigarette advertising and
smoking (whether cigarette ads encouraged smoking or whether smokers paid
more attention to cigarette ads; Moschis 1989). In order to examine the causal
effects of smoking-related advertising in a controlled experimental setting,
Pechmann and Ratneshwar (1994) conducted an efficacy study.
Efficacy Study Design
The study participants were 304 seventh grade students from four middle
schools in southern California. Pechmann and Ratneshwar (1994) indicated that
their primary consideration in selecting the schools for the experiment was
ensuring that the participants roughly mirrored the California student population
with regard to gender, socioeconomic status, and race. Participants were
randomly assigned to one of three advertisement conditions (cigarette ads,
antismoking ads, or unrelated to smoking/control ads). The ads were embedded
in a magazine. Magazine ads were used because they are a primary medium
through which youth are exposed to cigarette ads (Davis 1987; Mazis et al. 1992)
and because using billboard advertisements was not feasible for the experiment.
Four pages of stimulus ads were embedded into a 22-page mock-up magazine
with 10 pages of text and 7 pages of filler ads. The authors used a fictitious, non-
gender oriented magazine modeled after “Entertainment Weekly” to control for
prior exposure. The cigarette ads were chosen from 1990-1991 issues of national
magazines, the antismoking ads were provided by federal and state health
departments and by consumer groups, and the unrelated to smoking/control ads
promoted Reebok shoes, American Airlines, and Estee Lauder eye cream. The
cigarette and anti-smoking ads were pre-tested with seventh graders and the
three cigarette ads and the three antismoking ads that students evaluated as
most effective were selected as the experimental stimuli for the study.
Data were collected in 1991. The students completed a survey with
questions on the use of cigarettes and other products, media exposure levels,
and demographics. Each student was instructed to review the magazine for five
minutes, to look at each and every page, and to look at both the articles and ads.
Later, in a separate study, the students were randomly assigned to read
trait information about a peer who was either identified or not identified as a
smoker. The students read the trait information on school computers. The
students were told to spend as much time as they wanted and the computer
recorded the amount of time spent on each screen. The participants’ perceptions
of their peer and their thoughts and inferences about this person were then
Strengths and Limitations
The Pechmann and Ratneshwar (1994) experiment had some key
advantages. First, it enabled the researchers to develop a true randomized
experiment with ad exposure and control conditions enabling them to be more
confident that any observed associations would be causal. Second, it ensured
that the participants were indeed exposed to the stimulus materials – no need for
self-reported exposure or estimating exposure at the population level.
However, the study has some limitations. The key limitation is that it does
not measure reaction to cigarette or antismoking ads under natural conditions.
Respondents were exposed under controlled, or “forced exposure,” conditions
that were different from how they would be exposed in real life. Audiences might
react differently when exposed to the advertisements outside of the experimental
setting. Second, participants were only exposed to the advertisements once.
The design did not enable repeated exposure or longitudinal tracking within
condition. Third, carrying out an experiment such as this can be costly in both
staff hours (e.g., traveling to the experiment site and administering the
experiment to participants) and production and printing costs for the stimulus
materials. In order for efficacy studies such as this to be practical on a larger
scale, these limitations would need to be overcome.
Case Study Using a Sample from a Probability Based Web-based panel: Parents
Speak Up National Campaign (PSUNC) Evaluation
We highlight a second study as an example of the current state of work in
health communication and marketing efficacy research. The design and process
are illustrative of the enhanced capabilities that are possible when conducting an
efficacy study online, which overcome many of the limitations of Pechmann and
The U.S. Department of Health and Human Services (DHHS) developed
the Parents Speak Up National Campaign (PSUNC). PSUNC is a national multi-
media campaign that has the primary message that parents should talk “early
and often” to their pre-adolescent and adolescent children aged 10-14 years
about delaying the onset of sexual activity. The purpose of PSUNC is to increase
parent-child communication as a proximal behavioral outcome leading to delayed
onset of sexual activity. The campaign was publicly released on June 21, 2007.
PSUNC applies many of the transferable principles of marketing. In
particular, the campaign uses a theory-based behavior change model that
hypothesizes increased parent-child communication will result from positive
message reactions to the public service advertising. Socio-demographic, family
and community-level social influences will moderate these relationships. PSUNC
also develops a credible and likeable “argument” for delaying initiation of sexual
activity by communicating personal (social, educational, career-related)
advantages of abstinence (Petty & Cacioppo, 1986). The campaign also utilizes
risk communication strategies, such as promoting self-efficacy and using fear
appeals, to communicate the health risks of early sexual debut and benefits of
waiting (Slovic et al., 2004; Witte, 1992).
The primary communication channel for the campaign is unpaid Public
Service Announcements (PSAs) designed for a general audience, and targeted
versions of the general PSAs for African American, Hispanic, and Native
American audiences. The campaign also utilizes outdoor advertising (billboards),
bus media, posters, Web banners, and media kits.
In addition to the public awareness campaign, PSUNC includes a second
component, the 4Parents.gov Web site and associated parent and adolescent
guides. The 4parents.gov Web site is intended to provide information (as
opposed to motivational messages) for parents about how to talk with their pre-
teen or teen about waiting to have sex, accuracy of social norms among teens,
perceived barriers among parents to talking with their child, broader related
topics such as parent-child relationship quality and setting goals for the future,
establishing rules and expectations about dating and sex, teaching refusal skills,
characteristics of healthy relationships, consequences of teen sexual activity, and
some special topics (e.g., what to do if the teen has already had sex,
homosexuality, pregnancy, contraception, and legal issues) (U.S. Department of
Health and Human Services, 2007).
Efficacy Study Design
The use of unpaid PSAs as a primary message channel raised a number
of logistical and practical considerations in choosing an appropriate evaluation
method. Because PSAs are typically seen by relatively few viewers at any given
point in time, and are designed to impact audiences with social marketing
messages over a much longer time span than paid media, this implementation
plan implied that most cities and other locations across the country would not be
substantially exposed to PSUNC.
In addition to PSAs, the campaign was designed to be supplemented by
paid media aimed at specific target populations, including African Americans,
Hispanic Americans, and Native Americans in selected DMAs. Those DMAs that
received the additional “dose” of the campaign through the negotiated media
buys would potentially have substantially more exposure. However, whether the
higher doses at these sites would be intensive enough for evaluation was not
known at the time evaluation plans were completed.
The potential for there to be high- and low-dose markets, but uncertainty
as to whether the campaign would be implemented in this way within a timeframe
suitable for in-field data collection led evaluators to conclude that an
effectiveness evaluation would not be an appropriate design option. This was
mainly because the use of PSAs was unlikely to generate significant levels of
audience exposure and awareness, resulting in low statistical power to detect
campaign effects based on self-reported measures of awareness. As a result, an
efficacy study design was selected to evaluate the PSUNC and its messages
under controlled conditions. This represented the best possible opportunity to
maximize the quality of the design and answer key evaluation questions. The
approach entailed selecting a random sample of parents, dividing them into
treatment (PSUNC exposure) and control (no exposure) conditions and collecting
baseline data on their knowledge, attitudes, beliefs, intentions, and behaviors
related to parent-child communication about abstinence and the PSUNC’s central
message to delay the onset of sexual activity. Respondents would be exposed
to the campaign spots (TV, radio, print, as appropriate) through delivery over the
Internet on a home computer, thus under controlled conditions without the
distractions and variability of potential exposure in the real world.
The existence of an existing panel study facilitated a factorial design (e.g.,
message exposure x time) and randomizing participants to condition. Knowledge
Networks was chosen for the PSUNC evaluation because it offered a probability-
based sample of the U.S. population and had sufficient sample of the study
target population. Knowledge Networks panellists all have Internet connectivity
through a computer or Web TV, which enables a television to access the
Internet. Thus, once recruited into the study, participants can readily complete
surveys and view and hear online video, audio, and print (static page) content.
The resulting PSUNC evaluation study is a randomized controlled trial
(RCT) of parents of children aged 10 to 14, the campaign’s target audience.
Mothers and fathers, sampled from the overall Knowledge Networks panel, were
randomly assigned to control, treatment (exposure to a core set of PSUNC
messages), and treatment plus booster (core plus additional and more frequent
PSUNC messages) conditions and surveyed at baseline (prior to message
exposure) and 4 follow up time points at 4 weeks, 6 months, 12 months, and 18
months. The purpose of this RCT is to determine the efficacy of PSUNC
messages in increasing timing and frequency, and improving content of parent-
child communication over time. The study also examines the role of potential
mediators such as parent self-efficacy, outcome efficacy, attitudes and beliefs, as
well as potential moderating influences such as socio-demographics, family
dynamics, media and community-level social influences. Figure 1 presents the
conceptual framework for the PSUNC RCT.
Figure 1. PSUNC RCT Conceptual Framework
Baseline data were collected from August to October 2007, and the first
follow up data were collected in October and November 2007. Initial findings from
the study appear elsewhere (Evans and Davis, 2008). Key evaluation questions
answered in this design include:
• What are parents’ social cognitions and behaviors prior to exposure to
• What is the change in social cognitions and parent-child communication
from one time point to another in absence of exposure to campaign
• Other abstinence promotion activities
Parents’ other media influences
• Internet use
• Health information sources
• Initiation (by parent or child)
• Family norms (parent-child interaction)
• Family religiosity
• Parent perceptions of child’s sexual
Parents Speak Up
national campaign message
awareness and receptivity
Parents Speak Up
national campaign exposure
Teen social cognitions
Teen sexual activity
= Direct effect
= Long-term effect
• Parent perceptions of child’s past
• Parent perceptions of child’s
participation in school-based sex
• Parent perceptions of child’s
pubertal developmental stage
• Parental involvement
• Parental monitoring
= PSUNC (independent variable)
• Does parent-child communication increase more among parents who are
exposed to campaign messages compared to those who are not
• Does parent-child communication increase more among parents who
received the “high” exposure condition compared to those who received
the “normal” exposure condition?
• Do parents with lower intentions to communicate with their children about
abstinence at the beginning of the study tend to have greater increases in
intentions over time?
Strengths and Limitations
The final PSUNC evaluation design had several key advantages over
effectiveness design alternatives. First, it enabled researchers to develop a true
randomized experiment with campaign exposure (treatment), booster, and
control conditions. Second, it allows the application of experimental designs to
large population-based samples of participants, mitigating the problems of low
external validity associated with earlier small-sample laboratory efficacy studies.
While there are examples of such designs in social marketing field experiments
(Evans, Necheles, Longjohn, and Christoffel, 2007), they are relatively rare.
Most importantly, the efficacy approach permitted dose-response
comparisons. Second, it enabled longitudinal tracking within condition, which
creates the opportunity for casual attribution of campaign effects. Third, it
avoided troublesome issues of potentially low campaign exposure that plagued
the available effectiveness design options.
However, like Pechmann and Ratneshwar (1994) the study has the key
limitation that it does not measure actual campaign implementation, or reaction to
campaign ads under natural conditions. Respondents were exposed under
controlled, or “forced exposure,” conditions that will be different from how they
would be exposed in real life. Thus results of the experiment do not necessarily
imply how exposed audiences will react when the campaign is implemented.
In this paper, we highlighted some of the key differences between efficacy
versus effectiveness evaluations in health communication and marketing. Using
a dual case study approach, we illustrated how efficacy studies can be
implemented, what research questions they address, what measures they use,
and how efficacy research methods have evolved with the advent of online
surveys and Web panels. Below, we summarize some of the advantages of
using an efficacy methodology to evaluate health communication and marketing
campaigns, discuss key considerations when employing an efficacy approach,
and outline a future research agenda.
Efficacy study designs offer advantages over effectiveness designs.
Specifically, they offer health communication and marketing researchers:
• an important opportunity to answer research questions about
message reactions, dose-response relationships, and causal
effects of message exposure not easily, or as rigorously, answered
by effectiveness studies;
• solutions to problems in terms of comparison conditions and
experimental design facing effectiveness studies;
• an opportunity to optimize exposure to messages independent of
the timing or logistics of campaign implementation.
Additionally, online efficacy studies based on existing panels are
significantly cheaper than field-based studies that rely on telephone surveys. This
is because 1) recruitment is easier, and 2) surveys are self administered and
thus do not require telephone or in-person interviewers. Some campaigns are
only piloted in select communities and have severely restricted advertising
budgets and restricted evaluation budgets to match. Efficacy studies are an
economical evaluation tool compared to field-based studies.
There are several important considerations in choosing when to conduct
efficacy studies. First, are there important theory-based questions about
message effectiveness to be answered? As discussed above, efficacy studies
allow for much more experimental control and are thus best suited to answering
questions of message effectiveness. This is because factors related to message
effectiveness (e.g., stylistic features, content of messages) can be isolated, built
into the design, and measured.
Second, as illustrated by the PSUNC case study, do evaluators anticipate
low levels of measurable exposure to the health communication and marketing
campaign? As in many unpaid media campaigns exposure may occur, but not be
measurable in an effectiveness design for practical reasons.
If these conditions are met, then conducting campaign research under
controlled conditions may be the best evaluation option. Finally, is the campaign
aimed at a highly specific target population, such as people within a limited age
range, or who engage in a specific behavior? If so, there may be existing survey
panels that provide easy access to populations of interest.
Efficacy studies can be used in several different ways: 1) alone to
determine the efficacy of campaign messages prior to implementation of a
campaign (ideal); 2) alone to determine the efficacy of campaign messages on a
campaign that is already being implemented – this can be an option when
designing an effectiveness/field-based evaluation study is not feasible for a
number of reasons – timing, anticipated low exposure or reach, funding, etc.; or
(3) in combination with other methods as part of a comprehensive campaign
evaluation where results from different data collection efforts (e.g., using
qualitative methods) can be triangulated (Rossi and Freeman, 1993).
Given the advantages of efficacy studies, and advances in technology, the
research agenda in this area should be further developed. In particular, efficacy
studies should be conducted to determine how messages affect audiences’
health knowledge, attitudes, beliefs, intentions, and behaviors. An efficacy study
may be chosen as the preferred evaluation design for a few different reasons.
First, an efficacy study may be conducted to ensure that campaign messages are
efficacious prior to a health communication or marketing campaign’s
implementation. As a result, only efficacious messages would be used in a
subsequent campaign and later evaluated in a field-based effectiveness trial.
Second, an efficacy study is an option to test the efficacy of campaign messages
for an existing campaign when a field-based effectiveness trial is not feasible
(e.g., due to timing or low exposure). Finally, an efficacy study may be used in
combination with other methods. In this last scenario, results from different data
collection methods can be triangulated (Rossi and Freeman 1993) to create a
more comprehensive picture of the effectiveness of a campaign.
In short, the advantages of efficacy designs in health communication and
marketing research, and gaps in the evidence base on message effectiveness
described earlier, suggest the need to develop a targeted research agenda. Such
an agenda would build on the work described in this paper, and have the
following broad characteristics:
• Develop techniques to maximize external validity of efficacy studies
while maintaining already high internal validity
• Conduct online access mode studies to compare findings
• Examine the effects of different marketing strategies and message
strategies across different health topics
• Conduct comparative studies of health and behavioral outcomes
observed in efficacy versus effectiveness studies
This paper suggests the importance of efficacy research in health
communication and marketing. In the field as a whole, there has been increased
dialogue recently about the problem of weak evaluation design in many health
communication campaigns and resulting lack of evidence of effectiveness (Noar,
2006; Noar, Palmgreen, Chabot, et al., in press). In particular, there is a dearth of
evidence on how message and marketing strategies work, and efficacy methods
can help to build the evidence base in these areas. With a rapidly evolving media
environment, there are new and expanding opportunities to reach highly targeted
audiences (Lefebvre, 2007). As a result, efficacy research is needed to
understand message effectiveness in this new world.
Aitken, P.P., Eadie, D.R., Hastings, G.B., and A.J. Haywood. Predisposing
Effects of Cigarette Advertising on Children’s Intentions to Smoke When
Older. British Journal of Addiction, 1991; 86: 383-390.
Aitken, P.P., and D.R. Eadie. Reinforcing Effects of Cigarette Advertising
on Under-Age Smoking. British Journal of Addiction, 1990; 85: 399-412.
Arnett, J.J., and G. Terhanian. “Adolescents’ Responses to Cigarette
Advertisemetns: Links between Exposure, Liking, and the Appeal of
Smoking.” Tobacco Control, 1998; 7(2):129-133.
Arnett, J.J., and G. Terhanian. “Adolescents’ Responses to Cigarette
Attributions and Smoking Intentions among Seventh Grade Students.
Journal of Applied Social Psychology, 1998; 19(8): 656-664.
Barton, J. L. Chassin, C.C. Presson and S. J. Sherman. Social Image Factors as
Motivators of Smoking Initiation in Early and Middle Adolescence. Child
Development, 1982; 53: 1499-1511.
Biener, L., J. Ming, E.A. Gilpin, and A.B. Albers. “The Impact of Emotional Tone,
Message, and Broadcast Parameters in Youth Anti-smoking
Advertisements.” Journal of Health Communication, 2004; 9(3):259-274.
Botvin, G.J., Goldberg, C.J., Botvin, E.M., and L. Dusenbury. “Smoking Behavior
of Adolescents Exposed to Cigarette Advertising.” Public Health Reports,
Burton, D. S. Sussman, W.B. Hansen, C. A. Johnson and B.R. Flay. Image
Attributions and Smoking Intentions among Seventh Grade Students.
Journal of Applied Social Psychology, 1989;19(8): 656-664.
Chassin, L., C. C. Presson, S. J. Sherman, E. Corty and R.W. Olshavsky. Self
Images and Cigarette Smoking in Adolescence." Personality and Social
Psychology Bulletin, 1981;7 (December), 670-676.
Collins, L.M., S. Sussman, J.M. Rauch, C.W. Dent, C.A. Johnson, W.B. Hansen
and B.R. Flay. Psychosocial Predictors of Young Adolescent Cigarette
Smoking: A Sixteen-Month, Three-Wave Longitudinal Study. Journal of
Applied Social Psychology, 1987;17(6): 554-573.
Davis, R.M. 1987. Current Trends in Cigarrette Advertising and Marketing. New
England Journal of Medicine, 316: 725-732.
DiFranza, J.R., Richards, J.W., Paulman, P.M., Wolf-Gillespie, N., and C.
Fletcher, “Effects of Cigarette Advertising on Children’s Intentions to
Smoke When Older. British Journal of Addiction, 86: 383-390.
Dillard, J. P., Weber, K. M., & Vail, Renata, G. (2007). The relationship between
the perceived and actual effectiveness of persuasive messages: A meta-
analysis with implications for formative campaign research. Journal of
Communication, 57, 613-631.
Dillard, J.P., and E. Peck. “Persuasion and the Structure of Affect: Dual Systems
and Discrete Emotions as Complementary Models.” Human
Communication Research, 2001; 27(1):38-68.
Evans, W.D., Davis, K.C. Social Marketing Research with New Media: Case
Study of the Parents Speak Up National Campaign Evaluation. Cases in
Public Health Communication and Marketing. 2008.
Evans, W.D., and G. Hastings. “Public Health Branding: Recognition, Promise,
and Delivery of Healthy Lifestyles.” In Public Health Branding: Applying
Marketing for Social Change, W.D. Evans and G. Hastings (Eds.), Oxford
University Press. London, United Kingdom. 2008.
Evans, W.D., J. Necheles, M. Longjohn, K. Kristoffel. “The 5-4-3-2-1 Go!
Intervention: Social Marketing for Nutrition.” Journal of Nutrition Education
and Behavior, 2007; 39(2)(S1): S55-S59.
Farrelly, M.C., K.C. Davis, M.L. Haviland, P. Messeri, and C.G. Healton.
“Evidence of a Dose-Response Relationship Between ‘truth’ Antismoking
Ads and Youth Smoking Prevalence.” American Journal of Public Health,
Fazio, R.H., and C.J. Williams. “Attitude Accessibility as a Moderator of the
Attitude Perception and Attitude-Behavior Relations.” Journal of
Personality and Social Psychology, 1986; 51:505-514.
Feighery E., Borzekowski, D.L.G., Schooler C., and J. Flora. “Seeing, Wanting,
Owning: The Relationship between Receptivity to Tobacco Marketing and
Smoking Susceptibility in Young People.” Tobacco Control, 1998; 7:123-
Fischer, P.M., M.P. Schwartz, J.W. Richards, A.O. Goldstein, and T.H. Rojas.
Brand Logo Recognition by Children Aged 3 to 6 Years. Journal of the
American Medical Association, 1991;11: 3145-3148.
Fishbein, M. “The Role of Theory in HIV Prevention.” AIDS Care, 2001;
Fishbein, M., and I. Ajzen. Belief, Attitude, Intention, and Behavior: An
Introduction to Theory and Research. Reading, MA: Addison Wesley.
Flay, B.R. Efficacy and effectiveness trials (and other phases of research) in the
development of health promotion programs. Prev Med, 1986;15(5): 451-
Glasgow, R. E., Lichtenstein, E., & Marcus, A. C. Why don’t we see more
translation of health promotion research to practice? Rethinking the
efficacy-to-effectiveness transition. American Journal of Public Health,
Glasgow, R.E. Evaluation of theory-based interventions: The RE-AIM model. In
K. Glanz, F.M. Lewis, & B.K. Rimer (Eds) Health behavior and health
education: Theory, research, and practice (3rd Edition). (pp. 531-544).
San Francisco, CA: John Wiley & Sons. 2002.
Goldstein, A.O., Fischer, P.M., J.W. Richards, and D. Creten. Relationship
Between High School Student Smoking and Recognition of Cigarette
Advertisements. Journal of Pediatrics, 1987;110: 488-491.
Gordon, N.P. Never Smokers, Triers and Current Smokers: RJR Nabisco’s
Cartoon Camel Promotes Camel Cigarettes to Children. Journal of the
American Medical Association, 1986;11: 3145-3148.
Grube, J., Wier, I.L., Getzlaf, S., and M. Rokeach. Own Value System, Value
Images, and Cigarette Smoking. Personality and Social Psychology
Bulletin, 1984;10: 306-313.
Hornik R, and I. Yanovitsky. “Using theory to design evaluations of
communication campaigns: the case of the national youth anti-drug media
campaign.” Commun Theory, 2003;13:204-24.
Jenkins, J. Tobacco Advertising and Children: Some Canadian Findings.
International Journal of Advertising, 1988;7(4): 357-363.
Lefebvre, R.C. “The New Technology: The Consumer as Participant Rather than
Target Audience.” Presented at the 11th Innovations in Social Marketing
Conference, Baltimore, MD, 2007.
MacFayden L., Hastings G., and A.M. MaccKintosh. “Cross Sectional Study of
Young People’s Awareness of and Involvement with Tobacco Marketing.”
British Medical Journal, 2001; 322:513-517.
Mazis, J.B., D.J. Ringold, E.S. Perry, and Daniel W. Denman. Perceived Age and
Attractiveness of Models in Cigarette Advertisements. Journal of
Marketing, 1992; 56: 22-37.
McAlister, A.L., J.A. Krosnick and M.A. Milburn. Causes of Adolescent Cigarette
Smoking: Tests of a Structural Equation Model. Social Psychology
Quarterly, 1984;47(1): 24-36.
McElrath, Y., M. Wakefield, E. Ruel, et al. “The Effect of Antismoking
Advertisement Executional Characteristics on Youth Comprehension,
Appraisal, Recall, and Engagement.” Journal of Health Communication,
McGuire, W.J. “Behavioral Medicine, Public Health and Communication
Theories.” Health Education, 1981;12(3):8-13.
Moschis, G.P. “Point of View: Cigarette Advertising and Young Smokers.”
Journal of Advertising Research, 1989;29: 51-60.
Niederdeppe J., Lindsey D., Girlando M., Ulasevich A., and M.C. Farrelly. First
Look Report 12: Exposure to Pro-Tobacco Messages Among Teens and
Young Adults: Results from Three National Surveys. American Legacy
Foundation: Washington, DC. 2003.
Noar, S. M. (2006). A 10-year retrospective of research in health mass media
campaigns: Where do we go from here? Journal of Health
Communication, 11(1), 21-42.
Noar, S. M., Palmgreen, P., Chabot, M., Dobransky, N., & Zimmerman, R. S. A
10-year systematic review of HIV/AIDS mass communication campaigns:
Have we made progress? Journal of Health Communication. In press.
Palmgreen, P. C., Lorch, E. P., Stephenson, M. T., Hoyle, R. H., & Donohew, L.
Effects of the Office of National Drug Control Policy’s marijuana initiative
campaign on high-sensation-seeking adolescents. American Journal of
Public Health, 2007;97: 1644-1649.
Pechmann, C., and E.T. Reibling. 2000. “Planning an Effective Antismoking
Mass Media Campaign Targeting Adolescents.” Journal of Public Health
Management Practice, 6(3):80-94.
Pechmann, C., and S. Ratneshwar. 1994. "The Effects of Anti-Smoking and
Cigarette Advertising on Young Adolescents' Perceptions of Peers Who
Smoke." Journal of Consumer Research, 21:236-251.
Petty, R.E., and J.T. Cacioppo. Communication and Persuasion: Central and
Peripheral Routes to Attitude Change. New York: Springer-Verlag. 1986.
Jaffe, R.D., and D. Murray. RJR Nabisco’s Cartoon Camel Promotes Camel
Cigarettes to Children. Journal of the American Medical Association,
on Under-Age Smoking. British Journal of Addiction, 85: 399-412.
Rice, R.E., and Atkin, C.K. (Eds.), Public communication campaigns, (3rd Edition)
. Thousand Oaks, California: Sage. 2001.
Rossi, P., and H. Freeman. Evaluation: A Systematic Approach. Newbury Park,
CA: Sage Publications, 5th Edition. 1993
Slovic, P., Finucane, M., Peters, E., and D. MacGregor. Risk as Analysis and
Risk as Feelings: Some Thoughts about Affect, Reason, Risk, and
Rationality. Risk Analysis, 2004; 24(2):311-322.
Snyder, L.B., and M.A. Hamilton. Meta-analysis of U.S. health campaign effects
on behavior: Emphasize enforcement, exposure, and new information,
and beware the secular trend. In R. Hornik (Ed.) Public health
34 Download full-text
communication: Evidence for behavior change. Hillsdale, NJ: Lawrence
Erlbaum Associates. 2002.
Southwell, B.G., Barmada, C.H., Hornik, R.C., and D.M. Maklan. “Can we
Measure Exposure? Evidence from a National Campaign.” Journal of
Health Communication, 2002; 7:445-453.
United States Department of Health and Human Services. Parents Speak Up
National Campaign. Washington, DC: Noral Group. 2006.
United States Department of Health and Human Services. 4parents.gov [Web
Page]. Retrieved October 29, 2007 from the World Wide Web
Witte, K. Putting the fear back into fear appeals: the extended parallel process
model. Communication Monographs, 1992; 59:329–49.