Efficacy methods to evaluate health communication and marketing campaigns.
ABSTRACT Communication and marketing are growing areas of health research, but relatively few rigorous efficacy studies have been conducted in these fields. In this article, we review recent health communication and marketing efficacy research, present two case studies that illustrate some of the considerations in making efficacy design choices, and advocate for greater emphasis on rigorous health communication and marketing efficacy research and the development of a research agenda. Much of the outcomes research in health communication and marketing, especially mass media, utilizes effectiveness designs conducted in real time, in the media markets or communities in which messages are delivered. Such evaluations may be impractical or impossible, however, imiting opportunities to advance the state of health communication and marketing research and the knowledge base on effective campaign strategies, messages, and channels. Efficacy and effectiveness studies use similar measures of behavior change. Efficacy studies, however, offer greater opportunities for experimental control, message exposure, and testing of health communication and marketing theory. By examining the literature and two in-depth case studies, we identify advantages and limitations to efficacy studies. We also identify considerations for when to adopt efficacy and effectiveness methods, alone or in combination. Finally, we outline a research agenda to investigate issues of internal and external validity, mode of message presentation, differences between marketing and message strategies, and behavioral outcomes.
- SourceAvailable from: Doug Evans
- [Show abstract] [Hide abstract]
ABSTRACT: Communication to promote behaviours like energy saving can use significant resources. What is less clear is the comparative value of different approaches available to communicators. While it is generally agreed that ‘bottom-up’ approaches, where individuals are actively involved rather than passive, are preferable to ‘top-down’ authority-led projects, there is a dearth of evidence that verifies why this should be. Additionally, while the literature has examined the mechanics of the different approaches, there has been less attention paid to the associated psychological implications. This paper reports on an exploratory comparative study that examined the effects of six distinct communication activities. The activities used different communication approaches, some participative and others more top-down informational. Two theories, from behavioural studies and communication, were used to identify key variables for consideration in this field-based evaluation. The evaluation aimed to assess not just which activity might be most successful, as this has limited generalisability, but to also gain insight into what psychological impacts might contribute to success. Analysis found support for the general hypothesis that bottom-up approaches have more impact on behaviour change than top-down. The study also identified that, in this instance, the difference in reported behaviour across the activities related partly to the extent to which intentions to change behaviour were implemented. One possible explanation for the difference in reported behaviour change across the activities is that a bottom-up approach may offer a supportive environment where participants can discuss progress with like-minded individuals. A further possible explanation is that despite controlling for intention at an individual level, the pre-existence of strong intentions may have an effect on group success. These suggestive findings point toward the critical need for additional and larger-scale studies. The challenges associated with field-based evaluative research and the role of theory are discussed. The design approach and measures used in this study may be useful to other evaluations that seek to compare different communicative approaches.Energy Efficiency 01/2013; 6(1). · 1.15 Impact Factor
- [Show abstract] [Hide abstract]
ABSTRACT: The use of mobile phone technologies for health promotion and disease prevention has advanced rapidly in recent years. Text4baby is a theory-based mobile health (mHealth) program in which text messages are delivered to pregnant women and new mothers to improve their health care beliefs and behaviors and improve health status and clinical outcomes. Recent evaluations of Text4baby have found that it improves targeted health attitudes and beliefs, but effects on behavior have not yet been determined.Journal of Medical Internet Research 01/2014; 16(5):e131. · 3.77 Impact Factor
Efficacy Methods to Evaluate Health Communication and Marketing Campaigns
W. Douglas Evans1
1 The George Washington University
2 RTI International
Running Head: Efficacy Methods to Evaluate Health Communication and
Send all correspondence to:
W. Douglas Evans, Ph.D.
The George Washington University
2175 K Street, NW, Suite 700
Washington, DC 20037
Voice: (202) 416-0496
Fax: (202) 416-0433
Communication and marketing are growing areas of health research, but
relatively few rigorous efficacy studies have been conducted in these fields. In
this paper, we review recent health communication and marketing efficacy
research, present two case studies that illustrate some of the considerations in
making efficacy design choices, and advocate for greater emphasis on rigorous
health communication and marketing efficacy research and the development of a
Much of the outcomes research in health communication and marketing,
especially mass media, utilizes effectiveness designs conducted in real-time, in
the media markets or communities in which messages are delivered. However,
such evaluations may be impractical or impossible, limiting opportunities to
advance the state of health communication and marketing research and the
knowledge base on effective campaign strategies, messages, and channels.
Efficacy and effectiveness studies use similar measures of behavior change.
However, efficacy studies offer greater opportunities for experimental control,
message exposure, and testing of health communication and marketing theory.
By examining the literature and two in-depth case studies, we identify
advantages and limitations to efficacy studies. We also identify considerations for
when to adopt efficacy and effectiveness methods, alone or in combination.
Finally, we outline a research agenda to investigate issues of internal and
external validity, mode of message presentation, differences between marketing
and message strategies, and behavioral outcomes.
Communication and marketing are growing areas of health research, and
there is evidence that communication and marketing campaigns are effective in
changing health behavior (Snyder and Hamilton, 2002; Evans, 2006). Evaluating
the effectiveness of health communication and marketing efforts is common, but
relatively few rigorous efficacy studies have been conducted in these fields.
An efficacy study is designed to evaluate the effects of an intervention
under optimum conditions whereas an effectiveness study evaluates the effects
of an intervention under real world conditions (Flay 1986). In medical research,
efficacy studies are regarded as necessary in the development of a new drug,
treatment, or technology (Flay 1986). As a result, when medical researchers
conduct an effectiveness study, it is only after the intervention has been shown to
be efficacious. In contrast, in health communication and marketing evaluations,
effectiveness studies are often conducted without previously determining whether
the campaign messages are truly efficacious. Rather, the messages are tested
using qualitative, formative research methods, implemented in the campaign, and
then evaluated as part of an effectiveness study. Without first determining the
efficacy of the campaign messages, results of effectiveness studies can be more
difficult to interpret (Flay 1986). For example, if a campaign produces negative
or null results, is it because the campaign was implemented in an ineffective way
or because the campaign messages were not efficacious? If efficacy was not
established prior to the effectiveness study, this question is difficult to answer. A
well-done process evaluation would provide information on implementation
fidelity, but it would not provide a good answer to the question of whether and if
so how the message failed in terms of audience reaction and receptivity. Efficacy
studies help to address this gap in the evidence base.
Efficacy studies permit researchers to assess message effects under
controlled experimental conditions. Key design features of efficacy studies
include the ability to randomly assign participants to treatment and control
conditions and the researcher having control over treatment condition
participants’ exposure to campaign messages (see Table 1). These features lead
to high internal validity facilitating much stronger causal inferences than typically
possible in effectiveness studies. On the other hand, efficacy studies typically
have lower external validity than do effectiveness studies. This is because
efficacy studies have traditionally been implemented in-person, in laboratory
settings with small samples. Also, they do not allow researchers to make
conclusions about the effectiveness of the campaign messages under real world
conditions nor to make population-based estimates of actual awareness of or
exposure to campaign messages that are possible with effectiveness studies.
Effectiveness studies are typically conducted in the field as a campaign is
running. Most often, campaigns that are the subject of effectiveness studies are
not implemented with evaluation designs in mind. For example, campaigns
typically aim to maximize exposure given limited resources rather than utilize
delivery strategies that generate significant variations in exposure (which is
advantageous for evaluation). As a result, effectiveness designs often do not
allow for randomization of participants into treatment and control conditions nor
do they enable the researcher to manipulate participants’ exposure to campaign
messages. These design features lead to lower internal validity but higher
external validity, the ability to derive conclusions about the effectiveness of the
campaign messages under real world conditions, and ability to determine
estimates of campaign awareness and exposure.
Recently, there has been increased interest in conducting efficacy studies
due to the existence of Web-based research. Opportunities for efficacy research
are greater now because online survey panels allow respondents to view
multimedia and provide data from their own homes with relative ease. Table 1
compares efficacy and effectiveness studies on key design criteria.
Table 1: Key Design Features and Benefits of Efficacy Versus Effectiveness
Feature Efficacy Study
Randomize participants Yes
Manipulate intervention Yes
Longitudinal tracking Yes
Internal Validity Higher
Measure dose response Yes
External Validity Lower
Yes, with limitations
Yes, with limitations
While there are important distinctions between efficacy and effectiveness
methods, are not mutually exclusive and overlap in many important respects.
Efficacy and effectiveness research in health communication and marketing, like
other fields of public health, are inter-related and may together form part of an
integrated model to evaluate intervention effects on health behavior, such as the
RE-AIM framework (Glasgow, 2002; Glasgow et al., 2003). At the same time, as
tools to answer specific health communication and marketing research questions
described below, these methods have distinctive advantages and disadvantages.
Blended studies, or real-world campaign effectiveness trials, represent an
overlapping methodology in which participants are randomized to conditions
(e.g., comparison cities receiving and not receiving a campaign) but are naturally
exposed to messages. Although exposure can be controlled in terms of who gets
it (e.g., which cities, dose of ads), it is still in a real-world setting and can be
confounded in various ways. Exposure also must be measured, with potential
error (requires self-report or media market data). By contrast, in efficacy studies
as discussed in this paper, participants rely on exposing participants under
controlled conditions in which dose and mode of exposure is known.
The main purposes of this paper are to outline the parameters of, and
methods used in, recent health communication and marketing efficacy research
studies. We review recent efficacy research and present two case studies to
illustrate design options and decisions. In the discussion section, we advocate for
greater emphasis on rigorous health communication and marketing efficacy
research and the development of a research agenda.
Much of the outcomes research in health communication and marketing,
utilizes effectiveness designs conducted in real-time, in the media markets or
communities in which messages are delivered. For example, the national
evaluation of American Legacy Foundation’s truth® campaign was based on a
quasi-experimental design in which campaign exposure was measured both from
environmental measures (Gross Rating Points, or GRPs, derived from
Designated Market Areas, or DMAs) and self-reported “confirmed awareness” of
campaign ads. Based on a nationally representative sample of students aged
12-18, this study found that smoking prevalence declined from 25.3% to 18.0%,
and that the truth® campaign was associated with approximately 22% of that
decline (Farrelly et al. 2005).
In substance abuse, the Office of National Drug Control Policy (ONDCP)
developed a large-scale, multi-channel, anti-drug campaign delivered nationwide
though print, radio, TV and Internet. The ONDCP evaluation (Hornik and
Yanovitsky, 2003) was also based on a repeated cross-sectional, quasi-
experimental design that collected data from a nationally representative sample
of adults and youth in DMAs across the U.S. In this instance, the campaign
strategy and evaluation design may have contributed to negative findings for its
effects on marijuana and other substance use (Palmgreen et al., 2007).
Decisions about campaign strategy and evaluation methods may be driven by
political considerations in such large-scale initiatives (Rice and Aikin, 2001).
However, in many instances, such evaluations are impractical or
impossible, limiting opportunities to advance the state of health communication
and marketing research. In the case of paid media campaigns, the reasons for
such limitations include limited ability to generate sufficient levels of exposure to
detect campaign effects using population survey methodologies. There may also
be logistical constraints such as campaign implementation timelines and lengthy
Federal Office of Management and Budget (OMB) reviews in the case of
Federally-funded evaluation contracts that preclude collection of pre-campaign
In the case of unpaid media, or public service announcements (PSAs),
there are additional limitations in that campaign exposure in any given DMA is
typically low. For example, television PSAs typically air at times when audiences
are small (e.g., late at night or on mid-week afternoons). Thus relatively few
individuals in a given DMA receive the message during a set period of time, and
it is therefore difficult to detect campaign effects, resulting in low statistical power.
There is very limited published research on the reliability and validity of
health marketing measures and methods tested in a controlled environment (i.e.,
laboratory type setting), testing of health messages, and development of refined
messages and materials in controlled settings—so-called “forced exposure”
tests. Much of this research comes from the commercial product literature and is
often proprietary and not published.
There are reasons why commercial firms devote substantial resources to
efficacy research on advertising campaigns before a single media spot is
purchased. The high cost of implementing the campaign, let alone the cost of
designing the product to be advertised, requires anticipating what factors may
affect sales. While few companies release data that indicate they wasted money
on poor advertising, examples of failed attempts are easy to find. Chevrolet’s
initial marketing of the Nova automobile (meaning “no va” or “no go” in Spanish)
is a classic example of failure to conduct adequate research. Public health,
because it often has far less to spend on health communication and marketing
campaigns, must be just as careful in its investigations.
Health communication and marketing researchers often choose short-term
measures of responses to public health campaigns that are predictive of long-
term behavioral changes, particularly those linked to well-established and time-
tested behavior change theory (e.g., the Theory of Reasoned Action, Fishbein
and Azjen, 1975). These include recall and recognition of messages, appraisals
of ad appeal, ad processing, perceived efficacy, attitudinal correlates of
campaign-targeted behaviors, behavioral intentions, and behavior change
(Southwell, et al., 2002). These measures underlie a social process of diffusion
that occurs as individuals are exposed to health marketing messages and gain
new information, process the information, form attitudes and intentions related to
the information, and finally implement new behaviors promoted in the message
In addition, short-term, immediate ad reactions borrowed from commercial
advertising (e.g., likeability, credibility, perceived message effectiveness) have
been employed as outcome measures (Dillard et al., 2007; Dillard and Peck,
2001; Biener et al., 2004). These measures are limited, however, in that they are
not tied to theory-driven models of persuasion and have not been adequately
validated (at least in social marketing contexts).
Table 2 illustrates typical measures used in health communication and
marketing efficacy and effectiveness evaluation designs.
Table 2. Measures of Efficacy and Effectiveness Used in Health Communication
and Marketing Studies
Recall and Recognition
Yes, but requires self-
Yes Reactions to message targeting, tailoring, and
specific ad executions
Behavioral precursors (e.g., attitudes, beliefs,
intentions) compared across exposed and
Behavior change compared across exposed
and unexposed populations
Paid media campaign implementation (e.g.,
Unpaid media campaign implementation (e.g.,
Moderators of behavior change
Mediators and other constructs from
communication and behavior change theory
Yes Yes, but requires self-
Yes Yes, but requires self-
No Possible, but rarely
Yes, with limitations
Health communication and marketing efficacy and effectiveness studies
use similar measures in terms of message reactions, moderators, and mediators
of behavior change. Major differences between the two design approaches arise
in that efficacy studies assume, and control the conditions of, exposure to
messages. Audience members are shown messages (e.g., anti-smoking
advertisements) in a defined protocol as part of the experiment. In effectiveness
studies, exposure must be measured, either through self-reported recall and
recognition, or through environmental measures such as Gross Rating Points
(GRPs) or Total Rating Points (TRPs) that capture media market estimates of
population exposure to a message (Evans, Blitstein, Hersey, 2008).
As in other fields of health research, health communication and marketing
efficacy studies typically have higher internal validity and lower external validity,
while effectiveness studies have the reverse characteristics. The main
differences in validity stem from the researcher’s ability to: 1) control the
conditions of message exposure, and 2) measure exposure under conditions in
which audiences are likely to actually encounter messages.
The ultimate goal of health communication and marketing campaigns is to
promote positive changes in health behaviors. Theories of behavior change, such
as the Integrated Model of Behavior Change (Fishbein, 2001) predict that
changes in behavior in the context of social marketing campaigns occur through
a process whereby individuals are first exposed to a health message; react to the
message; form attitudes, beliefs, and intentions around that message; and finally
act upon the information provided in the message by changing behavior.
Unfortunately, research on message effectiveness that employs audience
response methodologies says little about the effects of exposure to social
marketing messages on campaign-targeted behaviors.
For example, experimental studies with well-controlled treatment
conditions offer no evidence on whether exposure to smoking cessation ads
generates an increase in quit attempts among current smokers due to their focus
on immediate cognitive outcomes. This gap in the current body of research is
due largely to the absence of follow-up components in studies that use audience
response methodologies with experimental treatment and control conditions. This
gap is particularly glaring in light of the fact that follow-up components have long
been commonplace in commercial advertising research, upon which health
communication and marketing seeks to build (Evans and Hastings, 2008;
In the following case studies, we illustrate the opportunities to fill this gap
in the literature using efficacy research designs. We start by highlighting an older
case that represents how those few efficacy studies in the literature were
historically conducted, and a recent example that shows how the field is evolving.
The literature on health communication and marketing suggests
opportunities to build the knowledge base through more targeted and effective
use of efficacy study designs. As noted earlier, Web-based panel studies offer
greater opportunities for efficacy studies. Examples include Knowledge Networks
(www.knowledgenetworks.com), which offers a probability-based sample. There
are also numerous “opt-in” market research panels such as Harris Interactive
(www.harrisinteractive.com), e-Rewards (www.erewards.com), and Mysurvey
(www.mysurvey.com). Each of these panels provides the opportunity to expose
participants to multimedia and to select a sample based on specific target
audience characteristics. These features facilitate multi-factorial designs (e.g.,
exposure x time) to test the efficacy of messages and specific advertising
To illustrate the range of potential design options and to demonstrate how
the field has progressed, we describe two case studies below. The first example
focused on testing the effects of antismoking and cigarette advertising on youths’
perceptions of a peer who smokes (Pechmann and Ratneshwar 1994). The
experiment was conducted on-site at four middle schools in California. The
second example focused on testing the efficacy of messages from the Parents
Speak Up National Campaign (PSUNC). The entire experiment was conducted
online with a probability based sample of participants from Knowledge Networks’
Case Study Using a Sample of Students from Four Middle Schools In California:
Tobacco Advertising’s Impact On Youth
We begin by highlighting a health communication efficacy study conducted
over 15 years ago as an example of the history of the field. The study design and
process used in this case study are illustrative of the way in which early efficacy
studies were conducted (e.g., in person, on-site, without longitudinal follow-up).
At the time of this study (Pechmann and Ratneshwar, 1994), tobacco
advertising had been banned from radio and television since the early 1970s.
However, a number of legal marketing channels for cigarettes remained,
including billboards, magazines, in-store posters, and promotional activities at
“adult only” establishments such as bars and clubs (Niederdeppe et al., 2003).
This study also predated the 1998 Master Settlement Agreement between five
major tobacco companies and 46 state Attorneys General that established a
number of restrictions on the marketing of cigarettes toward youth.
An accumulation of evidence suggests that youth are highly aware of
tobacco advertising (Arnett and Terhanian, 1998; Fischer et al., 1991) and that
exposure to and liking cigarette ads are related to future smoking initiation and
escalated consumption (Botvin et al., 1993; Feighery et al., 1998; MacFayden,
Hastings, and MacKintosh, 2001). Researchers have speculated that the images
portrayed in cigarette ads help to create and/or perpetuate favorable myths about
smokers and that antismoking ads are needed to counter these myths (Burton et
al. 1989; Davis 1987; DiFranza et al. 1991; Fischer et al. 1991; Gordon 1986).
Youth smoking and intentions to smoke have been associated with believing that
smokers look cool, sexy, exciting, and mature (Barton et al. 1982; Chassin et al.
1981; Collins et al. 1987; Gordon 1986; Grube et al. 1984; McAlister et al. 1984).
However, no causal link has been established between cigarette advertising and
smoking (whether cigarette ads encouraged smoking or whether smokers paid
more attention to cigarette ads; Moschis 1989). In order to examine the causal
effects of smoking-related advertising in a controlled experimental setting,
Pechmann and Ratneshwar (1994) conducted an efficacy study.
Efficacy Study Design
The study participants were 304 seventh grade students from four middle
schools in southern California. Pechmann and Ratneshwar (1994) indicated that
their primary consideration in selecting the schools for the experiment was
ensuring that the participants roughly mirrored the California student population
with regard to gender, socioeconomic status, and race. Participants were
randomly assigned to one of three advertisement conditions (cigarette ads,
antismoking ads, or unrelated to smoking/control ads). The ads were embedded
in a magazine. Magazine ads were used because they are a primary medium
through which youth are exposed to cigarette ads (Davis 1987; Mazis et al. 1992)
and because using billboard advertisements was not feasible for the experiment.
Four pages of stimulus ads were embedded into a 22-page mock-up magazine
with 10 pages of text and 7 pages of filler ads. The authors used a fictitious, non-
gender oriented magazine modeled after “Entertainment Weekly” to control for
prior exposure. The cigarette ads were chosen from 1990-1991 issues of national
magazines, the antismoking ads were provided by federal and state health
departments and by consumer groups, and the unrelated to smoking/control ads
promoted Reebok shoes, American Airlines, and Estee Lauder eye cream. The
cigarette and anti-smoking ads were pre-tested with seventh graders and the
three cigarette ads and the three antismoking ads that students evaluated as
most effective were selected as the experimental stimuli for the study.
Data were collected in 1991. The students completed a survey with
questions on the use of cigarettes and other products, media exposure levels,
and demographics. Each student was instructed to review the magazine for five
minutes, to look at each and every page, and to look at both the articles and ads.
Later, in a separate study, the students were randomly assigned to read
trait information about a peer who was either identified or not identified as a
smoker. The students read the trait information on school computers. The
students were told to spend as much time as they wanted and the computer
recorded the amount of time spent on each screen. The participants’ perceptions
of their peer and their thoughts and inferences about this person were then
Strengths and Limitations