Content uploaded by J. Scott Armstrong
Author content
All content in this area was uploaded by J. Scott Armstrong
Content may be subject to copyright.
Content uploaded by J. Scott Armstrong
Author content
All content in this area was uploaded by J. Scott Armstrong
Content may be subject to copyright.
743 743
International Journal of Advertising, 30(5), pp. 743–767
© 2011 Advertising Association
Published by Warc, www.warc.com
DOI: 10.2501/IJA-30-5-743-767
Evidence-based advertising
An application to persuasion
J. Scott Armstrong
University of Pennsylvania
Complex phenomena such as advertising are difficult to understand. As a result, exten-
sive and repeated testing of diverse alternative reasonable hypotheses is necessary in
order to increase knowledge about advertising. This calls for experimental studies:
laboratory, field, and quasi-experimental studies. Fortunately, much useful empirical
research of this kind has already been conducted on how to create persuasive adver-
tisements. A literature review, conducted over 16 years, summarised knowledge from
687 sources that drew upon more than 3,000 studies (Armstrong 2010). The review led
to the development of 195 principles (condition-action statements) for advertising. We
were unable to find any of these principles in a convenience sample of nine advertising
textbooks and three practitioner handbooks. The advice in these books ignored con-
ditions for the most part. The books also tended to ignore empirical evidence, which
is how we learn about conditions; of the more than 7,200 sources referenced in these
books, only 30 overlapped with the 687 used to develop the principles. By using the
evidence-based principles, practitioners may be able to increase the persuasiveness of
advertisements. Relevant evidence-based papers have been published at the rate of 20
per year from 2000 to 2010. The rate of knowledge development could be increased if
journal editors invited papers with evidence-based research findings and if open peer
review were provided on a continuing basis.
This paper is concerned with only one aspect of advertising – that being
persuasion. I use a broad common-sense definition of persuasive advertis-
ing: it is the attempt to use primarily one-way communication to influence
attitudes and beliefs. And by influence, I mean to either change or main-
tain attitudes and behaviour.
Most of the ideas about how to persuade others are due to the efforts
of thousands of advertisers and others in the persuasion business who
developed and implemented creative approaches. Starting in the early
1900s, advertisers began to conduct experiments to see what worked,
744
INTERNATIONAL JOURNAL OF ADVERTISING, 2011, 30(5)
especially on direct mail advertisements. Academic researchers then
took up the task of assessing what worked. This experimental research
allowed us to determine how advertising is affected by conditions. The
advancement of knowledge in advertising depends on this cumulating
body of research.
I made a key assumption about evidence-based advertising: advertisers
who have access to understandable evidence-based knowledge should be
able to produce more persuasive advertising than they would without this
knowledge. There are two reasons for this. First, creativity can be greatly
enhanced by providing a large variety of persuasive techniques that can
be considered in a given situation. And second, the ability of people to
evaluate ads can be greatly enhanced if they use a structured approach for
evaluating the extent to which ads conform to evidence-based principles
for persuasion.
Why evidence-based advertising has been ignored
If you believe that you can only learn from experience,
how can you learn that you cannot?
Adapted and revised from Einhorn and Hogarth (1978)
There are a number of explanations why the scientific evidence on per-
suasive advertising has been ignored. One is that practitioners do not like
rules. In-depth interviews with 28 account managers, account planners,
and creative people in advertising agencies showed much agreement in
the old saying ‘The only rule: there are no rules’ (Nyilasy & Reid 2009).
The Nyilasy and Reid interviews also showed that many advertisers
have no interest in scientific findings because they believe that their expe-
rience is sufficient. Such feelings are not unique to advertising. They apply
to all complex areas that involve uncertainty, especially when feedback is
poor. This issue has been widely studied since the 1930s (Armstrong
1980). Research since then has added support. Of particular importance,
Tetlock (2005) conducted a 20-year experiment that examined the ability
of 284 economics and political experts to predict the outcomes of various
events in their area of expertise. The experts did no better than people
with little expertise – or than simple rules. Of course, everyone believes
745
EVIDENCE-BASED ADVERTISING
that this finding does not apply to them. Thus, I named it the Seer-sucker
theory – ‘No matter how much evidence exists that seers do not exist, suck-
ers will pay for the existence of seers.’
Other reasons to ignore experimental evidence are the (1) difficulty
of finding useful papers in advertising (my own estimate is that fewer
than 5% of papers published in leading academic journals are useful), (2)
being able to understand the obtuse writing found in most papers, and
(3) the lack of replications for many studies. As a result, it would not be
sensible for practitioners to study the academic literature. It is not sur-
prising then that Helgesen (1994), in his survey of 40 respondents from
the ten largest advertising agencies in Norway, found that they were
largely ignorant of the research literature on advertising. Similar results
were found in the US surveys of 40 advertising practitioners by Nyilasy
and Reid (2009).
Principles as a way to summarise evidence-based
knowledge
In order for findings to be useful, they must be presented as spe-
cific operational condition/action steps. These are referred to here as
principles.
Books by the great advertisers played an important role in describing
the action steps for persuasion. The most important of these was Ogilvy.
Here is one of his recommended action steps: ‘Do not put a period at the
end of a headline’ (Ogilvy 1985, p. 96).
Action steps are not sufficient. It is necessary to identify the conditions
under which they work. For example, Hopkins (1923, p. 233) concluded
that long copy is effective: ‘the more you tell, the more you sell.’ This
works well in most situations – but not all. In another example, it has
been suggested that sellers should not offer a large number of choices;
but research has found that this generalisation is not helpful to advertisers
(Scheibehenne et al. 2010).
Because of the need to consider conditions and because conditions
vary widely, there are many principles – 195 so far. If this seems perplex-
ing, consider an analogy to medicine: what if doctors were to diagnose all
patients by using only ten principles?
746
INTERNATIONAL JOURNAL OF ADVERTISING, 2011, 30(5)
Types of evidence used
What leads to progress? Chamberlin (1890) raised this question, having
noticed that some scientific fields made rapid advances, while others did
not. The key to progress, he concluded, lay in the testing of alternative
reasonable hypotheses. For fields that study complex phenomena about
which there is much uncertainty, experimentation is needed.
For example, agriculture progressed slowly for centuries. Then, in
the UK in the early 1700s, wealthy farmers created a revolution by
experimenting with alternative ways of growing crops (Kealey 1996,
pp. 47–59).
Another example is seen in the Industrial Revolution, which began in
the late 1700s by individuals who tested alternative ways to solve problems
for customers. Much of this work came from a relatively small number of
researchers from Scotland. Adam Smith asked why academics in Scotland
were so important to the Industrial Revolution, while England’s large
number of academicians produced little. His conclusion was that the
academics in England were well supported by the state, so they had little
need to conduct useful research (Kealey 1996, pp. 60–89).
Medicine offers yet another example. Diseases are so complex that doc-
tors were unable to learn from experience about which treatments would
be best for a patient. Advances developed slowly for centuries. However,
after 1940, experimentation became common in medicine and doctors
began to apply findings reported in scientific journals (Gratzer 2006).
Today, evidence-based findings in medicine are easily available on the
Internet (e.g. Cochrane.org).
The testing of multiple reasonable hypotheses is not popular in the
management sciences. Instead, advocacy dominates whereby research-
ers posit their favoured approaches and ignore or even try to suppress
evidence that favours alternative approaches. A publication audit of
over 1,700 empirical papers in six leading marketing journals during
1984–1999, found that 74% used the advocacy approach and 13% the
exploratory approach, while only 13% tested alternative hypotheses. Of
those studies testing alternative hypotheses, only 14% also examined
the effects of conditions (Armstrong et al. 2001). Thus, only about 2%
of the studies in marketing were well designed to advance knowledge
in marketing. As noted above, experimentation is the primary approach
747
EVIDENCE-BASED ADVERTISING
to knowledge development in fields when there is complexity and
uncertainty.
Analyses of non-experimental data are useful for simple problems, espe-
cially if you have much reliable data. For example, substantial amounts of
data are available on professional sports. These have been used success-
fully in recent years by baseball, hockey, football and basketball teams.
When problems are complex, the analysis of non-experimental data
breaks down, even if there are enormous sample sizes. Such non-
experimental analyses are commonly reported in the press with respect
to health and economics. They lead to speculation, re-analyses, and chal-
lenges – and they are often misleading. For example, people who are
concerned about their health seek out the latest treatments. As a result,
non-experimental data show that those using the latest treatments are
healthier than those who are not, even when the treatment has no proven
benefits or may even be potentially harmful, as has been alleged in the
case of female hormone therapy (Avorn 2004, pp. 23–38).
Despite the development of sophisticated methods of statistical analy-
sis and the development of large data banks with advertisements, non-
experimental studies have encountered difficulties in assessing the effects
of conditions. This was shown by some excellent large-scale studies (e.g.
Stewart & Furse 1986).
There are three types of experimentation: laboratory, field, and quasi-
experimental. They each offer advantages and disadvantages. Laboratory
experiments allow for the greatest control of the conditions, but also raise
the issue of the extent to which the findings are realistic. Field experi-
ments add realism, but also the danger that there may have been unob-
served changes in the application of the treatments or in the conditions.
Quasi-experimental studies involve the testing of alternative treatments
in situations where many but not all key conditions have been controlled.
These experiments can be natural or planned. For example, governments
sometimes introduce policy changes in some areas while other areas remain
unaffected (e.g. laws related to gun control). This allows for comparisons
among the different areas. For a general discussion of quasi-experimental
research and a review of prior literature, see Woodside et al. (1997).
The validity of field and laboratory experiments was tested by Locke
(1986). He asked leading researchers in 11 areas of human and organi-
sational behaviour to compare the findings from field experiments with
748
INTERNATIONAL JOURNAL OF ADVERTISING, 2011, 30(5)
those from laboratory experiments. The findings showed close corre-
spondence across the methods. An analysis of 40 studies on sources of
communication found similar findings from field and laboratory studies
(Wilson & Sherrell 1993).
Meta-analyses involve the systematic and objective search for all rel-
evant prior research, followed by use of pre-specified rules for selecting
and quantifying the findings. It may also be sensible to include analyses
of non-experimental data, especially if the data sets are subject to different
biases. Meta-analyses provide the gold standard for knowledge creation
when they focus primarily on experimental evidence.
Knowledge base for advertising
From 1994 up to 2010, I searched for evidence on persuasive advertising.
This involved computer searches, contacting key researchers, posting
requests on email lists, and tracking down papers from references in key
papers.
The search was difficult because the relevant papers are spread over
such areas as law, marketing, mass communications, psychology, and
medicine – and each field uses different terms. Quite often the titles gave
no clues that the papers related to persuasive advertising. Moreover, com-
puter searches typically yield only a small portion of the studies relevant
to a particular topic. For example, in research on forecasting, the compu-
ter searches I used led to only about 1/6 of the relevant papers that were
eventually found (Armstrong & Pagell 2003). Most of the relevant studies
were obtained from citations in other papers, and many were suggested
by key researchers.
In all, I read about 2,400 papers and books that looked promising in
order to find the 687 sources that were used. Many of these were meta-
analyses and reviews that relied on earlier empirical research. By counting
the number of studies in the meta-analyses and by estimating the number
of sources used for traditional reviews, I concluded that the relevant
knowledge base drew upon more than 3,000 studies (Armstrong 2010,
p. 3).
This knowledge was derived primarily from academic research although
Ipsos-ASI, an advertising research company, provided unpublished stud-
ies that they had conducted. As a rough count, 81% of the references in
749
EVIDENCE-BASED ADVERTISING
Persuasive Advertising (Armstrong 2010, hereafter PA) were from academic
journals or conferences, 17% from books, and 2% from mass media,
practitioner-oriented publications, and the Internet. If the analysis is
restricted to papers with experimental evidence, nearly all came from aca-
demic sources. These research papers were scattered across 159 journals.
There was a lack of evidence for many of the principles. To deal with
this, we analysed quasi-experimental data on the print advertisements
from Which Ad Pulled Best (hereafter WAPB) editions five through nine
(Burton & Purvis, 1987–2002). Each edition contains 50 pairs of ads
(except for the ninth edition, which has 40 pairs). These advertisements,
prepared by leading US advertisers, were tested by Gallup & Robinson.
The pairs were similar with respect to product, target market, and media.
Of the 240 pairs of advertisements, 123 were paired against an ad for the
same brand. The ad pairs differed with respect to illustrations, headlines,
colours, and text. In addition, the time periods for the showing of the alter-
native ads differed somewhat.
‘WAPB analyses’ were used for 56 principles. Table 1 presents the ten
most important principles from these analyses (assuming sample sizes of
at least 20 pairs of ads). They are listed by the gain in day-after recall for
ads that followed the given principle. Table 1 shows the average ratio of
recall for ads that properly applied the principle divided by the average
recall for matched ads that did not. (Note that the short summary of the
Table 1: Most important principles from the analysis of print ads (from
WAPB
)
Principle Recall gain (pairs)
Communicate a Unique Selling Proposition (not claimed by other brands) 2.04 (45)
Make the first paragraph relevant 1.74 (46)
Include brand and company names (double-branding) 1.71 (21)
Provide news, but only if it is real 1.64 (20)
Use positive arguments 1.60 (24)
Illustrations should support the basic message 1.54 (43)
Use descriptive headlines for high-involvement products 1.52 (24)
Balance the layout 1.50 (36)
Include the brand name in the headline 1.49 (24)
For high-involvement products, the reasons should be strong 1.48 (25)
750
INTERNATIONAL JOURNAL OF ADVERTISING, 2011, 30(5)
principles does not typically include the conditions. The full descriptions
are provided in PA.)
To assess the validity of quasi-experimental data, the findings with
respect to direction of effects were compared with findings of other
types of experimental evidence. The primary concerns were 1) the
WAPB data used day-after-recall, whereas the other approaches used
many different criteria of effectiveness, and 2) the WAPB samples
were small (an average of 31 pairs with a range from 6 to 118).
Despite the problems, the findings from the quasi-experimental analy-
ses were in agreement with respect to the direction of effects of all
seven principles for which there were meta-analyses, all 26 principles
for which there were lab experiments, and all seven principles for
which there were field experiments. In contrast, non-experimental analy-
ses disagreed on direction of effects with the quasi-experimental findings
for eight of the 24 principles that allowed for comparisons (Armstrong &
Patnaik 2009), thus emphasising the need for caution when using findings
from non-experimental data.
Meta-analyses proved to be extremely important for the develop-
ment of the persuasion principles. Daniel O’Keefe authored 11 of the 33
meta-analyses.
To help ensure the summaries were accurate, I read all of the sources
that were cited.
1
In addition, I asked the experts who were cited to check
whether the summaries of their findings were correct. The vast majority
of those who could be located replied, often with important corrections.
For many of the principles, there were a number of researchers who com-
mented. Reviewers helped to make the principles accurate and editors
helped to make the explanations easy to understand.
The intent was to summarise all evidence relevant to persuasion in
advertising. Persuasive Advertising provides advice on what types of evi-
dence are most important. The various types of experimental evidence
were always in agreement with one another with respect to the directional
effects of principles. This is no accident. My intent was to include only
principles for which the experimental evidence was, for the most part,
consistent. I omitted many potential principles due to a lack of consist-
1
It is common for scientists to cite studies that they have not read and to cite them incorrectly. See Wright
and Armstrong (2008).
751
EVIDENCE-BASED ADVERTISING
ency. For example, prospect theory was dropped in view of large-scale
meta-analyses with conflicting results.
Application of evidence-based principles
There are two ways in which the persuasion principles can be used. One is
to stimulate creativity and the other is to evaluate and improve ads.
Stimulating creativity
The principles offer a structured checklist for advertisers to use as they
create ads. This would increase the amount of time spent on creativity,
but in most practical situations, this would represent only a small fraction
of the advertising costs.
The checklist should be able to enhance the creativity of any user. It is
expected that practice should improve one’s ability to use the principles.
In addition, creativity can be enhanced by obtaining recommendations
from a number of individuals who independently apply the checklist.
Contrary to common opinions, such structured techniques have been
shown to be vastly superior to unstructured techniques, such as group
meetings, when it comes to creativity.
Given that the principles have not previously been available, their use
should produce advertising that differs substantially from what is currently
being done.
Evaluation and improvement of advertisements
The principles are also useful for evaluating and improving advertise-
ments. Evaluators need to understand the principles and evaluate the
extent to which an advertisement adheres to the principles.
The AdPrin Audit software, (available on AdPrin.com), is essentially a
principles-orientated checklist to guide the evaluation process. Checklists
have been found to yield enormous improvements in decision-making. In
life-threatening situations, such as flying an airplane, a pilot who did not
use a checklist would be thought to be foolish. An experimental study
of eight hospitals in eight cities around the world found that the use of
752
INTERNATIONAL JOURNAL OF ADVERTISING, 2011, 30(5)
a 19-item checklist reduced deaths in the month after an operation from
1.5% to 0.8% (Haynes et al. 2009).
The evaluation phase calls for people who are good at logical reason-
ing. Training and practice are expected to lead to gains in the ability to
judge whether an ad properly applies the principles. In research that we
are currently conducting, we find that people can use a two-hour self-
training module, and then rate how well ads conform to the principles. For
example, they average about half an hour to rate a print ad. Their ratings
lead to substantial improvements in assessing the effectiveness of ads, in
comparison with their unaided judgments.
Examples of evidence-based principles
Here are three examples of principles that followed from the above-
mentioned procedures. The major conditions are often stated in the
principles , but in general, the conditions are mentioned in the text. These
are abridged versions from the PA book. The numbers in the parentheses
correspond to those in the book.
Do not mix rational and emotional appeals (3.1.1)
While many advertising experts have suggested that an emotional compo-
nent would strengthen almost any ad, the evidence suggests the opposite.
Rational and emotional appeals are likely to interfere with each other.
Evidence on the effects of mixing rational and emotional appeals
In an experiment involving donations to Save the Children, a narrative
description of a victim’s plight led to higher donations than when the
description also included statistics about how the donations would help.
Apparently, the latter information inhibited the emotional effect and
led people to think about how their efforts would help; it also led them
to determine that their contributions would be negligible (Small et al.
2006).
We analysed 50 pairs of WAPB print ads in which one ad had either a
rational or emotional appeal while the other ad used both rational and
emotional appeals. Recall for ads that did not mix the appeals was 1.24
times better than the ads that mixed them.
753
EVIDENCE-BASED ADVERTISING
An analysis of 80 automobile ads found that recall of those using either
a rational or emotional appeal yielded better recall than those using both
types of appeals (Mehta & Purvis 2006).
Eye-tracking studies of 190 subjects as they watched Dutch TV com-
mercials found that people were overwhelmed when both emotion and
information were present, and thus they were more likely to fast-forward
through such ads (Elpers et al. 2003).
TV commercials containing ‘a balance of rational and emotional appeals’
were lower on comprehension and much below average with respect to
persuasion in comparison with commercials that did not contain such a
balance (Stewart & Furse 1986, p. 154, item 139).
If resistance is expected, use indirect conclusions when the
arguments are strong and obvious (5.9.2)
The direct approach may cause people to feel a loss of freedom when they
are not already favourable to the product, especially for high-involvement
products.
There are a variety of indirect approaches. One is simply to present the
arguments and then let the customer decide what to do. For example, an
advertisement by Saab presented performance attributes for a Saab and a
BMW. It then invited customers to ‘compare the value you will get’, fol-
lowed by ‘and then you make the decision’.
Another indirect approach is to allow the reader or viewer to observe
others arguing each side of an issue. This should reduce reader or view-
er’s predilection to counter-arguing, because someone else is doing the
counter-arguing. This can be done in advertising by showing someone
who is being persuaded on-screen by another person.
The indirect approach is advisable when the source is regarded as
biased and when the message is directed at an intelligent audience.
Evidence on effects of indirect conclusions when resistance is expected
A review of research, including over 40 studies, found that attempts to
restrict people’s freedom by providing direct conclusions often led them
to reassert their beliefs (Clee & Wicklund 1980).
Other research reviews suggest that indirect conclusions are most
persuasive when the communicator is perceived as biased, presumably
754
INTERNATIONAL JOURNAL OF ADVERTISING, 2011, 30(5)
because customers would otherwise be more likely to counter-argue.
Indirect conclusions are also more appropriate when the members of the
target market are intelligent because they would be more likely to under-
stand the conclusions on their own, and self-persuasion is convincing
(Chebat et al. 2001). Finally, there is little need for direct conclusions when
exposure to the campaign will be frequent.
In a lab experiment, booklets were shown to 211 subjects. They con-
tained ads with either an open-ended conclusion (e.g. ‘Now that you know
the difference, decide for yourself which disposable razor you should buy’)
or a closed-ended conclusion (‘Now that you know the difference, shave
with Edge, the disposable razor that is best for you’). Purchase intentions
were higher for the open-ended ads. Similar results were obtained with an
ad for compact disk players (Ahearne et al. 2000).
In a small-scale lab experiment, 24 Japanese subjects saw online ads for
15 products (e.g. movies). Near the end of each ad, the subjects saw one
of two scenes: a life-like agent talking to and looking at the viewer or two
life-like agents looking at each other and conversing. In each case, the
persuader agent used the same words, e.g. ‘You have to watch this movie;
it’s very interesting.’ Purchase intentions for the indirect approach – the
overheard conversation – were 31% higher (Suzuki & Yamada 2004).
In another lab experiment, in which 261 students viewed cellular phone
ads, indirect conclusions were relatively more effective than direct con-
clusions when there were strong arguments for the brand than when the
arguments were weak (Martin et al. 2003/4).
Print ads for CD players were shown to 192 subjects. The ads contained
either explicit or implicit conclusions. Highly involved subjects were more
likely to infer omitted conclusions, and when they did, they reported more
favourable brand attitudes (Kardes 1988).
Do not invite customers to evaluate their satisfaction while
using a product. (5.11.3)
A British Airways advertising campaign invited people to try its business
class. Consumers who were not satisfied would receive free coach tickets
for another trip. Was that a good idea?
When consumers expect to report about their level of satisfaction with
a product or service, they adopt a critical attitude and search for
755
EVIDENCE-BASED ADVERTISING
things that are wrong. This leads them to have a less enjoyable experi-
ence. Their complaints may also reduce satisfaction for those providing
the services.
This principle is widely violated by hotels, automobile dealerships,
telephone companies, stock brokers, and other firms that routinely use
preannounced satisfaction surveys. Universities have long used them in an
attempt to assess student satisfaction; unfortunately, they reduce student
and teacher satisfaction, harm learning, and increase administrative costs
(Armstrong 2004).
Evidence on the effects of preannounced satisfaction surveys
Experiments were conducted with a computer company, electric utility,
supermarket, drug store, magazine, and electronic equipment company.
Some customers, randomly assigned, were told that they would be asked
later about their satisfaction with the service, while others were not
informed about the satisfaction survey. In the long-term follow-up satis-
faction survey, those in the pre-announced-survey group were much less
satisfied than those who had not expected to receive a satisfaction survey.
People in the pre-announced group were looking for reasons to be dissatis-
fied – and they found them (Ofir & Simonson 2001).
A role-playing experiment of a banking service was used to evaluate
responses to a negative situation (rude behaviour by a bank teller). The
subjects in a preannounced survey group gave a substantially poorer rat-
ing of service quality than did those who were not told there would be a
satisfaction survey. They also reported themselves as being more likely to
switch banks. In addition, they were less likely to provide details about
their complaint because they had already rated their dissatisfaction on the
survey – thus, the bank would not have learned why they were dissatisfied
(Lane & Keaveney 2005).
Usefulness of evidence-based principles
The value of evidence-based principles would depend not only on their
validity, but also on the extent to which they lead to advertising proce-
dures that differ from common sense and from current practice.
Consider the results from a convenience sample of people who took
the ‘Test your advertising IQ’ on AdPrin.com. Guessing would lead to a
756
INTERNATIONAL JOURNAL OF ADVERTISING, 2011, 30(5)
score of about 8 out of 20. The median score for the 110 people who took
this test online in late October and early November 2010 was 8. Thus, the
principles are not based on common sense.
To test whether the principles are being learned in other courses
or reading, a 67-item true-false test was administered to 18 Wharton
undergraduates on their first session in an upper-level undergraduate
advertising class at the Wharton School in January 2011. This was an
elective course, so the students had an interest in advertising. As this
was a higher-level course, most had taken relevant courses such as con-
sumer behaviour or communications. In addition some had read relevant
pop-management books, and a few had relevant work experience. The
test was one that had been prepared for the final exam in this course,
so the goal was to include as many of the principles as possible via true-
false questions. The students were correct on only 53.6% of the items.
The scores for those with a more extensive background – based on prior
courses, experience, and reading – were marginally lower than those
with less relevant backgrounds.
Diffusion of knowledge about evidence-based
principles
In some fields such as in engineering and the natural sciences, the basic
principles are accessible in textbooks. To see whether some principles
have been passed along by advertising textbooks, I, along with two
research assistants, examined a convenience sample of nine advertising
texts. The number of references was counted, and then we coded which
of these were research papers (primarily those published in academic
journals). In addition, we examined the overlap between the sources cited
in the textbooks and the 687 sources in PA. The findings are provided in
Table 2.
As noted in the last column, the books drew upon different sources than
does PA. Few of the 687 sources cited in PA were cited, most of these
being in the books by Rossiter. Including Rossiter’s books, only 30 (0.4%)
of the 7,236 references matched those from PA. Excluding Rossiter, there
were only nine citations to the evidence in PA.
We also went through each textbook and handbook page by page to
count the number of persuasion principles that were presented. We found
757
EVIDENCE-BASED ADVERTISING
no evidence-based principles in these books.
2
One key issue was that the
authors typically ignored conditions when they provided advice. This is
not to say that there were no evidence-based principles in these advertis-
ing books, only that we were unable to find any. We also contacted the
authors of these books to see if they could provide any examples from
their books that we might have overlooked. No such principles were
reported to us.
Note that Rossiter and Bellman (2005) differs substantially from other
textbooks in its number of references to research sources. There were 13
sources that overlapped with PA. However, these 13 references were not
used to present principles. For example, one reference was summarised
as ‘glasses added to the impressions of intelligence, industriousness and
honesty’ (p. 412) without any condition/action statement.
The above analyses of textbooks were limited to persuasion. To be
sure, there are other areas of advertising, such as media expenditures.
However, the general lack of use of academic research in textbooks and
handbooks poses a problem. Is it helpful to simply provide opinions?
2
In a related study, based on a sample of leading texts on marketing principles, Armstrong and Schultz
(1993) could find no evidence-based principles for marketing.
Table 2: References to evidence-based persuasion in leading advertising textbooks
Total % research # in
Persuasive Advertising
Textbooks
Rossiter & Bellman (2005) 658 62 13
Shimp (2000; 5th edn.) 1,133 20 0
Belch & Belch (2009; 8th edn.) 1,271 19 0
Clow & Baack (2010; 4th edn.) 473 15 3
Rossiter & Percy (1997; 2nd edn.) 789 16 8
O’Guinn et al. (2003; 3rd edn.) 698 6 0
Duncan (2005; 2nd edn.) 516 6 0
Wells
et al.
(2006; 7th edn.) 345 4 0
Lane
et al.
(2011; 18th edn.)
681 0.3 0
Totals 6,564 24
Handbooks
Brierley (2002; 2nd edn.) 155 8 0
Dupont (1990) 517 18 6
Lewis & Nelson (1999) 0 0 0
Totals 672 6
758
INTERNATIONAL JOURNAL OF ADVERTISING, 2011, 30(5)
My original training was in engineering, and I recall no textbooks that
provided opinions about such things as how to build safe bridges and
highways.
An earlier study on evidence-based findings in communication text-
books provided similar results (Allen & Preiss 1998). That study coded
21 textbooks; two of these were by well-known experts on meta-analysis,
so they were excluded from the following analysis. The objective was to
assess whether the findings in the books were consistent with the evi-
dence, as determined from earlier meta-analyses. Eleven widely studied
areas were included (e.g. fear appeals, distraction). None of the text-
books disagreed with the notion that evidence is persuasive. However,
for the remaining ten areas there were 13 cases where the textbooks
agreed with the evidence, 15 where they contradicted the evidence, and
13 where their position was not clear. (They also ignored many of the
topics.) In short, the textbook writers paid little attention to the prior
experimental evidence in presenting generalisations. There was no use
of principles.
Overcoming barriers to evidence-based advertising
To encourage the use of evidence-based principles, it is important
to make them easily available to advertisers and advertising agen-
cies, when they need them. In medicine for example, sites such as
Cochrane.org allow patients as well as doctors – and researchers – to
access the latest knowledge on the best ways to treat diseases. In
addition, the principles should be understandable. Finally, they should
be actionable. These criteria were used for the design of AdPrin.com.
Apparently, AdPrin.com is meeting a need for advertisers, agencies,
researchers, and students. By mid-2011, visits were running at the rate
of 40,000 per month.
Researchers can publish evidence-based findings on AdPrin.com so that
others can use them – or test them – immediately. This would also allow
researchers to stake a claim for their discovery prior to journal publica-
tion, and to obtain feedback from others. To aid in this, a section called
‘Commentaries on evidence-based advertising’ has been established on
the site.
759
EVIDENCE-BASED ADVERTISING
Some agencies will try the principles in the hope of gaining a competi-
tive advantage. Others might contribute to the development of principles
as a way of advancing the field, even if they share only a portion of the
gains. Furthermore, advertisers might ask their agencies to implement
these principles or to explain why they did not do so. The AdPrin Audit
software will, after some practice, enable them to rate the effectiveness
of an advertisement in about an hour per coder – and, most important, to
suggest how to improve the ad.
Suggestions for further research
I examined the rate of progress in developing useful evidence-based
findings on persuasive advertising. This was assessed by examining the
number of papers that contributed to the development of principles
over the past decade. Of the sources in PA that were published between
2000 and 2010, I identified those that contained evidence related to the
principles . This yielded 193 references, or about 19 per year. Given that
there are thousands of academicians who are publishing in fields related
to persuasion, this productivity seems low.
To examine whether researchers value evidence-based advertising, I
examined the 75 ‘most-cited advertising works’ from 1982 through 1995
from Pasadeos et al. (1998). Only 15% of these papers were used in formulat-
ing the 195 advertising principles.
I looked at the authors (or teams)
who were most cited for support
on the principles (defined as those
cited in the development of the
principles on pages 26 through
277 of PA) to see whether their
papers on principles were on that
top papers’ list. Table 3 lists those
who contributed to at least six
principles. Of these, only Stewart
et al. (1986) were listed among
the authors of the 75 most-cited
advertising papers.
Table 3: List of leading researchers
who contributed to the development
of principles
Authors
Number of
principles
Stewart, D.W., Furse, D.H. & Koslow, S. 28
Walker, D. 22
O’Keefe, D. 11
Stanton, J.L. & Burke, J. 10
Cialdini, R. 9
Pieters, R. & Wedel, M. 8
Jacoby, J. & Hoyer, W. 7
Woodside, A. 6
760
INTERNATIONAL JOURNAL OF ADVERTISING, 2011, 30(5)
Overcoming barriers to publication of evidence-based
papers in advertising
One barrier to journal publication of evidence-based advertising is that the
findings often conflict with commonly held views. Journals typically reject
such papers. This occurs because the current peer review system allows
reviewers to block, or at least delay, papers containing findings with which
they disagree. For example, in Mahoney’s (1977) experiment, 75 psychol-
ogists thought that they were providing reviews of an actual submission;
half of the reviewers received a version of the paper that supported exist-
ing beliefs, while the other half received one that refuted these beliefs.
The reviewers who received the disconfirming version were much more
likely to reject the paper, explaining that the methodology was flawed. As
it happened, the methodology was the same for both versions of this ficti-
tious submission. For further evidence, see Armstrong (1996 and 1997)
and Benda and Engels (2011).
Another explanation is that the use of statistical significance has led
researchers to ignore practical significance. Papers with ‘null results’ are
rejected, even if important, such as when a well-regarded treatment is
shown to be useless (Hubbard & Armstrong 1992), and those with statisti-
cal significance (almost any finding with large sample sizes) are favoured
even if they lack practical significance.
Directed research
Invited papers provide the easiest and most common way for journal
editors to ‘direct’ research on advertising principles. For example, the
Journal of Economic Perspectives invites researchers to publish papers on
specified topics, and these authors seek their own peer review. Papers
could be invited for important principles that lack strong evidence. Papers
could also be invited for replications of important evidence-based papers.
Researchers could then focus on the topic without fear of being rejected
should their findings challenge existing beliefs. Reviewers would be
asked how to improve each paper.
The cost of invited papers is low because all invited papers are
accepted, whereas under the traditional approach, about seven papers
are reviewed for every one accepted. In addition, invited papers are
761
EVIDENCE-BASED ADVERTISING
expected to have higher impact. In a study involving research on fore-
casting, papers receiving special treatment (primarily invited) were
judged as 20 times more impactful. Impact was based on two factors: the
citation rate and whether findings were useful in the development of
forecasting principles.
Advertisers could help to direct research by doing research on principles
and sharing the findings, providing funding for research on principles, or
providing data for testing principles.
One idea is to direct researchers to areas that currently lack experimen-
tal evidence. Many of the persuasion principles require more evidence.
Of the 195 persuasion principles, three rested on common sense and
thus required no testing. Based on my codings for each principle, ample
experimental evidence was available for only 22% of the principles that
required testing. A summary of the amount of evidence for the principles
is provided in Table 4. (My codings for the principles are provided on
AdPrin.com.)
To determine which principles are in need of research, I focused on
the 58 principles that were based on, at most, a single experiment; of
these, I looked for those that seemed to be violated often. (A listing
of these areas is provided on the Research repository on AdPrin.com.)
Some of the more important principles in need of study are listed in
Table 5 (again using the short-form of the principles with conditions
omitted).
Table 4: Strength of evidence for the principles
Number % Evidence (listed by strength of evidence)
42 22 Much experimental evidence
33 17 Some experimental evidence plus non-experimental evidence
59 31 Some experimental evidence
18 9.5 One experiment plus non-experimental evidence
22 11.5 One experiment
8 4 Non-experimental evidence
6 3 Received wisdom
4 2 Speculative (no evidence)
192 100
762
INTERNATIONAL JOURNAL OF ADVERTISING, 2011, 30(5)
Alternatives to journal publication
Books can serve a useful function by reporting new findings. However,
there are long time lags associated with publishing in books.
The Internet offers a faster alternative. This should be especially
appealing to researchers who have important findings and who prefer to
seek their own peer review. Those with new discoveries on advertising
principles can stake their claim by publishing on the Internet. AdPrin.com
welcomes such papers.
Overcoming barriers to practice
The evidence-based principles differ markedly from practitioners’ current
beliefs and practices. As a result, the use of these evidence-based princi-
ples should lead to more effective advertising. But how could this occur
given a presumption that ‘the best rules are no rules’?
In my review of the evidence, I found rules that were not useful. All
actions were dependent upon the conditions. So I am proposing a third
way, the use of principles. Given that there are many agencies and adver-
tisers, I expect that some of them will use the principles. Why, for exam-
ple, would one insist on putting a period at the end of a headline for a
high-involvement product with good arguments?
Table 5: Ten principles in need of experimental research (principle # in
Persuasive
Advertising
)
Principle # in
PA
Provoke customers only when it attracts attention to a selling point (3.6.1)
Focus on benefits or features rather than choices (5.2.2)
When the target market has an opposing viewpoint, consider using a story (5.3.1)
Include brand and company names (double-branding) (5.5.2)
Alert the target market early and prominently (8.1.1)
Keep the headline short for low-involvement goods only (9.1.4)
Use clear and readable captions for pictures (9.2.3)
Repeat the main message at the end of the ad (9.3.3)
Squeeze inter-letter spacing gently (9.4.6)
Avoid large pictures in informative ads (9.6.2)
763
EVIDENCE-BASED ADVERTISING
Another force for the use of evidence-based principles is that clients
can question why certain seemingly relevant principles have been violated
in ads proposed by their agency. They will have the same power that we
now have as medical patients. For example, by reviewing the evidence, I
have declined the advice of my doctor on many of his recommendations
(such as his suggestion on Lipitor). I am better qualified to draw conclu-
sions from experimental evidence than most doctors. Similarly, all large
organisations will have people who are capable of understanding and using
experimental evidence on advertising principles.
Continuing with the example of medicine, the adoption of the principles
has been aided there by the fact that doctors who fail to use evidence-based
treatments face possible lawsuits when patients suffer undesirable outcomes.
The transition to evidence-based advertising will be slow. However,
it will occur because there is money to be made by improving the effec-
tiveness of advertising. I suspect that implementation will be driven by
clients who are concerned whether their advertising investments are
profitable, and who will ask advertising agencies why seemingly relevant
principles are being violated or ignored. I expect that these client needs
will eventually be met by new advertising research firms.
Conclusions
Over the past century, experts and researchers have produced a valuable
and remarkable storehouse of knowledge on how to persuade via advertis-
ing. Given the complexity of advertising, experimentation provided the
key to the formulation of the principles; one could not learn about the
effects of conditions from their experience. This knowledge has now been
converted to operational principles that are, in effect, new to the field.
They conflict with common practice. The principles are freely available.
This should lead to substantial gains in the effectiveness of advertising.
Evidence-based advertising is in its infancy. I expect that there will
continue to be improvements in the current principles and that new prin-
ciples will be added. Directed research, spurred on by invited papers and
by Internet publication, would speed advances. The Internet will lead to
faster dissemination of new findings. Open peer review on the Internet
will help to ensure that the findings are useful.
764
INTERNATIONAL JOURNAL OF ADVERTISING, 2011, 30(5)
Acknowledgements
Kay A. Armstrong, Kesten C. Green, Gerry Lukeman, Sandeep Patnaik,
Daniel O’Keefe, Don E. Schultz, Arch Woodside, Dave Walker and
Malcolm Wright provided peer review. Further peer review was provided
by the editor, Charles Raymond Taylor, and by the three commentators.
Ranti Odujinrin, Elliot Tusk and Martin Wong coded data on the use
of principles in textbooks. Kanica Allagh, Alexandra House, Kathy Lin,
Kelsey Matevish, Ranti Odujinrin and Elliot Tusk edited the paper. I also
thank the many authors who responded to our survey as to whether my
summaries of their research studies were correct.
References
Ahearne, M., Gruen, T. & Saxton, M.K. (2000) When the product is complex, does
the advertisement’s conclusion matter? Journal of Business Research, 48, pp. 55–62.
Allen, M. & Preiss, R.W. (1998) Evaluating the advice offered by the tool users. In
M. Allen & R.W. Preiss (eds) Persuasion: Advances through Meta-Analysis. Cresskill,
NJ: Hampton Press, pp. 243–255.
Armstrong, J.S. (1980) The seer-sucker theory: the value of experts in forecasting.
Technology Review, 83 (June/July), pp. 18–24.
Armstrong, J.S. (1996) Management folklore and management science: on portfolio
planning, escalation bias and such. Interfaces, 26(4), pp. 28–42 (commentaries follow).
Armstrong, J.S.(1997) Peer review for journals: evidence on quality control, fairness,
and innovation. Science and Engineering Ethics, 3, pp. 63–84.
Armstrong, J.S. (2004) How to improve service quality and satisfaction. Online
at: http://marketing.wharton.upenn.edu/people/faculty/armstrong/ELMAR/
HowToImproveServiceQuality.pdf (accessed 10 October 2011).
Armstrong, J.S. (2010) Persuasive Advertising. London: Palgrave Macmillan.
Armstrong, J.S. & Green, K.C. (2011) AdPrin.com: Advertising Principles –
Evidence-Based Knowledge on Persuasion through Advertising. Online at: http://
advertisingprinciples.com
Armstrong, J.S. & Pagell, R. (2003) Reaping benefits from management research:
lessons from the forecasting principles project. Interfaces, 33(6), pp. 89–111.
Armstrong, J.S. & Patnaik, S. (2009) Using quasi-experimental data to develop
empirical generalizations for persuasive advertising. Journal of Advertising Research,
49(2), pp. 170–175.
Armstrong, J.S. & Schultz, R. (1993) Principles involving marketing policies: an
empirical assessment. Marketing Letters, 4, pp. 253–265.
Armstrong, J.S., Brodie, R. & Parsons, A. (2001) Hypotheses in marketing science:
literature review and publication audit. Marketing Letters, 12, pp. 171–187.
Avorn, J. (2004) Powerful Medicines. New York: Alfred A. Knopf.
765
EVIDENCE-BASED ADVERTISING
Belch, G. & Belch, M. (2009) Advertising and Promotion: An Integrated Marketing
Communications Perspective. New York: McGraw-Hill.
Benda, W.G.G. & Engels, T.C.E. (2011) The predictive validity of peer review: a
selective review of the judgmental forecasting qualities of peers, and implications
for innovation in science. International Journal of Forecasting, 27, pp. 166–182.
Brierley, S. (2002) The Advertising Handbook. New York: Routledge.
Burton, P.W. & Purvis, S.C. (edition 5 in 1987 through edition 9 in 2002) Which Ad
Pulled Best? Chicago, Illinois: Crain Books.
Chamberlin, T.C. (1890, 1965) The method of multiple working hypotheses. Science,
148, pp. 754–759. (Reprint of a 1890 paper.)
Chebat, J-C, Charlebois, M. & Gélinas-Chebat, C. (2001) What makes open vs.
closed conclusion advertisement more persuasive? The moderating role of prior
knowledge and involvement. Journal of Business Research, 53, pp. 93–102.
Clee, M.A. & Wicklund, R.A. (1980) Consumer behavior and psychological reactance.
Journal of Consumer Research, 6, pp. 389–405.
Clow, K.E. & Baack, D. (2010) Integrated Advertising, Promotion, and Marketing
Communications. 4th edition. Upper Saddle River, New Jersey: Prentice Hall.
Duncan, T. (2004) Principles of Advertising & IMC. New York: McGraw-Hill/Irwin.
Dupont, L. (1990) 1001 Advertising Tips. Quebec, Canada: White Rock Publishing Inc.
Einhorn, H.J. & Hogarth, R.M. (1978) Confidence in judgment: persistence of the
illusion of validity. Psychological Review, 85, pp. 395–416
Gratzer, D. (2006) The Cure. New York: Encounter Books.
Haynes, A.B., Weiser, T.G., Berry, W.R. et al. (2009) A surgical checklist to reduce
morbidity and mortality in a global population. New England Journal of Medicine,
360 (January), pp. 491–499.
Helgesen, T. (1994) Advertising awards and advertising agency performance criteria.
Journal of Advertising Research, 34 (July/August), pp. 43–53.
Hopkins, C.C. (1923) Scientific Advertising. Bramhall, U.K.: Cool Publications Limited.
Hubbard, R. & Armstrong, J.S. (1992) Are null results becoming an endangered
species in marketing? Marketing Letters, 3, pp. 127–136.
Kardes, F.R. (1988) Spontaneous inference processes in advertising: the effects of
conclusion omission and involvement on persuasion. Journal of Consumer Research,
15 (September), pp. 225–233.
Kealey, T. (1996) The Economic Laws of Scientific Research. London: Macmillan.
Lane, V.R. & Keaveney, S.M. (2005) The negative effects of expecting to evaluate:
reexamination and extension in the context of service failure. Psychology and
Marketing, 22(11), pp. 857–885.
Lane, W.R., King, K.W. & Reichert, T. (2011) Advertising Procedure. 18th edition. New
York: Prentice Hall.
Lewis, H.G. & Nelson, C. (1999) Handbook of Advertising. Lincolnwood, Illinois:
NTC Business Books.
Locke, E.A. (1986) Generalizing from Laboratory to Field Settings. Lexington, MA:
Lexington Books.
Mahoney, M. (1977) Publication prejudices: an experimental study of confirmatory
bias in the peer review system. Cognitive Therapy and Research, 1, pp. 161–175.
766
INTERNATIONAL JOURNAL OF ADVERTISING, 2011, 30(5)
Martin, B.A.S., Lang, B. & Wong, S. (2003/4) Conclusion explicitness in advertising.
Journal of Advertising, 32(4), pp. 57–65.
Mehta, A. & Purvis, S.C. (2006) Reconsidering recall and emotion in advertising.
Journal of Advertising Research, 46(1), pp. 49–56.
Nyilasy, G. & Reid, L.N. (2009) Agency practitioners’ meta-theories of advertising.
International Journal of Advertising, 28(4), pp. 639–668.
Ofir, C. & Simonson, I. (2001) In search of negative customer feedback: the effect of
expecting to evaluate on satisfaction evaluations. Journal of Marketing Research, 38,
pp. 170–182.
Ogilvy, D. (1985) Ogilvy on Advertising. New York: Vintage Books.
O’Guinn, T.C., Allen, C.T. & Semenik, R.J. (2003) Advertising and Integrated Brand
Promotion. Mason, Ohio: South-western Publishing.
O’Keefe, D.J. (2000) Guilt and social influence, in M.E. Roloff (ed.), Communication
Yearbook. Thousand Oaks, CA: Sage Publications, vol. 23, pp. 67–101.
Pasadeos, Y., Phelps, J. & Kim, B-H. (1998) Disciplinary impact of advertising
scholars: temporal comparisons of influential authors, works and research
networks. Journal of Advertising, 27(4), pp. 53–70.
Rossiter, J.R. & Bellman, S. (2005) Marketing Communications: Theory and Applications.
French Forest, NSW, Australia: Pearson Education.
Rossiter, J.R. & Percy, L. (1997) Advertising Communications and Promotion Management.
New York: McGraw Hill.
Scheibehenne, B. Greifeneder, R. & Todd, P.M. (2010) Can there ever be too many
options? A meta-analytic review of choice overload. Journal of Consumer Research,
37, pp. 409–425.
Shimp, T.A. (2000) Advertising Promotion: Supplemental Aspects of Integrated Marketing
Communications. Orlando, FL: Harcourt College Publishers.
Small, D.A., Loewenstein, G. & Slovic, P. (2006) Sympathy and callousness: the
impact of deliberative thought on donations to identifiable and statistical victims.
Organizational Behavior and Human Decision Processes, 102, pp. 143–153.
Stewart, D.W. & Furse, D.H. (1986) Effective Television Advertising: A Study of 1000
Commercials. Lexington, MA: Lexington Books.
Suzuki, S. & Yamada, S. (2004) Persuasion through overheard communication by life-
like agents. IEEE/WIC/ACM International Conference on Intelligent Agent Technology.
Online at: http://www.computer.org/portal/web/csdl/doi/10.1109/IAT.2004.1342948
(accessed 10 October 2011).
Tetlock, P.E. (2005) Expert Political Judgment. Princeton: Princeton University Press.
Walker, D. (personal correspondence, 2006–2009, using data analyzed by Ipsos ASI).
Wells, W., Moriarty, S. & Burnett, J. (2006) Advertising Principles & Practice. Upper
Saddle River, New Jersey: Pearson Education.
Wilson, E.J. & Sherrell, D.J. (1993) Source effects in communication and persuasion:
a meta-analysis of effect size. Journal of the Academy of Marketing Science, 21(2),
pp. 101–112.
Woltman-Elpers, J., Wedel, M. & Pieters, R.G.M. (2003) Why do consumers stop viewing
television commercials? Two experiments on the influence of moment-to-moment
entertainment and information value. Journal of Marketing Research, 40, pp. 437–453.
767
EVIDENCE-BASED ADVERTISING
Woodside, A.G., Trappey, R.J. III & MacDonald, R. (1997) Measuring linkage-
advertising effects on customer behavior and net revenue: using quasi-
experiments of advertising treatments with novice and experienced product users.
Canadian Journal of Administrative Sciences, 14, pp. 214–228.
Wright, M. & Armstrong, J.S. (2008) Verification of citations: fawlty towers of
knowledge. Interfaces, 38(2), pp. 125–139.
About the author
J. Scott Armstrong (PhD from MIT in 1968) is Professor of Marketing at
the Wharton School, University of Pennsylvania, where he has been teach-
ing advertising since 1968. He has been a Visiting Professor in Argentina,
Australia, New Zealand, South Africa, Spain, Sweden, Switzerland,
Thailand, and the UK. He is the founder and director of advertising-
principles.com, which received the 2004 MERLOT Award for the ‘Best
Internet Site in Business Education’. In 2000, he received the Society for
Marketing Advances Distinguished Scholar Award. He is the most fre-
quently cited professor in the history of the Wharton School’s Marketing
Department. His book, Persuasive Advertising, was published in 2010.
Address correspondence to: J. Scott Armstrong, Wharton School, JMHH
747, University of Pennsylvania, Philadelphia, PA 19104, United States.
Email: armstrong@wharton.upenn.edu
Copyright of International Journal of Advertising is the property of Warc LTD and its content may not be
copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written
permission. However, users may print, download, or email articles for individual use.
