Journal of Health Economics 24 (2005) 1030–1033
Extraordinary claims require extraordinary evidence
Donald W. Lighta,b,∗, Rebecca N. Warburtonc
aUniversity of Medicine and Dentistry of New Jersey, Stratford, NJ 08084, USA
bCenter for Bioethics, University of Pennsylvania, USA
cMichael Smith Foundation for Health Research Scholar, School of Public Administration,
University of Victoria, Vic., BC, Canada
JEL classification: L65; O31
Keywords: Pharmaceutical industry; R&D cost; Internal validity; External validity; Bias [1, 253]
1. Extraordinary claims require extraordinary evidence
At the beginning of 2003, the Journal of Health Economics published a paper of great
importance in public policy by DiMasi, Hansen, and Grabowski (referred to hereafter as
“DHG 2003”). The paper is based primarily on confidential, proprietary data supplied by
2004a,b). This commentary is intended to invite discussion among health economists and
other researchers about the quality of data and sampling used in estimating the costs of
DHG 2003 estimates that it costs $802 million on average (in 2000 dollars) to research
and develop a self-originated new chemical entity, including failures and cost of capital. It
is worth noting that after adjusting for inflation, the DHG 2003 cost estimates are roughly
two to four times as high as other estimates of pre-approval drug R&D costs (Love, 2003;
Public Citizen, 2001; OTA, 1993). The 2003 article represents a sophisticated analysis that
builds on the authors’ equally important article in 1991, and adds several refinements and
extensions of that prior analysis. There are, however, problems with the data and sampling
on which these results depend, and this commentary focuses on those problems.
∗Corresponding author. Tel.: +1 609 915 1588; fax: +1 609 924 1830.
E-mail address: email@example.com (D.W. Light).
0167-6296/$ – see front matter © 2005 Elsevier B.V. All rights reserved.
D.W. Light, R.N. Warburton / Journal of Health Economics 24 (2005) 1030–1033
A careful review of the article identified six serious sources of doubt about the validity
and usefulness of the source data and methods used in DHG 2003:
(1) First, the inherent comparability and reliability of the survey data must be questioned
because of variations in internal company cost allocation methods over time and across
companies. Because cost data used was proprietary and confidential, readers cannot
independent verification of the accuracy of the information is possible. Firms reported
R&D expenditures stretching back more than 10 years (to 1980), during which several
firms underwent mergers and/or changed accounting systems or practices. The degree
of potential variation is large, and these many variations in practice may compound on
each other, making any point estimate misleading. (Internal validity)
(2) Second, considering the clear interest of pharmaceutical companies in higher (rather
the intended use of the survey data, it is not unlikely that companies would deliberately
and systematically overstate costs in their survey responses (OTA, 1993). The survey
design did not permit independent review of the reported costs, so upward bias cannot
be ruled out. (Internal validity.)
(3) Third, the small, non-random firm sample (n=10) and drug sample (n=68) introduce
another potentially large source of variation and error into cost estimates. Although
the sampling of drugs was reported to be random, this is misleading, because the
selection of firms participating in the survey (which preceded the selection of drugs)
was not random (DHG 2003, pp. 157–158); randomization cannot be recovered once
lost at the first stage of sample selection. A total of 24 firms were invited to respond;
12 firms accepted and were asked to provide data on an unstated number of drugs,
randomly selected from those companies’ drugs in the proprietary database; 10 firms
provided usable data, covering development of 76 drugs; but drug data were usable for
only 68, and complete for even fewer (it is noted as an example that only 66 drugs
had Phase I trial cost data). No information is provided about how invited firms were
selected, nor whether they were selected from the universe of all US research-oriented
pharmaceutical firms or some other less representative universe. The 42% of invited
firms that responded (10 of 24) self-selected, and given the industry interest in higher
likely to choose to participate. (Internal validity)
that were researched, discovered, and developed in-house), whose costs are higher than
those for more typical “new” drugs. Only 35% of new drugs approved by the FDA
(from 1990 to 2000, FDA, 2004; from 1989 to 2000, NIHCREF, 2002) contained a
new molecular entity, and only 62.4% of the survey firms’ approved NCEs were said
to be self-originated (DiMasi et al., 2003b, p. 3, note 1). Thus “self-originated new
chemical entities” represented about 22% (62.4 of 35%) of new drug approvals. The
number of truly self-originated NCEs may be even smaller, because the authors note
that all phases of work may not have been done in-house and because there are well-
documented examples of companies making such claims that do not comport with the
facts (Mitsuya et al., 1989 Weinhold General Accounting Office, 2003). This might
D.W. Light, R.N. Warburton / Journal of Health Economics 24 (2005) 1030–1033
not matter much, in terms of estimating typical drug development costs, if all drug
development costs were similar, but they are not. According to DiMasi et al., 1991
(footnote 48), self-originated NCEs are 3.7 times more costly to develop than acquired
or administrations of existing drugs. The DHG 2003 estimates therefore pertain to the
most costly 22% of new drugs. (External validity.)
(5) Fifth, estimates of company spending on drug development are presented without
deducting (or at least identifying) government subsidies to this work. The industry
receives taxpayer funds from the NIH and other agencies, though amounts are not dis-
closed at the request of drug companies (General Accounting Office, 2003; National
Science Foundation, 2003). Given the use of cost data to justify drug prices and patent
protection, private (company-paid) cost and not social (total) cost is the policy-relevant
figure. (External validity)
(6) Finally, the cost estimates are not adjusted for tax deductions and credits. Drug R&D
expenses are fully tax-deductible each year, and there are special drug R&D tax provi-
by nearly 50%. Lower tax rates in the 1990s might reduce that figure somewhat, but
pre-tax costs clearly overstate true private (company) costs by a substantial percentage.
These significant concerns about internal and external validity call the study results into
question. Good science depends on different investigators analyzing the same data. Yet this
data and analyze it so that policy can be based on solid, objective, reproducible evidence.
The estimate of R&D cost in this article is widely cited and accepted as an authoritative
use of these cost estimates by the pharmaceutical industry and its advocates to influence
national and international policies, it is critical that they be scientifically valid and relevant
to the policy uses made of them. Shortly after DHG 2003 appeared, the Tufts Center for the
new chemical entity, including post-approval studies, was $897 million (TCSDD, 2003;
unsystematic data, and has dubious scientific validity. In addition, adding post-approval
studies to the costs of R&D is inherently questionable, because these “seeding trials” are
designed primarily to familiarize physicians with the new drug and encourage its use; they
are rarely randomized or blinded, but instead feature open-label case series, and are often
sponsored by company marketing departments (Kessler et al., 2004).
costs. Journal of Health Economics 22, 151–185.
costs Appendix B (Manuscript). Center for the Study of Drug Development, Boston.
D.W. Light, R.N. Warburton / Journal of Health Economics 24 (2005) 1030–1033 Download full-text
DiMasi, J.A., Hansen, R.W., Grabowski, H.G., Lasagna, L., 1991. Cost of innovation in the pharmaceutical
industry. Journal of Health Economics 10, 107–142.
and chemical types. Accessed at http://www.fda.gov/cder/rdmt/pstable.htm.
GAO General Accounting Office, 2003. Technology Transfer. NIH-Private Sector Partnership in the Development
of Taxol, Washington, DC.
Kaitin, K.I., 2003. Post-approval R&D raises total drug development costs to $897 million. Tufts Center for the
Study of Drug Development, Boston, Impact Report 5 (3), RS 2307.
Kessler, D.A., Rose, J.L., Temple, R.J., Schapiro, R., Griffin, J.P., 2004. Therapeutic-class wars-drug pro-
motion in a competitive marketplace. New England Journal of Medicine 331 (20), 1350–1353, Accessed
Love, J., 2003. Evidence Regarding Research and Development Investments in Innovative and Non-Innovative
Medicines. Consumer Project on Technology, Washington, DC.
Mitsuya, H., Weinhold, K., Yarchoan, R., Bolognesi, D., Broder, S., 1989. Credit Government Scientists with
Developing Anti-AIDS Drugs. New York Times, Letter, New York.
Patterns of Pharmaceutical Innovation, Washington, DC.
NSF National Science Foundation, 2000. Research and Development in Industry. National Science Foundation
Division of Science Resources Statistics, Arlington, VA.
Public Citizen, 2001. Rx R&D Myths. The Case Against the Drug Industry’s R&D Scare Card, Washington, DC.
is $897 million. Boston. Accessed at http://18.104.22.168/NewsEvents/RecentNews.asp?newsid=29 January
TCSDD (Tufts Center for the Study of Drug Development), 2004a. Sponsorship, Boston. Accessed at
TCSDD (Tufts Center for the Study of Drug Development), 2004b. Research Staff. Accessed at
Office of Technology Assessment, 1993. Costs, Risks and Rewards. Pharmaceutical R&D, Washington, DC.