Content uploaded by Alfred Uhl
Author content
All content in this area was uploaded by Alfred Uhl on Apr 11, 2018
Content may be subject to copyright.
© 2014 by Elsevier B.V. doi: 10.1016/j.drugpo.2014.10.014
International Journal of Drug Policy 26 (2015) 119-121
This is a preprint version and not perfectly identical to the final version. The final version is available via
http://www.ijdp.org/article/S0955-3959%2814%2900295-3/abstract
Editorial
How credible are international databases for understanding substance use
and related problems?
Alfred Uhl, Geoffrey Hunt, Wim van den Brink, Gerry V. Stimson
An important aspect of understanding the dynamics of substance use in societies is to study how traditions,
fashions and policy interventions impact upon substance use and related problems. Since large, well-controlled,
long-term policy experiments are hardly feasible - they fail due to ethical, practical, and financial limitations -
much hope is often vested in naturalistic studies using high-quality databases, perceived to contain reliable, valid
data and comparable data from different countries. Many multi-lateral agencies, such as the World Bank, the
World Health Organization (WHO), the European Monitoring Centre on Drugs and Drug Addiction (EMCDDA),
the United Nations Office on Drugs and Crime (UNODC) and EUROSTAT have invested considerable resources
in collecting, analysing and publishing such international data.
The contents of these data collection systems often include vital statistics, service activity data, survey data,
expert opinions and complex calculations based on the former data. They are commonly used for quick and
simple ad hoc conclusions by policy analysts, the media, researchers and politicians. They are sometimes used as
primary data by scientists to perform secondary analyses which are used for conclusions often beyond the
objectives of the original data collection plan. As we hope to show, some scepticism is needed about how such
data can be used.
Researchers, practitioners, journalists and policy makers, who are not personally involved in the data collection
and aggregation process, usually trust tables produced by renowned institutions and use these data for simple
conclusions or for secondary analyses.
It is not our intention to claim that all international information systems are flawed. Nevertheless, we would
argue that scepticism should be our default position until we have good evidence that the data are of a robust
nature. A blind trust in these (aggregated) data is unjustified. We provide a number of examples where such
scepticism seems justified.
Drug related deaths in Europe - the plausibility test
How many people die of drug-related causes in Europe, and how does this vary between countries? This might
seem straightforward to establish. A commonly cited and often used source is the EMCDDA database on “drug
related deaths” (DRDs). Their most recent report (EMCDDA, 2014) reports that the drug related population
mortality rates per 100,000 inhabitants are lowest in France (0.9), Hungary (2.8), and Romania (3.0), and highest
in Norway (48.1), Denmark (50.9), and the UK (52.9).
These data suggest that the mortality rate in the UK is almost 60 times higher than in France. The difference in
these rates suggests that the data fail the plausibility test. No researcher aware of the drug situation in Europe
would take such a difference seriously. Hence, instead of accepting the data on face value, the central message to
take from this table should be that deriving conclusions from these data makes little or no sense. Furthermore,
any attempt by policy experts, the media and/or politicians to conclude that the drug policy in one country is
more successful than in another because of the number of DRDs is somewhat lower, as recently happened in
Austria, is also not justified.
Stimulants, amphetamine, Red Bull and accidental solvent sniffing: do respondents understand survey
questions?
ESPAD - the European School Survey Project on Alcohol and Other Drugs is a collaborative effort of
independent research teams in more than forty European countries. As its website indicates it is ‘the largest
cross-national research project on adolescent substance use in the world. The overall aim with the project is to
repeatedly collect comparable data on substance use among 15-16 year old students in as many European
countries as possible.’
Austria took part in the ESPAD study in 2003 and 2007. Because the Principal Investigators were sceptical as to
some of the 2003 results, they decided to design a validity study within the 2007 project. For this purpose, 100
students - one or two randomly selected from school classes that were taking part, were asked after they had
completed the initial survey to answer further questions. The selected students were asked whether they
understood the selected questions correctly, had known the correct answers and had provided the correct answers
(Schmutterer et al., 2008). The questions on amphetamine and solvents use in the total survey population yielded
8% lifetime prevalence of amphetamine use and 14% lifetime prevalence of sniffing solvents. Results from the
validation study showed that most students had coded energy drinks (primarily “Red Bull”) as
“stimulants/amphetamine use”. They also classified unintentionally smelling glue, paint or gasoline in everyday
life or intentionally smelling gas from a lighter to see if it had any effect as “sniffing solvents”. The validation
data suggest that the vast majority of the students classified as amphetamine users and solvent users in Austria
were false positives. The researchers speculated that a high true prevalence of stimulant and solvent use,
reflecting familiarity with these phenomena, might yield low empirical prevalence rates in student surveys, while
a low true prevalence, where individuals misinterpret these phenomena, paradoxically might yield relatively high
empirical prevalence rates. From this example, it becomes clear that at least in this case, the results of the of the
student survey on amphetamine and solvent use makes little sense and as a result the researchers made explicit
reference to these problems within their reports. However, in spite of these research caveats the EMCDDA’s
(2009) annual drug report rated Austria as having the highest amphetamine use among European student
populations.
Liver cirrhoses in Europe - coding variability and discretion
It is widely accepted that a high percentage of all liver cirrhoses in the developed world are alcohol induced,
even though alcohol is not the only cause of liver cirrhosis. According to the International Classification of
Diseases (ICD) coding system alcohol induced liver cirrhosis diagnoses should be coded “alcoholic liver
cirrhosis”. Using this classification system, Ramstedt (2002) in analysing death certificate data from all
European Union countries, discovered that only 8.5% of death from liver cirrhoses in Austria, 7.1% in Greece
and 6.6% in Italy were coded “alcohol related”, while in Finland the percentage was 80.3%. While the former
rates were incredibly low, the latter rate was extremely high. To understand why we should be sceptical of
accepting these figures at face-value and also why such differences exist, let us examine in more detail the
processes by which these figures are compiled by considering the case of Austria. Coders in Austria received a
directive that they should choose only one ICD code out of a list of several diseases in the death certificates and
discovered that the code “alcoholic liver cirrhosis” gave them a chance to aggregate the code “liver cirrhosis”
and the code “alcohol dependence” into one code. However, they misunderstood the intention of ICD. As a
consequence many cirrhoses that are aetiologically non-alcohol related are miscoded as “alcoholic liver
cirrhosis” if the death certificate mentions “alcohol dependence,” and cirrhoses with a clear alcohol aetiology are
misclassified as “non-alcohol induced” if the death certificate does not mention any “alcohol dependence” (Uhl
et al., 2009). The problem was recognised by some alcohol researchers and as a consequence they suggested
ignoring the attribute “alcohol induced" and instead consider the global “liver cirrhoses” rate only (Rehm et al.,
2010). Liver cirrhoses are not solely caused by alcohol use and other causes, like viral hepatitis, vary greatly
from country to country. Substituting the criterion “alcoholic liver cirrhoses” with “liver cirrhoses” for alcohol
studies is also rather questionable.
We also need to ask whether the standardised death rates (SDR) for liver cirrhosis ranging from very low in
Norway (3.4), the Netherlands (3.8), and Malta (4.8) to very high in Lithuania (27.7), Hungary (37.2), and
Romania (46.6) provide a plausible picture. Of course, we cannot rule out that such large differences actually
exist, but knowing the problems involved when assessing causes of death in different countries, we should not
accept such figures without questioning their authenticity.
Alcoholic beverages - empirical problems and conceptual flaws
An example of the combination of empirical problems and conceptual flaws relates to the average prices for
beer, wine and spirits as reported by WHO (2014). Experts in all EU countries are annually asked to provide the
average price of the most popular beer, wine, local spirit and imported spirit. Asking national experts to provide
data is a very cheap way of doing empirical research - and commonly quite successful, even if the experts
contacted do not know the facts and have to rely on more or less educated guess-work. The conceptual problem
here is that there is no such thing as the “most popular beer, wine, local spirit and imported spirit” in most
countries. The empirical problem is that even if a certain product dominated the market, retail prices could vary
enormously.
When an Austrian expert asked a WHO officer, who was in charge of such data collection: "What is the purpose
collecting arbitrary guesses from international experts without minimum levels of validity and reliability?", the
reaction of the WHO officer in charge was: “I agree this is a simple and not very precise way of estimating
prices in Member States. This questionnaire is very simple because it is a global questionnaire. ... Anyway, we
hope the data we get will provide some idea of the different prices in Europe (and globally). The prices will be
compared with data on Purchasing Power Parity for each country. I hope you will be able to find prices for me,
the next time you visit your supermarket" (World Health Organization, 2002). Some experts apparently were and
are willing to visit their supermarkets to help out WHO. As a result we learn in the WHO database that the most
popular local beer in Serbia costs 4.00 US Dollars while a 0.7 L bottle of the most popular local spirit costs 70
US Cents, that the most popular table wine in Greece costs 9.40 US Dollars, that the most popular local spirit in
Greece costs 22.30 US Dollar, while the most popular imported spirit in Greece costs 12.00 US Dollar. Although
the Austrian expert was ultimately able to avoid providing such data, experts in other countries continued to do
so, which resulted in many of the implausibilities noted above.
Another example is the attempt by a research institution to assess whether people drink with their meals, when
collecting data for the “Global Burden of Disease” study. One of the questions was: “How frequently do men eat
when they are drinking?" and the second one was formulated in the same way, but targeted women. The response
categories were “usually", “often", “rarely", “not at all". Even if a researcher knew exactly how often and in
which situations people drank and ate in a specific country, they could not possibly give a sensible answer. To
make it sound more scientific, the responding expert had to code the level of evidence based on the categories
“good survey of whole population", “reliable local survey", “based on other evidence", or “my best estimate".
The information that in some countries alcohol consumption is embedded in meals, while in other countries
alcohol consumption primarily takes place independently of meals sounds at first sight sensible. However,
knowing that this inference may possibly result from a question like the one above or similar ones should cast
serious doubts concerning the validity of the claim.
Conclusions
The aim of this editorial has been to raise critical awareness in researchers, scientific experts and policy makers
about interpreting and using data from international information systems, expecially when making cross-national
comparisons. The issue is particularly critical if the sources for such data are not disclosed in an explicit and
accessible way. We can only make sensible use of international comparative data if we truly understand the
processes by which the data are produced.
The examples also raise critical issues about the role of experts who supply data to European or international
databases. To adopt and uphold a critical perspective in one’s own work is often not easy, either for those
supplying the data or for those utilizing these databases. To behave like the three famous monkeys may make life
easier, but if we are interested in contributing to a better understanding of the world, we need to insist on
minimal standards and refuse to undertake tasks that cannot be done sensibly or even done at all. In this
commentary, we have only been able to provide a few examples which show that data should be interpreted with
scepticism and that data quality should be improved. Without doing elaborate validation projects, we cannot
quantify the error magnitude of different data sets. Consequently, we selected cases where the differences
between countries were so large that serious doubts can be raised even without such validation projects and/or
where the way the data collection procedure called for serious doubts about the data validity.
The data described in this editorial play an important role in the daily political discourse on drug and alcohol
policy and frequently referred to in scientific publications. The official numbers of drug related deaths and
particularly the rates of student substance use are commonly cited in relation to drug policy discourses. Rates of
liver cirrhoses, average prices of different beverages, and drinking with or without meals are frequently cited
issues when discussions on drug or alcohol policy take place.
As Dyson (2007) noted “The public prefers to listen to scientists who give confident answers to questions and
make confident predictions of what will happen as a result of human activities. ... They make confident
predictions about the future, and end up believing their own predictions. Their predictions become dogmas
which they do not question. The public is led to believe that the fashionable scientific dogmas are true. ... That is
why heretics who question the dogmas are needed.” This may go some way to explaining why absurd
procedures, questionable data and illogical conclusions, that should induce doubt and even amusement among
academically trained scientists, journalists and politicians remain widely unchallenged in the scientific world.
References
Dyson, F. (2007). Many colored glass: Reflections on the place of life in the universe (excerpt). VA: University
of Virginia Press.
EMCDDA. (2009). The state of the drugs problem in Europe annual report 2009. Luxembourg: European
Monitoring Centre for Drugs and Drug Addiction.
EMCDDA. (2014). Population mortality rates, 2005 or last year with available information.
http://www.emcdda.europa.eu/stats07/drdtab05a (01.04.14)
Ramstedt, M. (2002). Alcohol-related mortality in 15 European countries in the Postwar period. European
Journal of Population, 18(4), 307-323.
Rehm, J., Baliunas, D., Borges, G. L., Graham, K., Irving, H., Kehoe, T., et al. (2010). The relation between
different dimensions of alcohol consumption and burden of disease: An overview. Addiction, 105(5), 817-843.
Schmutterer, I., Uhl, A., Strizek, J., Bachmayer, S., Puhm, A., Kobrna, U., et al. (2008). ESPAD Austria 2007:
Europäische Schülerlnnenstudie zu Alkohol und anderen Drogen - Band 2: Validierungsstudie. Wien:
Bundesministerium fur Gesundheit.
Uhl, A., Bachmayer, S., Kobrna, U., Puhm, A., Springer, A., Kopf, N., et al. (2009).
Handbuch: Alkohol - Österreich: Zahlen, Daten, Fakten, Trends 2009 (Dritte über- arbeitete und ergänzte
Auflage). Wien: BMGFJ.
World Health Organization. (2002). Personal e-mail communication.
World Health Organization. (2014). Global health observatory data repository (European region).
http://apps.who.int/gho/data/node.main-euro.A1118? lang=en&showonly=GISAH (01.04.14)
Acknowledgements
The ideas in this editorial were first discussed at a workshop in Vienna hosted by Sigmund Freud University and
held at the Beethoven Hotel, Vienna in December 2013, and we are grateful for comments from participants
including Franca Beccaria, Julian Strizek, Betsy Thom, Sébastien Tutenges, and Laura Williamson.
Conflict of interest statement
Alfred Uhl: Regularly provided data for international data bases and interpreted data from these databases within
projects funded by the EU and the Austrian Ministry of Health.
Gerry V. Stimson and Geoffrey Hunt: None.
Wim van den Brink: Received speakers honoraria from Lundbeck, Merck Serono, Schering-Plough, Reckitt
Benckiser, Pfizer, and Eli Lilly, investigator initiated industry grants from Alkermes, Neurotech, and Eli Lilly,
has been a consultant to Lundbeck, Merck Serono, and Schering-Plough, and has performed paid expert
testimony for Schering-Plough.
Corresponding author:
Alfred Uhl, Addiction Research and Documentation of the Anton-Proksch-Institute, Graefin Zichy-Straße 6,
1230 Vienna, Austria and Sigmund Freud Private University, Schnirchgasse 9A, 1030 Vienna, Austria,
Tel.: +43 650 2888883, E-mail address: alfred.uhl@uhls.at
Co-authors:
Geoffrey Hunt, Centre for Alcohol and Drug Research, University of Aarhus, Denmark
Wim van den Brink Psychiatry and Addiction, Academic Medical Centre University of Amsterdam, Amsterdam,
The Netherlands
Gerry V. Stimson, Emeritus Professor Imperial College London, United Kingdom and Visiting Professor London
School of Hygiene and Tropical Medicine, United Kingdom
