Initializing the VA Medication Reference Terminology Using UMLS
John S. Carter, MBA1'2, Steven H. Brown, MD, MS3'4, Mark S. Erlbaum, MD, MS2,
William Gregg, MD4, Peter L. Elkin, MD5, Ted Speroff, PhD4, Mark S. Tuttle, FACMI2
'University ofUtah 2Apelon, Inc.3DepartmentofVeterans Affairs 4Vanderbilt University
5Mayo Foundation for Medical Education and Research
We developed and evaluated a UMLS Metathesaurus
medications and diseases they may treat. Based on
16 years of co-occurrence data, we created 977
candidate drug-disease pairs for a sample of 100
ingredients (50 commonly prescribed and 50 selected
at random). Our evaluation showed that more than
80% ofthe candidate drug-disease pairs were rated
"APPROPRIATE" byphysician raters. Additionally,
there was a highlysignificant correlation between the
overall frequency of citation and the likelihood that
the connection was rated "APPROPRIATE."
definitions in an ongoing effort to build a medication
Administration. Co-occurrence mining is a valuable
technique for initializing term definitions in a large-
scale reference terminology creation project.
BACKGROUND AND INTRODUCTION
The Department of Veterans Affairs (VA) Veterans
Health Administration (VHA) provides health care to
more than four million veterans and dependants.
VHA has developed and deployed
electronic tools to assist clinicians, including VISTA'
Architecture), CPRS2 (Computerized Patient Record
Administration), and others.
a variety of
The VHA National Drug File (NDF)4 is a nationally
maintained medication terminology used to support
VHA clinical applications. NDF is used at each of
AMIA 2002 Annual Symposium Proceedings
VHA's 172 medical centers for order entry, decision
regional automated mail-out pharmacies (57 million
prescriptions in 2001). It includes information about
drug costs, ingredients, and inventory management.
to send outpatient prescriptions
approximately 80,000 National Drug Codes (Figure
1). A cross-reference file lists the ingredients ofeach
of approximately 400
VHA is continually looking for ways to improve care
quality, promote patient safety, and reduce costs
Figure 1: Veterans Health Administration
National Drug File Drug Class Hierarchy
through a variety of means, including information
technology. One area under investigation is the use
National Drug File
: Digitalis Glycosides
: Beta Blockers/Related
FUROSEMIDE 1OMG/ML INJ
FUROSEMIDE 20MG TAB
Figure 2: The NDF-RT Model. Shield-shapes represent multiple-inheritance reference hierarchies. Rectangles
are named sets ofconcepts each representing a level of abstraction used to describe medications. Single-headed
arrows are IS-A hierarchies, and double-headed arrows represent other semantic relationships. Solid arrows are
the most commonly instantiated connections, while dotted arrows are used to describe unusual or problematic
cases. NDC=National Drug Code. VHA=Veterans Health Administration.
of reference terminologies and terminology services
to permit retrospective and real-time aggregation,
comparison, and sophisticated decision support.
terminology" as described by Rossi Mori.5 Reference
terminologies have a formal defmition for each term,
and are designed for data aggregation and retrieval.
Formal definitions can be represented using symbolic
Formal terminologies have been
found to aid the terminology mapping process8 and to
reduce term maintenance costs.9
area "third generation
NDF-RT, a formalization of the National Drug File.
Along with their many benefits, medications pose a
variety of risks including adverse reactions and
components of health care costs.
review of existing medication terminology products
found several areas for inprovement.'0 NDF-RT will
Finally, a recent
show how drug products are different (which could
be accomplished using simply a unique identifier
such as the National Drug Code), and also how they
The ability to browse, retrieve and
aggregate drugs along multiple axes of similarity is
necessary to support functions such as decision
support and retrospective analysis.
familiar Drug Class structure is also necessary for
users to have a familiar and clinically relevant drug
terminology. The three basic steps being followed to
create NDF-RT are model development, reference
taxonomy development, and term defmition.
NDF-RT uses a Description Logic-based reference
model (Figure 2) adapted from the Government
Computer-Based Patient Record
Project." The model includes orthogonal hierarchies
physiologic effect, clinical kinetics, and therapeutic
intent (disease) class, while preserving the existing
VHA Drug Classes.
We initialized the Mechanism of Action, Physiologic
Effect, and Chemical Structure axes by matching
VHA ingredient names to the National Library of
Subject Headings (MeSH)'2
terms. The MeSH "D - Chemicals and Drugs" tree
selected Pharmacologic Action links were used to
initialize the mechanism of action and physiologic
However, MeSH does not categorize
drugs by the diseases or manifestations they treat.
With more than 3,000 ingredients, each treating
multiple diseases, we sought a way to algorithmically
initialize the "therapeutic intent" axis as an efficiency
Medline indexing is based on assigned keywords
from the MeSH vocabulary. An article's keywords
also can be further specified using a controlled set of
qualifiers. The UMLS13 (Unified Medical Language
frequency of all qualified index term heading pairs.
Zeng and Cimino'4 stated that the co-occurrence pairs
show good sensitivity for drug-disease relationships.
Burgun and Bodenreider'5 found that drug-disease
co-occurrences were among the most frequent in the
Based on these findings, we hypothesized
that the MeSH co-occurrences would provide useful
drug-disease links and that an algorithm to usefully
occurrence data could be created. This hypothesis is
evaluated in the remainder of this manuscript.
We combined the co-occurrence files from 1986 to
2001 (total 16 years) and searched as follows.
we mapped ingredients from the VHA NDF to MeSH
a combination of lexical
matching and human review. Then, we collected all
co-occurrence pairs pointing from one of these drug
ingredients to a MeSH disease heading.
drug-disease pairs where the TU (therapeutic use)
qualifier frequency was greater than the maximum
frequency of either the AE (adverse effects), PO
(poisoning), or TO (toxicity) qualifiers.
excluded pairs where the number of drug-disease co-
occurrences identified by the TU qualifier was less
than 60% of the total co-occurrences (including the
AE, PO, TO, TU and all other qualifiers) for that
The resulting data produced a prohibitively large
number of drug-disease pairs for review, primarily
occurred once with the TU qualifier, our algorithm
manageable, we limited our review to those pairs that
occurred five or more times in the 16 years.
furosemide, a commonly prescribed ingredient in
diuretic medications, of which only the eight in
Figure 3 occurred five or more times.
Heart Failure, Congestive
Figure 3: Sample Candidate Ingredient-
For our experiment, we sought to determine whether
the algorithmic initialization of the terminology was
clinically relevant. We selected 100 ingredients from
the 3,300 total ingredients. First, we included the 50
most commonly prescribed drugs from the Nashville
prescriptions and more than half of total prescription
Therefore, we expected that the reviewers
would easily be able to review the drug-disease
connections for these ingredients, and that the review
of these ingredients would prove of immediate value
to VHA. We then selected 50 additional ingredients
at random from the remaining ingredients.
prepared a spreadsheet with one line for each drug-
disease candidate pair.
Three of the authors (SB, PE, WG) used the 9-point
connection according to the following criteria:
prophylaxis of, treatment of, or diagnosis of
symptoms, or closely associated diseases
APPROPRIATE (score 1-3) given the usual
course ofthe disease being treated, the usual
of themedication and
benefits derived from that medication."
is APPROPRIATE (score 7-9)
Within the larger groupings of APPROPRIATE,
reviewers considered the likelihood and sensibleness
of the disease-drug association based on their clinical
and did not discuss their results until all ratings were
The review task took between four and
eight hours per reviewer.
The reviewers worked independently
Of the 50 most common VHA ingredients,
not initialized either because they are not MeSH main
headings or because our mapping algorithm failed to
systematic discrepancy between one of the raters'
interpretation ofthe other two. Based on a follow-up
discussion among the raters, the outlying rater agreed
that he used the extreme ends and not the full 9-point
scale, resulting in dichotomizing the scale rather than
rating according to the written instructions.
rater's data are not included in the results presented.
Each of the two remaining reviewers examined a
total of 498 drug-disease co-occurrence pairs for the
ingredients. There was no statistically significant
difference between the raters for either the common
intraclass correlation coefficient = .47) or the random
intraclass correlation coefficient = .21).
the raters' scores were averaged into a pooled score.
Based on pooled scores, 414 of 498 (83.1%) of the
common ingredient co-occurrences and 378 of 479
(78.9%) of the random ingredient co-occurrences
were rated as "APPROPRIATE." There was a highly
significant (p < .001) correlation between the citation
appropriateness rating, as shown in Figure 4.
Of the remaining candidates, 14.9% of the common
and 19.6% of the random ingredient pairs were rated
"AMBIGUOUS" and the remainder
"NOT APPROPRIATE" (2% of common and 1.5%
Examination of the cases where the raters disagreed
scores) revealed that a small number of ingredients
caused the majority of these disagreements.
example, the ingredient "cilastatin sodium" was rated
"APPROPRIATE" against a wide range of infectious
diseases by one rater and "NOT APPROPRIATE" by
considered the ingredient as if it were combined with
Citation Frequency Decile
Figure 4: Highly significant correlation between
citation frequency and "APPROPRIATE" rating
(p < .001).
the antibiotic imipenem, which is the usual form in
which it is administered.
term definitions as part of the reference terminology
creation process can result in the creation of useful
data for human editors to review. Even though up to
30% of the NDF ingredients were not successfully
initialized in this experiment, the mining algorithm
uncovered more than 25,000 candidate drug-disease
This study suggests that more than
three fourths of those may be valid and retained in
As NDF-RT development continues, the task of the
human editors will be to eliminate inappropriate
diseases from the list generated in this experiment
rather than to create the entire disease list from
scratch. We anticipate that this will result in a
substantial productivity boost. The fact that only 50
ingredients account for a majority of prescriptions
success can be achieved without an extensive long-
term modeling process.
One limitation of this evaluation is that we do not
measure the false negatives, that is, the diseases that
are in fact treated by a drug but are not included in
the candidate pairs.Mining data from another
source, such as a corpus of patient records, will be
required to capture any missing "real-world" uses of
m /| ^~'A
The decision to include drug-disease links in the Download full-text
terminology can itself be challenged.
Rector'7 points out, the test of a terminology is how
well it supports software for key functions, including
data entry, information retrieval, mediation, indexing
In the VHA's clinical environment,
data entry, information retrieval and authoring for
medications most often takes place in the context ofa
approved for sale by the FDA in the context of a
specific, limited set of diseases, manifestations or
diagnostic situations in which they have been found
to be "safe and effective."
interactions and the like are indeed outside the scope
of the terminology, we argue that a limited list of
clinically important diseases appropriately treated by
a given drug do form a part of the drug's clinical
Further exploration of the boundaries
needed for medications and other subject domains.
Therefore, while a
Initializing the medication reference terminology
using UMLS Metathesaurus MeSH co-occurrences
provides a way to jump-start the terminology creation
and maintenance process while taking advantage of
work already done.
Our results show that this
terminology and knowledge base initialization efforts
could benefit from a similar method.
quantity of clinically
The authors would like to acknowledge the support
ofJim Demetriades ofthe VHA Enterprise Architects
Group and Don Lees, RPh, ofthe VHA Pharmacy
Benefits Management Group. Nels Olson and Munn
Maung provided programming support.
1. Kolodner R, editor. Computerizing large
integrated health networks: the VA success. New
york: Springer; 1997.
2. Meldrum K, Volpp B, Vertigan R. Department of
Veterans Affairs' Computerized Patient Record
System. Proceedings ofthe AMIA Annual
3. Johnson CL, Carlson RA, Tucker CL, Willette C.
Using BCMA software to improve patient safety in
Veterans Administration Medical Centers. Joumal of
Healthcare Information Management 2002;16(1):46-
5. Rossi Mori A, Consorti F, Galeazzi E. Standards
to support development ofterminological systems for
healthcare telematics. Methods ofInformation in
Medicine 1998;37(4-5):55 1-63.
6. Cimino JJ. Desiderata for controlled medical
vocabularies in the twenty-first century. Methods of
Information in Medicine 1998;37(4-5):394-403.
7. Subcommittee E. Standard Specification for
Quality Indicators for Controlled Health
Vocabularies. West Conshohocken: American
Society for Testing and Materials; 2001.
8. Lau LM, Johnson K, Monson K, Lam SH, Huff
SM. A method for the automated mapping of
laboratory results to LOINC. Proceedings ofthe
AMIA Annual Symposium 2000:472-6.
9. Cimino JJ, Clayton PD, Hripcsak G, Johnson SB.
Knowledge-based approaches to the maintenance ofa
large controlled medical terminology. Journal ofthe
American Medical Informatics Association
10. Lau LM, Lam SH. Applying the desiderata for
controlled medical vocabularies to drug information
databases. Proceedings ofthe AMIA Annual
11. Carter JS, Brown SH, Nelson SJ, Lincoln MJ,
Tuttle MS. The Creation and Use ofa Reference
Terminology for Inter-Agency Computer-based
Patient Records: The GCPRRTM Demonstration
Project. Proceedings ofthe AMIA Annual
12. Lipscomb CE. Medical Subject Headings
(MeSH). Bull Med Libr Assoc 2000;88(3):265-6.
14. Zeng Q, Cimino JJ. Automated knowledge
extraction from the UMLS. Proceedings ofthe AMIA
Annual Symposium 1998:568-72.
15. Burgun A, Bodenreider 0. Methods for
exploring the semantics ofthe relationships between
co-occurring UMLS concepts. Medinfo 2001 ;10(Pt
16. Merrick NJ, Fink A, Park RE, Brook RH,
Kosecoff J, Chassin MR, et al. Derivation ofclinical
indications for carotid endarterectomy by an expert
panel. American Joumal ofPublic Health
17. Rector AL. Thesauri and formal classifications:
terminologies for people and machines. Methods of
Information in Medicine 1998;37(4-5):501-9.