Content uploaded by Sanda Erdelez
Author content
All content in this area was uploaded by Sanda Erdelez on Dec 31, 2018
Content may be subject to copyright.
Role of Domain Knowledge in Developing User-Centered
Medical-Image Indexing
Xin Wang and Sanda Erdelez
School of Information Science and Learning Technologies, University of Missouri, 303 Townsend Hall, Columbia,
MO 65211. E-mail: xwang7708@gmail.com; erdelezs@missouri.edu
Carla Allen
School of Health Professions, University of Missouri, 619 Lewis Hall, Columbia, MO 65211.
E-mail: allencar@health.missouri.edu
Blake Anderson
Informatics Institute, University of Missouri, 241 Engineering Building West, Columbia, MO 65211.
E-mail: beakv6@mail.missouri.edu
Hongfei Cao
Department of Computer Science, University of Missouri, 238 Engineering Building West, Columbia, MO 65201.
E-mail: Hc79b@mail.missouri.edu
Chi-Ren Shyu
Informatics Institute, University of Missouri, 241 Engineering Building West, Columbia, MO 65211.
E-mail: shyuc@missouri.edu
An efficient and robust medical-image indexing proce-
dure should be user-oriented. It is essential to index
the images at the right level of description and ensure
that the indexed levels match the user’s interest level.
This study examines 240 medical-image descriptions pro-
duced by three different groups of medical-image users
(novices, intermediates, and experts) in the area of radio-
graphy. This article reports several important findings:
First, the effect of domain knowledge has a significant
relationship with the use of semantic image attributes in
image-users’ descriptions. We found that experts employ
more high-level image attributes which require high-
reasoning or diagnostic knowledge to search for a medi-
cal image (Abstract Objects and Scenes) than do novices;
novices are more likely to describe some basic objects
which do not require much radiological knowledge to
search for an image they need (Generic Objects) than are
experts. Second, all image users in this study prefer to
use image attributes of the semantic levels to represent
the image that they desired to find, especially using those
Received July 3, 2011; revised September 16, 2011; accepted September 16,
2011
© 2011 ASIS&T •Published online in Wiley Online Library
(wileyonlinelibrary.com). DOI: 10.1002/asi.21686
specific-level and scene-related attributes. Third, image
attributes generated by medical-image users can be
mapped to all levels of the pyramid model that was
developed to structure visual information. Therefore, the
pyramid model could be considered a robust instrument
for indexing medical imagery.
Introduction
Medical images provide vital clinical data and are
considered a powerful educational resource due to their
immediate, informative, and illustrative nature. Medical
images can be used by clinicians for their daily practice
of medicine, such as making diagnoses, planning treat-
ment, and monitoring responses to therapy as well as for
medical education and research (Cleveland & Cleveland,
2009; Kalpathy-Cramer & Hersh, 2010; Müller, Michoux,
Bandon, & Geissbuhler, 2004). Paststudies have reported sig-
nificant learning improvements when using medical images
during classes and self-education for medical students and
residents (Dawes, Vowler, Allen, & Dixon, 2004; Kalpathy-
Cramer & Hersh, 2010). A single hospital radiology depart-
ment alone produced 50,000 images per day in 2006 (Müller
© 2011 ASIS&T • Published online 24 October 2011 in Wiley Online
Library (wileyonlinelibrary. com). DOI: 10.1002/asi.21686
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY, 63(2):225–241, 2012
226 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—February 2012
DOI: 10.1002/asi
et al., 2007). With the dramatic explosion of digital image
collections in medicine, it is important to develop advanced
techniques for effective and efficient management of this
information, enabling users quick and easy access in a
clinically meaningful way.
Image information systems (e.g., picture archiving and
communication systems) provide rapid access to digital-
ized film images and allow users to access medical-image
databases based on combinations of a patient’s identification,
visit dates, and study characteristics (e.g., modality and study
description) (Müller et al., 2004). However, to fulfill users’
various requirements under different contexts of use (e.g.,
clinical decision-making, medical reference, and education),
it is essential to provide multifaceted and user-centered access
points to make this visual information findable through var-
ious means such as searching for the same physical regions,
image quality, or disease processes. Since management of and
access to these large medical-image repositories has become
increasingly challenging, an important question when design-
ing a medical-image retrieval system is: What should be
indexed in a medical image? An ideal image-retrieval sys-
tem should be based on indexing the images at the right level
of description and ensure that the indexed levels match users’
interest level (Jaimes, 2006).
To address this issue, this research aimed to examine what
levels of image attributes are needed by medical-image users
with different levels of domain knowledge. In this study,
image attributes refer to any kind of feature, component,
or property of an image that can be represented by a visual
information-processing system (Jörgensen, 1995).
Image indexing is the process of creating surrogates (e.g.,
color, shape, objects, and semantic meanings) for images
to represent the image content. When indexed, an image is
assigned to classes of similar-images sets. Indexing of med-
ical images is a difficult task due to the complex nature of
visual information, which is tacit and wordless with multi-
ple syntactic and semantic levels.Additional difficulties arise
from the complexities of medicine as a domain of profes-
sional practice. Medical images have a multifaceted structure
of image content, as do other consumer-oriented images (e.g.,
art photography). The content of medical images is highly
specialized and requires consideration of domain-specific
issues in indexing procedures. For instance, when indexing a
chest x-ray with an area of increased density in the lower part
of the left lung, different classes of image attributes, such as
the modality class (radiograph), the anatomic field of view
(anteroposterior view of the thorax), the major anatomic seg-
ments (left lung), and the local structure (increased density
in the left lower lobe), should be taken into account. Render-
ing the implicit meanings from x-ray images is a process of
decoding that requires knowledge of pathological anatomy
(Pasveer, 2006). In addition, medical images of a given type
(e.g., chest x-rays) are relatively similar and differ only in
minute details, as compared to general-purpose images. As a
result, highly precise and comprehensive indexing techniques
are needed to detect these subtle differences (Tang, Hanka, &
Ip, 1999). The retrieval of medical images requires robust
and effective indexing procedures (Cleveland & Cleveland,
2009), which eventually will reduce medical image users’
resource-seeking time and maximize their resource-using
time.
In the past several decades, two main approaches to
image indexing and retrieval have been developed: concept-
based approaches and content-based approaches (Chen &
Rasmussen, 1999; Chu, 2001; Enser, 2000; Rorissa, 2008).
Concept-based image retrieval, also variably named as
“description-based” or “text-based” image retrieval, refers
to retrieval from text-based indexing of images that may
employ keywords, subject headings, captions, or natural lan-
guage text (Chen & Rasmussen, 1999). Content-based image
retrieval refers to retrieval based on indexing visual features
extracted from the digital image itself through automated
computer analysis of image content such as color, shape, or
texture (Eakins, 2002; Chen & Rasmussen, 1999). Between
these two approaches, the discussion of “what should be
indexed in an image” is largely limited to the research com-
munity of concept-based indexing (e.g., Layne & Lunin,
1994; Rasmussen, 1998), where researchers are mainly from
the field of library and information science. For content-based
indexing and retrieval, a technique that originated from com-
puter science, the decisions of “what should be indexed” are
often made based on the capability of the automated fea-
ture extraction and matching techniques (Rasmussen, 1998)
rather than identifying the users’ needs for visual features
in various contexts and collections (Goodrum & Spink,
2001). As a result, the increasingly impressive techniques
(e.g., automated image retrieval based on color, size, shape,
etc.) in content-based image retrieval have not been broadly
accepted by users (Eakins, 2002; Liu, Zhang, Lu, & Ma,
2007). According to Yoon and O’Connor (2010), if indexed
image attributes do not represent users’ subjective feelings
and interpretations evoked while viewing images, the queries
will not retrieve satisfying search results because users’con-
notative search needs cannot be met. For instance, in the
medical domain, image-based medical practitioners do not
merely seek images that have similar low-level features such
as grey-scale pixel statistics, texture, or degrees of density
of an image but also want to locate an image based on
high-level reasoning features such as patients’ body parts
(e.g., lung, chest), positioning (e.g., supine, posteroanterior,
and oblique), or defects/diseases (e.g., pneumonia, pediatric
trauma). As Pauwels (2006) stated, to fulfill certain functions
and uses, visual representation must have necessary proper-
ties that not only include the characteristics of the media but
also the contexts of production and use. The content-based
image-retrieval researchers must resolve the discrepancy
between the limited functionality of content-based image
retrieval and users’ sophisticated needs. Another deficiency
of many content-based indexing techniques is that low-level
image features are not sufficiently precise for detecting sub-
tle differences in medical images (Tang et al., 1999). Tang
et al. (1999) suggested that indexing based on several con-
ceptual levels of image content seems key to developing an
intelligent medical information system.
227
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—February 2012
DOI: 10.1002/asi
To improve the precision and recall of image retrieval, and
to design usable and useful systems that provide easy access
to medical images, it is important to shift document-centered
indexing to user-centered indexing that is tailored to users’
requirements (Fidel, 1994). Jörgensen (1995) proposed that
more evidence about how humans describe images needs to
be gathered to improve image indexing. Without studying the
description of images, it is difficult to understand “what the
user is searching for and how she/he tries to find it” (Jaimes &
Chang, 2000, para. 3.1).
When developing a user-centered indexing procedure for
medical images, users’ domain knowledge is an important
factor. Past studies on the topic of text search have found
that domain knowledge plays an important role in deter-
mining how users describe and represent their information
needs (Drabenstott, 2003; Hsieh-Yee, 1993; Marchionini,
Dwiggins, Katz, & Lin, 1993; Sihvonen & Vakkari, 2004;
Vibert et al., 2009). However, the effect of domain knowl-
edge has not been adequately addressed in image-retrieval
studies.
The goal of this study is to examine the differences
when medical-image users with different levels of domain
knowledge describe their visual information needs. To serve
this research goal, we studied medical-image descriptions
produced by three groups of users (domain experts, inter-
mediates, and novices) in the subdomain of medicine—
radiography—to learn whether the effect of domain knowl-
edge impacts the classes (or “levels”) of image attributes
that these users employ. Based on a conceptual framework
(Jörgensen, Jaimes, Benitez, Shih-Fu, 2001) for classifying
visual information, we mapped end-users’ medical-image
descriptions to 10 levels (or classes): Type/Technique, Global
Distribution, Local Structure, Global Composition, Generic
Object, Generic Scene, Specific Object, Specific Scene,
Abstract Object, and Abstract Scene. The research findings
indicated that the three different end-user groups preferred
to employ different levels/classes of image attributes in their
descriptions. The findings provide important input for design-
ing the interfaces and functionality for both content-based and
concept-based image-retrieval systems. System developers or
library professionals may use these findings to provide better
user-oriented access points to these images.
Literature Review
Role of Domain Knowledge in Designing Information
Retrieval System
With human information needs increasingly extending to
visual materials, the issue of describing image information
needs becomes more important (Choi & Rasmussen, 2003;
Goodrum, 2005; Roberts, 2001). The problem of formulating
queries for searching images becomes magnified due to the
wordless nature of images (Cleveland & Cleveland, 2009).
Describing a desired image is often the first obstacle that
users encounter in the image information-searching process,
a task which is often time-consuming, tedious, and frustrating
(Kammerer, Frankewitsch, & Prokosch, 2009). Goodrum
(2005) stressed that image retrieval is actually an act of trans-
lation. Users have to first translate their image information
needs to suitable queries. O’Connor (1999) asserted that the
action of describing the needed image attributes is a process
of linking the knowledge state of users to the image doc-
ument. Therefore, when designing image-retrieval systems,
image searchers’ needs and their knowledge states need to be
examined to provide better match results.
The capability of users to describe their information needs
and formulate queries depends crucially on their domain
knowledge (Vakkari, 2002). Past studies have found that
domain knowledge influences the number of search terms
individuals may generate, the initial choice of search terms,
and the expansion upon and change of search terms. In
addition, domain knowledge also determines the quality of
search terms produced. Borgman (1996) found that a subject
expert can formulate a more specific and comprehensive ini-
tial query. Vakkari (2002) explored how experts and novices
in pedagogics expanded queries with the ERIC thesaurus.
This study found that domain knowledge was correlated with
the individuals’ ability to express their information needs
and the selection of search terms. Novices were reported
to need more terminological support from the system due to
their more limited vocabulary when they performed both easy
and difficult search tasks. Experts were found to use more of
their own terms in the easy tasks, but they also needed more
terminological support in the difficult tasks.
These past studies have suggested that users with dif-
ferent domain knowledge may employ different classes
of image attributes when they describe or represent their
visual-information needs. Therefore, a user-centered image-
indexing procedure needs to be established to accommo-
date different levels of user groups’ search behaviors and
preferences.
Many past studies have measured domain knowledge
through expert–novice comparison (e.g., Hsieh-Yee, 1993;
Shute & Smith, 1993; Vibert et al., 2009). In this study, we
operationally measured domain knowledge with three levels:
experts, intermediates, and novices, which was based on
the credentials of these participants (i.e., their training,
experience, job responsibilities).
Existing Conceptual Frameworks and Empirical Studies
for Indexing Images
Several conceptual models have been built to index images
based on image attributes. Panofsky (1962) proposed that
image attributes can be categorized into three levels: pre-
iconography level (basic or natural characteristics, usually
names of objects), iconographic level (attributes that have
meaning attached to them as a result of interpretation),
and iconological level (attributes that involves syntheses and
multiple interpretations). Shatford (1986) suggested that the
conceptual levels of images could be divided into a “spe-
cific level” (individually named people, animals, and things),
a “generic level” (kinds of people, animals, and things),
228 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—February 2012
DOI: 10.1002/asi
FIG. 1. A “pyramid” structure is represented by a pyramid. Adopted from Jörgensen, C., Jaimes, A., Benitez, A. B., & Shih-Fu, C. (2001). A conceptual
framework and empirical research for classifying visual descriptors. Journal of the American Society for Information Science &Technology, 52(11), 938–947.
Reprinted with permission from Wiley-Blackwell and theAmerican Society for Information Science and Technology.
and an “about level” (abstractions manifested or symbol-
ized by objects or beings). Jörgensen (1995) found 12
classes of image attributes: literal object, people, people-
related attributes, art historical information, color, visual ele-
ments, location, description, abstract concepts, content/story,
external relationship, and viewer responses. Most recently,
McCay-Peet and Toms (2009) suggested indexing image
attributes based on two categories: conceptual attributes
(the subject of the image, the “of-ness” and “about-ness”
of an image) and the descriptive attributes (nonconceptual,
explanatory characteristics of an image).
Based on these conceptual models, many researchers have
conducted empirical studies to understand how users access
and search images based on these attributes. For instance,
Armitage and Enser (1997) used the Panofsky–Shatford
model to ascertain which categories of image descriptions
were used most by users of seven libraries. They found
that the specific-who (individually named person, group,
thing), the generic-who (kind of person or thing), and the
specific-where (individually named geographical location)
categories were used significantly more than average. Fur-
thermore, Chen (2001) adopted Jorgenson’s (1995) 12 classes
and Enser and McGregor’s (1992) model to study image
queries of undergraduate students majoring in art history
and found that these end-users mainly employed image
attributes belonging to Enser and McGregor’s categories of
Unique and Non-unique, and Jörgensen classes of Loca-
tion, Literal Object, Art Historical Information, People, and
People-Related.
The aforementioned conceptual frameworks provide dif-
ferent perspectives for indexing and structuring visual
information, but a missing component in these frameworks
is the role of domain knowledge in image feature representa-
tions. Because a major purpose of this study is to investigate
whether domain knowledge impacts how different users rep-
resent their image information needs, we considered the
pyramid model (Jörgensen et al., 2001), to be more appropri-
ate to serve as a conceptual framework for this study. Another
important characteristic of this model is that it provides
a fine-grained division of levels in terms of visual-content
information and can potentially offer more precise and sensi-
tive measures of differences existing between different user
groups.
Conceptual Framework of This Study
After synthesizing concepts from the a broad body of lit-
erature in diverse fields such as cognitive psychology, library
science, art, and computer science, a 10-level pyramid for
indexing different aspects of visual information (see Fig-
ure 1) was developed and applied (Jörgensen et al., 2001,
p. 939). This pyramid classifies image features into 10 lev-
els: Levels 1 to 4 are labeled as “Syntax.” Syntactic levels
are the most basic levels and focus on those visual elements
that are arranged without considering the meaning of those
arrangements. These most basic levels refer to how people’s
senses perceive responses to light. Thus, it does not require
much knowledge to describe these visual elements, and no
interpretation takes place at these four levels.
On the other hand, Levels 5 to 10 are labeled as “Seman-
tics/Concepts.” To describe visual elements from Levels
5 to 10, users need to involve their background knowl-
edge, experiences, and interpretation. These semantic levels
refer to people’s conceptual responses when dealing with
the meaning of visual elements. Jaimes and Chang (2000)
stressed that in this pyramid, the higher the level, the more
knowledge is needed to formulate an image description.
Because this model addresses the indexing issues of a
broad range of digital images, it does not give specific con-
sideration for medical images. We modified explanations of
each level from the original model to make them more appli-
cable in the context of diagnostic imaging. A domain expert
229
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—February 2012
DOI: 10.1002/asi
FIG. 2. The modified pyramid for classifying x-ray image features.
in radiography was involved in the process of developing the
new operational definitions of these levels. Figure 2 shows
the modified model that we used for conducting this study.
Syntax Levels (Levels 1–4)
In our modified model, the Type/Technique (1st) level
refers to the techniques and modalities used to produce the
image. In radiography, there are different imaging modalities
such as “x-rays,” “CTs,” “MRIs,” and so on.
The Global Distribution (2nd) level refers to global con-
tent and is measured in terms of low-level perceptual features
such as spectral sensitivity (color) and frequency sensi-
tivity (texture). In radiography, it refers to overall image
characteristics such as “high or low contrast” and “grayed
out.” Descriptions at this level are often in the form of
judgment of technique, such as “overexposed,” “negative
image,” “grainy,” and “penetration.”
The Local Structure (3rd) level is concerned with extrac-
tion and characterization of the individual components of
the image. It refers to the basic visual elements (e.g., “dot,”
“line”) and local details (e.g., “localized density or texture,”
“dark spots,” “shadow,” “marker placement,” etc.).
At the Global Composition (4th ) level, we focus on ana-
lyzing the image as a whole, but use the basic elements such as
“line,” “circle,” and so on. Examples in radiography include
“symmetry,” “collimation,” and “the size of an image.”
Semantic Levels (Levels 5–10)
The Generic Objects (5th) level refers to the most general
level of object description, describing the image as a whole,
which can be recognized with everyday knowledge. Image
attributes provide a name, but not a specific medical name,
for generic objects. Examples are descriptions such as “left
leg,” “upper extremity,” “marker,” and “contrast studies.”
The Generic Scene (6th) level refers to scenes at their
most general level of description. For radiological images,
we focus on patient position or projection, scenarios involv-
ing the physical performance of the exam. Examples of
generic scenes include various judgments regarding patient
position or projection such as “posteroanterior,” “anteropos-
terior,” “upright,” “lateral,” and “oblique.”
The Specific Objects (7th) level refers to identified and
named objects that are the primary subject of the overall
image or local objects. Specific knowledge of the objects in
the image is required, and such knowledge is usually objec-
tive because it relies on known facts. Examples of this level
include “chest,” “lumbar spine,” “proximal femur,” “central
lines,” “ET tube,” “surgical clip,” and “sternal wires.”
The Specific Scene (8th) level refers to image attributes
that carry specific knowledge about the scene. Scenarios at
this level are where users make judgments of the quality
of the image; however, those judgments are not diagnostic
judgments. Examples include “proper central ray position,”
“distorted,” “clipped,” and “elevated.”
230 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—February 2012
DOI: 10.1002/asi
The Abstract Object (9th) level refers to specialized or
interpretative knowledge about what the objects represent.
Users assign meaning which summarizes the information
gleaned from the image. Examples include judgments about
the patients’ age and gender, such as “male patient,” “female
patient,” and “pediatric patient.”
Similarly, the Abstract Scene (10th) level refers to what
the image as a whole represents. Image features at this
level include diagnostic or pathology judgments, such as
“fracture,” “bowel obstruction,” and “perforation.”
Research Questions
Past studies have suggested that users with different
domain knowledge may represent their visual-information
needs using different means. In this study, we operationally
measured domain knowledge at three levels—experts, inter-
mediates, and novices—to form the novice–intermediate–
expert comparisons. In addition, based on the pyramid model,
there are two broad categories of image attributes: image
attributes at the syntactic levels (Levels 1–4) versus image
features at the semantic levels (Levels 5–10). Our first
research question aims to answer:
RQ1: Which levels of image attributes are used by medical-
image users when they intend to search medical images?
In addition, we aim to investigate how domain knowledge
affects the ways that image users represent their information
needs. With this in mind, another two research questions were
raised:
RQ2: Do domain experts, intermediates, and novices dif-
fer significantly in terms of employing image attributes at
the syntactic levels in their image descriptions? What are the
differences?
RQ3: Do domain experts, intermediates, and novices dif-
fer significantly in terms of employing image features at the
semantic levels in their image descriptions? What are the
differences?
Methods
The purpose of this study was to investigate the relation-
ships between domain knowledge and the use of different
levels of image attributes in image descriptions. In the fol-
lowing sections, we explain the selection of participants,
data-collection procedures, measures of variables, and data
analysis.
Participants
Forty-one participants with different levels of knowledge
in diagnostic imaging were recruited to participate in this
study. There were 14 junior students who were enrolled in
their second semester of study in the professional phase of
a diagnostic imaging program, 12 senior students who were
enrolled in their fifth semester of study in the professional
phase of a diagnostic imaging program, and 15 experts who
are radiologic technologists registered with the American
Registry of Radiologic Technologists and have at least 1 year
of full-time, professional work experience.
Junior students were novices who had not learned all
essential procedures. Senior students were intermediates who
had learned all essential procedures. The professional radi-
ologic technologists were experts who were practitioners in
local hospitals. All three groups of participants had back-
ground knowledge in anatomy and physiology. Novice and
intermediate participants in this study were recruited from the
radiography department of a state university. Student partici-
pants were offered “release time” from clinical obligations in
exchange for their participation. Each expert participant was
recruited from three local hospitals. Expert participants were
offered a $30 gift certificate as compensation for their time.
Participants were invited to designated classrooms on cam-
pus to complete the describing task. This study was approved
by the University of Missouri Health Science Institutional
Review Board. All participation was voluntary.
Procedure
Using the data-collection method reported in Jörgensen
(1998), six digital x-ray images were presented to the partic-
ipants on a laptop, and each participant was asked to produce
retrieval-oriented image descriptions (see Appendix B). Two
simulated retrieval scenarios designed with the help of a
domain expert were used to solicit users’ image descriptions.
We strove to create realistic scenarios that reflected the infor-
mation needs of users. For student groups, the scenario was:
“Suppose an image database at the Radiography program is
available for students searching for references as illustrations
in their projects. You need to find an image similar to the one
projected on the screen. How would you describe this image
that you intend to locate?” For the expert group, the scenario
was: “Suppose you are preparing training for fellow tech-
nologists and need to find an image similar to this one from
an image database at your hospital.” Then each participant
was asked to write down the image descriptions on paper
using complete sentences or longer phrases. Each participant
was asked to complete an image description within 3 min, a
suitable time period for a participant to finish describing an
image based on the feedback from our pilot study. In this pilot
study (Wang, 2011), a representative participant from each
user group completed the whole data-collection procedure to
ensure that the task instructions were clear and understand-
able. Based on the pilot study, we were able to determine how
long it takes for a participant to complete an image description
and how long to complete the whole study session.
Visual Materials
We obtained six x-ray images (see Appendix D) from the
centricity picture archiving and communication system of a
231
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—February 2012
DOI: 10.1002/asi
TABLE 1. Number and percentage of image descriptions produced by
novice, intermediate, and expert groups.
User groups No. of descriptions %
Novices 78 32.5
Intermediates 72 30.0
Experts 90 37.5
Total 240 100.0
local hospital and appropriately de-identified them to main-
tain patient confidentiality. The images were purposefully
selected to exhibit a wide range of image features, including
both syntactic and semantic attributes, for maximum descrip-
tive possibility. For example, all images demonstrate disease
or defect to enable participants to describe the image at that
level.
Data Analysis
In total, 246 image descriptions were collected (41
subjects ×6 images). Because 1 novice participant did not
follow the instructions, six image descriptions (Images A–F)
were not included in the final analysis. As a result, we ana-
lyzed 240 image descriptions. Table 1 displays the number
of descriptions collected from each user group.
After collecting the image descriptions, we applied content
analysis to analyze the data. First, we split image descrip-
tions into fragments suitable for categorization (Hollink,
Schreiber, Wielinga, & Worring, 2004). Care was taken in
the segmentation process to ensure that each fragment con-
sisted of a single image attribute. Fragments found to initially
contain multiple image attributes were split into smaller
fragments until just one image attribute existed. Based on
the parsing-process guidelines created by Hollink et al.
(2004), we developed a set of parsing rules that are proper
for segmenting medical-image descriptions (see Appendix
A). Figure 3 is one of the images (Image D) used in this
study. The following section illustrates the segmentation
process and demonstrates the differences typical of novice,
intermediate, and expert descriptions. We used “/” to indicate
the segmentation between fragments.
•A novice’s image descriptions for Image D.
Find an image of a/lumbar spine./ Spine should
show/scoliosis/in lumbar region./ Markers should be
present/with minimal collimation./
•An intermediate’s image descriptions for Image D.
It seems the image/was done/as an upright/abdomen/.
You can properly see/the diaphragms./ The patient
has/scoliosis/of the lumbar spine/. It looks like the
patient has/some gas/air/in the stomach/and intestines/.
The image has a/proper marker/and is properly posi-
tioned/. It seems the patient has/ some calcifications
/near the hip./
FIG. 3. Image D in this study.
•An expert’s image descriptions for Image D.
This appears to be an/x-ray/of the upper portion of the
patients abdomen./ This appears to be /supine/ because
there is no erect/upright label present. /There are sur-
gical clips present/on the right upper quadrant. /The
patient has mild to severe scoliosis/fracture /in the
thoracolumbar spine/.
The segmenting process resulted in a set of 2,005 frag-
ments. Among them, 398 fragments were removed because
they were not related directly to the image (e.g., “This is an
image of,” or “This is a,” “This patient is”). As a result, there
were 1,607 valid fragments available for analysis.
After segmenting these descriptions into fragments, we
mapped the fragments to the 10 levels of the pyramid
model. Because the mapping process involves subjective
decisions, two coders were involved in the coding process
and independently analyzed the segmented fragments. The
intercoder reliability (Cohen’s κ) reached 0.79, which reflects
a substantial agreement between two coders. All disagree-
ments between the two coders were discussed and eventually
resolved through discussion.
Variables and Measures
In this study, the effect of domain knowledge was the
independent variable, which was measured operationally by
the credentials and experiences of participants in the field of
radiography. Again, there were three levels of knowledge in
diagnostic imaging: novice, intermediate, and expert.
The dependent variables are the number of occurrences of
image attributes at the different levels of the pyramid model.
Table 2 demonstrates the measures for these independent
variables.
Results and Discussions
To build a user-centered indexing procedure, we analyzed
image attributes that occurred in novices,’ intermediates,’ and
232 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—February 2012
DOI: 10.1002/asi
TABLE 2. Measures of dependent variables.
Levels Variables Measures Examples of image features
1 Type/technique No. of occurrences of the modality features x-ray, radiograph, CT, MRI
2 Global distribution No. of occurrences of the image features which are
measured in terms of low-level perceptual features such
as spectral sensitivity (color) and frequency sensitivity
(texture)
Overall contrast–high or low contrast–“grayed out,”
density–black, white, “toast,” overexposed
3 Local structure No. of occurrences of the image features which are
the extraction and characterization of the individual
components of the image
Localized density or texture–dark spots, marker
placement, light spots, “shadow,” fuzzy, artifacts
with no specific names
4 Global composition No. of occurrences of the image features which describe
the image as a whole, but focus on the basic elements
such as line, circle, etc.
Symmetry; image is circular; collimation (an edge line
on x-ray images), size (14 ×17), barrel chest
5 Generic objects No. of occurrences of the image features which are at the
most general level of object description, describing the
image as a whole, which can be recognized with everyday
knowledge
left leg, right arm, upper extremity, contrast studies,
marker
6 Generic scene No. of occurrences of the image features which are
related to patient position or projection, or scenario
involves the physical performance of the exam
PA orAP, lateral, oblique; supine, upright, decubitus;
using a cone, portable
7 Specific objects No. of occurrences of the image features which are
named objects that are the primary subject of the
overall image
1. Global–Chest, Lumbar Spine, proximal femur ...
2. Local–central lines, ET tube, sternal wires,
“scapulae seen in lung fields,” “EKG lead,” “surgical
clip,” gas patterns, free air, calcifications, contrast
accumulation
8 Specific scene No. of occurrences of the image features that carry
specific knowledge about the scene such as a scenario
of how to make possible correction; scenario conveys
judgment about image quality, but not diagnostic
judgments.
Rotated/not rotated; proper central ray position, not
centered well; magnified; distorted; clipped; elevated.
9 Abstract objects No. of occurrences of the image features that requires
specialized or interpretative knowledge about what the
objects represent
male, female; older, pediatric patient
10 Abstract scene No. of occurrences of the image features which are
diagnostic-type terms or descriptions of action/scenario
surrounding what is seen
Fracture, bowel obstruction, perforation, other
diagnosis/pathology judgments
experts’ image-needs descriptions. The occurrence of image
attributes was examined at 10 levels of a medical-image
entity. The current study captured details about radiological
professionals and preprofessionals’ preferences in accessing
medical images. Specific findings are discussed next.
RQ1: Which Levels of Image Attributes Are Used by
Medical Image Users When They Intend to Search
Medical Images?
To answer the first research question, descriptive statis-
tical analyses were used to display a numerical overview
of the levels/classes used by three groups of participants,
respectively.
Table 3 shows that image descriptions produced by
novices, intermediates, and experts contain image attributes
at all levels of the pyramid model; that is, image attributes
at all levels of the pyramid model were used by all partic-
ipants. This is in line with Jörgensen et al.’s (2001) study,
which states that the pyramid model is robust enough to
accommodate a variety of attributes. In addition, the results
showed that:
•In expert descriptions, the top-five most-often-used image
levels ranked: Specific Objects (33.54%), Specific Scene
(21.52%), Generic Scene (13.29%), Abstract Scene (13.13%),
and Type/Technique (6.96%).
•In intermediate descriptions, the five most-often-used image
levels ranked identically to those of experts: Specific Objects
(38.36%), Specific Scene (23.06%), Generic Scene (10.90%),
Abstract Scene (9.22%), and Type/Technique (5.66%).
•As for novice descriptions, the five most-often-used image
levels were all semantic levels: Specific Objects (37.15%),
Specific Scene (24.50%), Generic Scene (10.04%), Abstract
Scene (6.43%), and Generic Objects (6.22%).
The results indicated that all participants (novices, inter-
mediates, and experts) preferred to use image attributes of the
semantic levels (over 85% of all image attributes) to represent
the image that they desired to find. This finding is in agree-
ment with Hollink et al.’s (2004) study. They found that 85%
233
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—February 2012
DOI: 10.1002/asi
TABLE 3. Image attributes identified in image descriptions of experts, intermediates, and novices.
Image attributes in novices’ Image attributes in intermediates’ Image attributes in experts’
descriptions descriptions descriptions
Pyramid levels Occurrence % Occurrence % Occurrence %
Syntactic levels
1. Technique (Type) 29 5.82 27 5.66 44 6.96
2. Global Distribution 14 2.81 7 1.47 7 1.11
3. Local Structure 9 1.81 9 1.89 15 2.37
4. Global composition 22 4.42 6 1.26 10 1.58
Subtotal 74 14.86 49 10.28 76 12.02
Semantic levels
5. Generic Objects 31 6.22 22 4.61 14 2.22
6. Generic Scene 50 10.04 52 10.9 84 13.29
7. Specific Objects 185 37.15 183 38.3 212 33.54
8. Specific Scene 122 24.50 110 23.0 136 21.52
9. Abstract Objects 4 0.80 17 3.56 27 4.27
10. Abstract Scene 32 6.43 44 9.22 83 13.13
Subtotal 424 85.14 428 89.59 556 87.97
Total 498 100.00 477 100.00 632 100.00
Note. Bold numbers indicate the top-five most frequently used classes of image attributes.
of the image descriptions were conceptual levels.Among the
five most-often used image levels, the Type/Technique level
was the sole Syntactic Level that was frequently used in expert
(6.96%) and intermediate (5.66%) descriptions.
Another interesting finding was that scene descriptions
were used more than two times as much as object descrip-
tions in expert and intermediate descriptions. This finding
opposed Hollink et al.’s (2004) finding, that object descrip-
tions are used twice as much as are scene descriptions (p.
623). This difference revealed that the profession of radiol-
ogy technology is not only keen to objects in an image such
as medical devices, body parts, and patient types but also
pays considerable attention to various situations/scenarios
such as positions of patients and conditions surrounding the
objects. In fact, the latter requires more domain knowledge
in the profession of radiography. Thus, it is not surprising
to see that experts and intermediates employed many scene-
related image attributes in their descriptions while novices
used almost the same number of both object-related and
scene-related image attributes.
A comparison of our results with the findings of previ-
ous studies showed both similarities and differences. Our
study found that the specific levels are the most frequently
used levels (including specific objects and specific scenes),
which is similar to the study of Armitage and Enser (1997).
They studied seven libraries with collections containing pho-
tos and films about geography, television, and local history,
and found that specific levels were the most frequently used
levels in image-users’search activities. However, our findings
were different from those of Hollink et al.’s (2004) study of
children and novel domains, in which the most frequently
used levels of description were general. A possible explana-
tion of this difference is that image searchers prefer using
image attributes that are as specific as possible to accurately
and efficiently locate images in fact-oriented domains (e.g.,
geography, history, news, radiology). This reflects that speci-
ficity enhancement is to be expected from an image-retrieval
system or from library or archive staff.
RQ2: Do Domain Experts, Intermediates, and Novices
Differ Significantly in Terms of Employing Image
Attributes at the Syntactic Levels in Their Image
Descriptions? What Are the Differences?
To answer the second research question, a multivari-
ate analysis of variance (MANOVA) was conducted, with
domain knowledge as an independent variable and four
syntactic levels of image attributes as dependent vari-
ables (Type/Technique, Global Distribution, Local Structure,
and Global Composition) to examine whether the levels
of domain knowledge influence the use of syntactic lev-
els of image features. Overall, there was a nonsignificant
main effect of domain knowledge on the use of syntactic-
level image attributes using Pillai’s trace, V=0.064,
F(8, 470) =1.951, p=0.051. Figure 4 shows the means
and standard errors for the number of syntactic-level image
attributes from the three user groups.
Although we did not find significant differences across the
three groups of users in terms of employing image attributes
in their descriptions, we identified some intriguing patterns.
First, we found that image attributes of the Global Distribu-
tion (2nd) level and Global Composition (4th) level occurred
more in the novices’ image descriptions than in those of
the intermediates and the experts. Some typical examples
of novices’ descriptions at the Global Distribution (2nd)
level include: “adequate penetration,” “the overall grayness,”
234 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—February 2012
DOI: 10.1002/asi
FIG. 4. The effect of domain knowledge on image attributes of syntactic levels.
“blurry” and so on. The examples of novices’ descriptions
at the Global Composition (4th) level were “symmetrical,”
“poorly collimated,” “centered,” “aligned,” and so on. These
identified differences were not statistically significant but
they suggest that novices were more interested in gross char-
acteristics of medical imagery than were intermediate and
expert participants. Perhaps novices did not possess adequate
knowledge to recognize, analyze, or evaluate details in local
regions of medical imagery. In addition, it is possible that
novices had not developed sufficient automaticity at the lower
levels of image understanding to free cognitive load for the
analysis of higher level features.
Another interesting pattern was that image attributes at
the Type/Technique (1st level) and the Local Structure (3rd)
occurred more in experts’image descriptions than they did in
those of intermediates and novices. These identified differ-
ences were not statistically significant. In ordinary conditions
(nonprofessional judgment on modalities), Type/Technique
is at the lowest level of the pyramid, which means that
the least amount of knowledge is required to describe it.
However, Jaimes and Chang (2000) noted an exceptional
situation of employing image attributes at this level:
An average observer, for example, may not be able to deter-
mine the technique that was used to produce a painting—but
an expert in art would be able to determine exactly what was
used. Indexing in this particular case would require more
knowledge at the type/technique level than at the generic
objects level, since special knowledge about art technique
would be needed.” (para. 3.1.11)
In our study, the experts were professional radiological
technologists, whose area of expertise was the production
of radiographs of parts of the human body for use in med-
ical diagnostics; therefore, it is not surprising to see more
Type/Technique-related attributes in the experts’ descrip-
tions. In the profession of radiography, the capability to
differentiate the modalities of various genres requires a cer-
tain amount of domain knowledge. With regard to the Local
Structure (3rd) level, we found that image attributes related
to “marker placement” occurred more in experts’ descrip-
tions than they did in those of intermediates and novices.
As Jaimes and Chang (2000) addressed, in x-rays and micro-
scopic images, there is often a strong concern for local details
such as “dark spots” in an x-ray and “lines in microscopic
image.” While the marker present on medical images conveys
information about the side of the body imaged, the radiologic
technologist is accustomed to using the marker to convey
meaning about patient position and lateral orientation to the
radiologist. Thus, it is understandable that they preferred to
use marker information (i.e., identifiable features) to search
for a desired image.
RQ3: Do Domain Experts, Intermediates, and Novices
Differ Significantly in Terms of Employing Image
Features at the Semantic Levels in Their Image
Descriptions? What Are the Differences?
Figure 5 displays means and standard errors for the
number of semantic-level image attributes occurring in
235
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—February 2012
DOI: 10.1002/asi
FIG. 5. The effect of domain knowledge on image attributes of semantic levels.
the three user groups. Image attributes at the Generic
Objects (5th) level (M=0.397, SE =0.061) occurred most
in novices’ image descriptions, followed by those of inter-
mediates (M=0.306, SE =0.063) and experts (M=0.156,
SE =0.056). Image attributes at the Generic Scene (6th) level
occurred most in experts’ image descriptions (M=0.933,
SE =0.087), followed by those of intermediates (M=0.722,
SE =0.097) and novices (M=0.641, SE =0.094). Image
attributes of Specific Objects (7th) level occurred most in
intermediates’ image descriptions (M=2.542, SE =0.0180),
followed by those of novices (M=2.372, SE =0.173)
and experts (M=2.356, SE =0.161). Image attributes of
the Specific Scene (8th) level occurred most in interme-
diates’ image descriptions (M=1.564, SE =0.174), fol-
lowed by those of experts (M=1.528, SE =0.181) and
novices (M=1.511, SE =0.162). Image attributes at the
Abstract Objects (9th) level occurred most in image
descriptions of experts (M=0.300, SE =0.048), followed
by those of intermediates (M=0.236, SE =0.054) and
novices (M=0.051, SE =0.052). Image attributes of the
Abstract Scene (10th) level occurred most in experts’
image descriptions (M=0.922, SE =0.090), followed by
those of intermediates (M=0.611, SE =0.101) and novices
(M=0.410, SE =0.097).
To answer the third research question, a MANOVA
analysis was conducted, with domain knowledge as an
independent variable and the six semantic levels of image
attributes as dependent variables to compare the effect
of domain knowledge on the use of semantic levels of
image attributes. Overall, there was a significant main effect
of domain knowledge on the use of semantic-level image
attributes using Pillai’s trace,V=0.161, F(12, 466) =3.409,
p=0.000. Follow-up univariate ANOVAs found significant
differences on three dependent variables: the Generic Objects
(5th) level, F(2, 237) =68.148, p<0.05, partial η2=0.036;
the Abstract Objects (9th) level, F(2, 237) =2.821, p<0.05,
partial η2=0.052; and the Abstract Scene (10th) level,
F(2, 230) =7.713, p<0.05, η2=0.061. Post hoc analysis
using the Games–Howell test found the following significant
differences:
•Novices employed significantly more image attributes of
Generic Objects than did experts. Descriptions of Generic
Objects involve only everyday knowledge and do not require
specific or specialized knowledge in descriptions, which may
explain this difference. The novices in this study were at
the beginning stage of learning diagnostic imaging; they
gained only some fundamental knowledge in radiography
and possibly did not possess adequate knowledge to rec-
ognize or name specific objects or higher level concepts.
Thus, novices relied more on describing those generic objects
such as “arms, legs, lower limb” in their image description
than did experts and intermediates.
•Experts and intermediates employed significantly more
image attributes of Abstract Objects than did novices.
This suggests that these two groups were more detailed
at summarizing the information gleaned from the image
and made interpretations based on the relationship of
various image attributes. Experts frequently inferred the
age and gender of the patient based on features in
an image.
•Experts employed significantly more image attributes of
Abstract Scenes than did novices. This finding indicates that
experts showed stronger intention to make diagnostic judg-
ments based on the meaning and element arrangements in an
image than did novices.
236 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—February 2012
DOI: 10.1002/asi
In addition, we found that domain knowledge does
not show an effect on three semantic levels of image
attributes: the Generic Scene (6th) level, F(2, 237) =2.821,
p=0.062, partial η2=0.023; the Specific Object level, F(2,
237) =0.348, p=0.706, partial η2=0.003; and the Spe-
cific Scene (8th) level, F(2, 230) =2.071, p=0.975, partial
η2=0.000. This suggests that the occurrence of image
attributes in image description at the Generic Scene (6th)
level, the Specific Object (7th) Level, and the Specific Scene
(8th) level are similar across novices, intermediates, and
experts. Furthermore, Table 3 shows that image attributes
at these three levels were most frequently used in all three
user-groups’image description. Thus, this nonsignificant sta-
tistical difference between three user groups suggests that all
three groups preferred to access images based on patient posi-
tion and/or projection features (Generic Scene level), con-
crete concepts and salient features (Specific Object level), and
features regarding the image quality and scenarios (Specific
Scene level) to represent their image-information needs.
Some current indexing practices focus primarily on the
needs and interests of domain experts. For instance, to
develop a novel multimodal algorithm for modality clas-
sification, Kalpathy-Cramer and Hersh (2010) surveyed 37
clinicians to determine their met and unmet needs in terms
of using a variety of medical and general-purpose image-
retrieval systems. Besides domain experts, the needs and uses
of nonprofessionals such as medical students and trainees
also should be considered to customize query processing for
different types of users. The findings of this study reveal the
preferred image levels employed by domain experts, interme-
diates, and novices. Multifaceted indexing procedures need
to be developed to match the interests of heterogeneous
categories of users. It also is important to provide correspond-
ing access points so that members of different user groups
can find desired images using their preferred approaches to
searching.
Limitations and Future Research
First, our study focused on one image genre, x-ray images,
and used a sample of these images to elicit participants’ image
descriptions. Thus, these results may not be generalizable to
other image genres such as pathological or dermatological
images. Second, our study provides empirical evidence to
show that domain knowledge is a factor that impacts how
medical-image users describe their image-information needs.
Future research may include more diverse user groups in
health science professions such as physicians and radiolo-
gists to verify this finding. Third, the simplified operational
definition of domain knowledge (i.e., domain knowledge
defined based on user-groups’ credentials and experiences)
may not have precisely measured the participants’ knowl-
edge level. In the future, more fine-grained or standardized
tests of domain knowledge may be employed. Finally, this
study focused on cross-group comparisons in terms of the
use of image attributes among novices, intermediates, and
experts at different levels; however, this type of compari-
son did not reveal the relationships among high-level and
low-level image attributes. Future studies should focus on
this issue and discover the link between high- and low-level
image attributes.
Conclusion
The topic of image indexing and retrieval in medical
applications has been widely discussed in the literature
(e.g., Avni, Greenspan, Konen, Sharon, & Goldberger, 2011;
Scholl, Aach, Deserno, & Kuhlen, 2011; Smeulders,Worring,
Santini, Gupta, & Jain, 2000), but few empirical studies have
investigated users’preferred indexing procedures. This study
examined 240 medical-image descriptions produced by 40
image users in the area of radiography. Four main findings
were identified.
First, image attributes generated by medical-image users
can be mapped to all levels of the pyramid model. This sug-
gests that the pyramid model is not only a suitable indexing
tool for general purpose images but also may be a robust
instrument for indexing medical imagery. Thus, developers
of medical-image systems may use the pyramid model to
structure medical visual information of medical imagery.
Second, the results indicate that all participants (novices,
intermediates, and experts) preferred to use image attributes
of the semantic levels to represent the image that they
desired to find, especially specific-level and scene-related
attributes. Therefore, providing access points to images at
specific levels (objects and scenes) and at scene-related
levels (generic, specific, and abstract scenes) could be bene-
ficial to medical-image users with different levels of domain
knowledge.
Third, Type/Technique is the only syntactic-level attribute
that occurred often in expert and intermediate groups.
Only experts showed higher interest in describing the
Type/Technique level (6.96% of all image attributes they
generated). This suggests that in the medical field, the
Type/Technique level should not be considered the lowest
level which does not require much knowledge. On the con-
trary, to recognize the genre of a medical image accurately
requires a certain amount of special training and knowl-
edge. Thus, when designing medical-image retrieval systems,
developers need to carefully consider whether novices are
capable of accessing an image with image attributes of
Type/Technique.
The main finding of this study is that the effect of domain
knowledge has a significant relationship with the use of
semantic image attributes in image-users’descriptions. More
specifically, our study found that experts were more likely
to employ high-level image attributes which require high-
reasoning or diagnostic knowledge to search for a medical
image (Abstract Objects and Scenes); novices were more
likely to describe some basic objects which do not require
much radiological knowledge to search for an image they
need (Generic Objects). The question of “what should be
indexed” can be better answered by discovering different
237
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—February 2012
DOI: 10.1002/asi
users’ preferred indexing levels. These preferences in turn
will help to enhance the efficiency and effectiveness of
medical-image retrieval systems. These differences in image
description could be used to develop “user models” for
information retrieval systems that are adaptable to users’char-
acteristics and preferences. In addition, the results of this
study also will be beneficial for the creation of adaptive and
supportive tool sets such as customized thesauri, browsing
functions, and customized interfaces that are appropriate for
particular user groups.
Acknowledgments
This project was supported by National Science Founda-
tion Grant IIS-0812515. We also thank those who facilitated
our study and all participants who provided us with the
opportunity to interview them.
References
Armitage, L.H., & Enser, P. (1997). Analysis of user need in image archives.
Journal of Information Science, 23(4), 287–299.
Avni, U., Greenspan, H., Konen, E., Sharon, M., & Goldberger, J. (2011).
X-ray categorization and retrieval on the organ and pathology level, using
patch-based visual words. IEEE Transactions on Medical Imaging, 30(3),
733–746.
Borgman, C.L. (1996). Why are online catalogs still hard to use? Journal of
the American Society for Information Science, 47(7), 493–503.
Chen, H.-L. (2001). An analysis of image queries in the field of art history.
Journal of the American Society for Information Science and Technology,
52(3), 260–273.
Chen, H.-L., & Rasmussen, E.M. (1999). Intellectual access to images.
Library Trends, 48(2), 291–302.
Choi, Y., & Rasmussen, E.M. (2003). Searching for images: The analy-
sis of users’ queries for image retrieval in American history. Journal of
the American Society for Information Science and Technology, 54(6),
498–511.
Chu, H. (2001). Research in image indexing and retrieval as reflected in the
literature. Journal of the American Society for Information Science and
Technology, 52(12), 1011–1018.
Cleveland, A.D., & Cleveland, D.B. (2009). Health informatics for medical
librarians. New York, NY: Neal-Schuman.
Dawes, T.J.W., Vowler, S.L.,Allen, C.M.C., & Dixon, A.K. (2004). Training
improves medical student performance in image interpretation. British
Journal of Radiology, 77(921), 775–776.
Drabenstott, K.M. (2003). Do nondomain experts enlist the strategies of
domain experts? Journal of the American Society for Information Science
and Technology, 54(9), 836–854.
Eakins, J.P. (2002). Towards intelligent image retrieval. Pattern Recognition,
35(1), 3–14.
Enser, P. (2000). Visual image retrieval: Seeking the alliance of concept-
based and content-based paradigms. Journal of Information Science,
26(4), 199–210.
Enser, P.G.B., & McGregor, C. (1992). Analysis of visual information
retrieval queries. British Library Research and Development Report.
London, United Kingdom: British Library.
Fidel, R. (1994). User-centered indexing. Journal of the American Society
for Information Science, 45(8), 572–576.
Goodrum, A.A. (2005). I can’t tell you what I want, but I’ll know it when I
see it. Reference & User Services Quarterly, 45(1), 46–53.
Goodrum, A.A., & Spink, A. (2001). Image searching on the Excite
Web search engine. Information Processing & Management, 37(2),
295–312.
Hollink, L., Schreiber, A.T., Wielinga, B.J., & Worring, M. (2004).
Classification of user image descriptions. International Journal of
Human–Computer Studies, 61(5), 601–626.
Hsieh-Yee, I. (1993). Effects of search experience and subject knowledge
on the search tactics of novice and experienced searchers. Journal of the
American Society for Information Science, 44(3), 161–174.
Jaimes, A. (2006). Human factors in automatic image retrieval system
design and evaluaition. In Proceedings of SPIE-The International Society
for Optical Engineering Internet Imaging VII (pp. 1–9). San José, CA:
IS&T/SPIE.
Jaimes, A., & Chang, S.-F. (2000). A conceptual framework for indexing
visual information at multiple levels. IS&T/SPIE Internet Imaging. San
José, CA: IS&T/SPIE.
Jörgensen, C. (1995). Image attibutes: An investigation (indexing systems,
retrieval systems, computerized). Dissertation Abstract International,
57(4), 1364A. (UMI No. AAT 9625856). Retrieved from Dissertation and
Theses database.
Jörgensen, C. (1998). Attributes of images in describing tasks. Information
Processing & Management, 34(2–3), 161–174.
Jörgensen, C., Jaimes, A., Benitez, A. B., & Shih-Fu, C. (2001). A concep-
tual framework and empirical research for classifying visual descriptors.
Journal of the American Society for Information Science & Technology,
52(11), 938–947.
Kalpathy-Cramer, J., & Hersh, W. (2010). Multimodal medical image
retrieval: Image categorization to improve search precision. In Proceed-
ings of the International Conference on Multimedia Information Retrieval
(pp. 165–173). New York, NY: ACM Press.
Kammerer, F.J., Frankewitsch, T., & Prokosch, H.U. (2009). Design of a web
portal for interdisciplinary image retrieval from multiple online image
resources. Methods of Information in Medicine, 48(4), 361–370.
Layne, S.S., & Lunin, L.F. (1994). Some issues in the indexing of images.
Journal of the American Society for Information Science, 45(8), 583–588.
Liu, Y., Zhang, D., Lu, G., & Ma, W.Y. (2007). A survey of content-based
image retrieval with high-level semantics. Pattern Recognition, 40(1),
262–282.
Marchionini, G., Dwiggins, S., Katz, A., & Lin, X. (1993). Information
seeking in full-text end-user-oriented search systems: The roles of domain
and search expertise. Library and Information Science Research, 15(1),
35–69.
McCay-Peet, L., & Toms, E. (2009). Image use within the work task model:
Images as information and illustration. Journal of the American Society
for Information Science and Technology, 60(12), 2416–2429.
Müller, H., Michoux, N., Bandon, D., & Geissbuhler, A. (2004). A review
of content-based image retrieval systems in medical applications—
Clinical benefits and future directions. International Journal of Medical
Informatics, 73(1), 1–23.
Müller, H., Zhou, X., Depeursinge, A., Pitkanen, M., Iavindrasana, J., &
Geissbuhler, A. (2007). Medical visual information retrieval: State of
the art and challenges ahead. Proceedings of 2007 IEEE International
Conference on Multimedia and Expo (pp. 683–686). Beijing: Institute of
Electrical and Electronics Engineers.
O’Connor, B.C. (1999). User reactions as access mechanism: An explo-
ration based on captions for images. Journal of the American Society for
Information Science, 50(8), 681–695.
Panofsky, E. (1962). Studies in iconology. New York, NY: Harper & Row.
Pasveer, B. (2006). Representing or mediating:A history and philosophy of
x-ray images in medicine. In L. Pauwels (Ed.), Visual cultures of science:
Rethinking representational practices in knowledge building and sceince
communication (pp. 41–62). Lebanon, NH: Dartmouth College Press–
University Press of New England.
Pauwels, L. (2006). A theoretical framework for assessing visual repre-
sentational practices in knowledge building and science communication.
In L. Pauwels (Ed.), Visual cultures of science: Rethinking represen-
tational practices in knowledge building and science communication
(pp. 1–25). Lebanon, NH: Dartmouth College Press–University Press of
New England.
Rasmussen, E.M. (1998). Indexing images. Annual Review of Information
Science and Technology, 32, 169–196.
238 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—February 2012
DOI: 10.1002/asi
Roberts, H.E. (2001). A picture is worth a thousand words: Art indexing
in electronic databases. Journal of the American Society for Information
Science and Technology, 52(11), 911–916.
Rorissa, A. (2008). User-generated descriptions of individual images ver-
sus labels of groups of images: A comparison using basic level theory.
Information Processing & Management, 44(5), 1741–1753.
Scholl, I., Aach, T., Deserno, T.M., & Kuhlen, T. (2011). Challenges of med-
ical image processing. Computer Science–Research and Development,
26(1–2), 5–13.
Shatford, S. (1986). Analyzing the subject of a picture: A theoretical
approach. Cataloging & Classification Quarterly, 6(3), 39–62.
Shute, S.J., & Smith, P.J. (1993). Knowledge-based search tactics. Informa-
tion Processing & Management, 29(1), 29–45.
Sihvonen, A., &Vakkari, P. (2004). Subject knowledge improves interactive
query expansion assisted by a thesaurus. Journal of Documentation, 60(6),
673–690.
Smeulders, A.W.M., Worring, M., Santini, S., Gupta, A., & Jain, R.
(2000). Content-based image retrieval at the end of the early years.
IEEE Transactions on PatternAnalysis and Machine Intelligence, 22(12),
1349–1380.
Tang,L.H.Y.,Hanka, R., & Ip, H.H.S. (1999). A review of intelligent content-
based indexing and browsing of medical images. Health Informatics
Journal, 5, 40–46.
Vakkari, P. (2002). Subject knowledge, source of terms, and term selection
in query expansion: An analytical study. In F. Crestani, M. Girolami, &
C. van Rijsbergen (Eds.), ECIR 2002 (pp. 101–106). Lecture Notes in
Computer Science 2291. Berlin/Heidelberg: Springer.
Vibert, N., Ros, C., Le Bigot, L., Ramond, M., Gatefin, J., & Rouet, J.-F.
(2009). Effects of domain knowledgeon reference search with the PubMed
database: An experimental study. Journal of the American Society for
Information Science and Technology, 60(7), 1423–1447.
Wang, X. (2011). [Doctoral Dissertation study]. Unpublished raw data.
Yoon, J.W., & O’Connor, B. (2010). Engineering an image-browsing
environment: Re-purposing existing denotative descriptors. Journal of
Documentation, 66(5), 750–774.
239
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—February 2012
DOI: 10.1002/asi
Appendix A
Guidelines of Splitting Descriptions into Fragments
No. Situations Rules Example
1 Adjectives & Adverbs 1.1: If they are image attributes (e.g., color, shape,
texture, action), they are classified separately with
the accompanying nouns or verbs.
1. an AP/abdominal film;
2. a/ bowel/obstruction/
1.2: If they are NOT image attributes, they are classified
together with the accompanying nouns or verbs.
Left hip; right shoulder
2 Words specifying an
amount or degree
2.1: They are not classified separately. The singular and
plural words are not differentiated in terms of
segmentation in this study.
an, numerals, a lot, little, very, badly, too, throughout,
enough
3 Conjunctions (e.g., and,
or, but)
3.1: Ideas joined by a conjunction are classified
separately.
1. Nearly emptied of contrast,/but some still remains.
2. Artifact throughout/and the shoulders/ were not
depressed
4 Verbs 4.1: Verbs are separate elements including different
forms (e.g., is rotated, is taken, laying).
1. The patient should/have been positioned/laying on
their back/.)
4.2: A verb and a noun are classified as one element if
the combination describes a specific, commonly
occurring action.
5 Indefinite words 5.1: Indefinite words such as someone, somewhere, and
anything are NOT classified as separate elements.
Image, or shot vs. radiograph or CT
6. Prepositional phrases 6.1: They are classified as one unit. /with a lot of gas/
6.2: Exception: The adjectives within the phrase
constitute separate image attributes.
with/female/lower extremity/images
7 Words that are linked by
“of” (possessive)
7.1: They are classified as one. lumbar spine of a patient
7.2: Exceptions: Two image attributesare linked by “of.” /by/adequate penetration/of/ the vertebrae/.
8 Linking verbs (e.g., am,
is, seem, appear)
8.1: If there are no image attributes included in the
sentence, then we keep the sentence as a whole unit.
The image is slightly cut off.
8.2: If the subject or the object includes image attributes,
we separate them from the rest of the words.
1. Fecal matter/is evident.
2. This is/an image of the vertebral column/.
3. The projection is /AP/.
9 Parentheses ( ) 9.1: If explanations are included in parentheses ( ), the
content in the parentheses are considered as a
whole unit. If more than one attribute is included
in the parentheses, then finer segmentations are
needed.
a contrast study/(either IVU/or Cystogram)
10 Contextual words 10.1: If a phrase describes context which is helpful for
understanding the meaning of a segment, it is NOT
classified a segment. If the contextual words
do not help with the understanding of the segment,
they are classified separately. Individual segments
should be meaningful. The example “I believe
this is” does not add information necessary for
understanding the adjacent segment and is
therefore classified separately.
1. I believe this is /a supine/KUB/radiograph/.
This is /an image of vertebral column/).
Appendix B
Instructions
Describing task (six images).
(Students): Suppose an image database at the Radiography
program is available for students searching for references as
illustrations in their projects. You need to find an image similar
to the one projected on the screen. How would you describe this
image that you intend to locate? Please use complete sentences
or longer phrases for your descriptions.
(Experts): Suppose you are preparing training for fellow tech-
nologists and need to find an image similar to this one from an
image database at your hospital.
Please use complete sentences or longer phrases for your
descriptions.
Appendix C
Sample Image Description Terms Generated by Medical-Image Users
Pyramid Levels ImageA Terms Image B Terms Image C Terms Image D Terms Image E Terms Image F Terms
(1) Type/Technique •radiograph •radiograph •x-ray
•radiograph
•radiograph •fluoroscopic image
•fluoroscopy procedure
•hysterosalanglogram
•radiograph
(2) Global Distribution •The technique looks
good.
•good techniques •cloudy •adequate technique •overall grayness to the
image
•It is very white down by
the elbow.
(3) Local Structure •left marker •a marker •heart margins •the finer marking of the
vertebrae
•some changes of the bone
•circular
•marker placement
•soggy looking bone
•nicely collimated film
•with jagged edges
(4) Global Composition •poorly collimated
•good centering
•asymmetry •not symmetrical
•bent up
•not centered
•not aligned
•poor collimation
•on 14 ×7 (size of the
image)
•anatomy centered
•at about L5–S1 joint •Collimation should be
present.
•clear anatomy
•misaligned
(5) Generic Objects •lower extremity
•the leg
•lower limb
•a few artifacts
•larger person
•belly
•artifacts
•several wires
•vertebral column
•spine
•bladder shot
•a contrast study
•right arm
•two fragments
(6) Generic Scene •AP
•hip projection
•in frog position
•laying supine
•supine •in the semi-upright position
•portable chest image
•PA
•AP
•not oblique
•an AP view
(7) Specific Objects •hip
•femur
•trochanter
•proximal femur
•abdomen
•KUB
•lots of gas
•symphysis pubis
•intestines
•chest
•costophrenic angles
•lungs
•elevated shoulders
•clavicle
•intervertebral joint spaces
•lumbar spine
•bladder
•bladder
•voiding cystogram
•a catheter
•sacrum
•humerus
•mid-humerus shaft
•growth plates
•immobilizing device
(8) Specific Scene •rotation
•in medially
•rotated externally
•enough diaphragm
•dispersed throughout
the entire image
•slightly rotated
•through a KUB
•not enough x-ray
penetration
•the shoulders forward and
depressed
•Angle is clipped.
•in the lumber
•lateral curvature
•in the bladder
•into the lower spine
•Elbow is cut off.
•in a cast
(9) Abstract Object •asthenic patient size
•a child’s picture
•adult •adult patient
•male
•older patient
•The patient appears to be
female.
•Patient is female.
•a female
•pediatric patient
•a child
(10) Abstract Scene •fracture
•broken
•arthritis present
•left fracture
•obstruction
•bowel obstruction
•bowel distension
•The heart is normal size. •with torsional scoliosis
•bone spurs
•some changes in the bone
•severe degeneration
changes
•previous surgery
•gallbladder removal
•blockage of the uterine tube
•a leak
•with mid shaft break
•bone growth
•shaft fracture
•Mid-humerus on the right
side are misaligned.
•lateral displacement
•spiral fracture
•clear clean break
241
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—February 2012
DOI: 10.1002/asi
Appendix D
Six images presented to participants
