ChapterPDF Available

Communication by Chemical Signals: Physiological Mechanisms, Ontogeny and Learning, Function, Evolution, and Cognition


Abstract and Figures

A chemical signal, such as a scent mark, is likely to provide unique and overlapping information about the sender, including its phenotype and genotype, which can be individually distinctive. Thus, the response to scent marks should be context-dependent, allowing receivers to adjust their responses accordingly, depending on the identity of the sender, past associations, and context. The response chosen will, in turn, be modulated by the receiver's neuroendocrine system and cognitive ability. The particular response should represent a balance of the costs and benefits associated with that choice and consider the fact that chemical signals provide inadvertent public information, reflecting the selective pressures placed on participants.
Content may be subject to copyright.
Provided for non-commercial research and educational use only.
Not for reproduction, distribution or commercial use.
This article was originally published in the book Hormones, Brain, and Behavior, 3rd
Edition, published by Elsevier, and the attached copy is provided by Elsevier for the
author's benefit and for the benefit of the author's institution, for non-commercial
research and educational use including without limitation use in instruction at your
institution, sending it to specific colleagues who you know, and providing a copy to your
institution's administrator.
All other uses, reproduction and distribution, including without limitation
commercial reprints, selling or licensing copies or access, or posting on open
internet sites, your personal or institution’s website or repository, are
prohibited. For exceptions, permission may be sought for such use through
Elsevier’s permissions site at:
From Ferkin, M.H., delBarco-Trillo, J., Petrulis, A. Communication by Chemical Signals:
Physiological Mechanisms, Ontogeny and Learning, Function, Evolution, and Cognition. In:
Pfaff, D.W and Joëls, M. (editors-in-chief), Hormones, Brain, and Behavior 3rd
edition, Vol 1. Oxford: Academic Press; 2017. pp. 285–327.
Copyright © 2017 Elsevier Inc. All rights reserved.
Academic Press
Author's personal copy
1.10 Communication by Chemical Signals: Physiological Mechanisms,
Ontogeny and Learning, Function, Evolution, and Cognition
MH Ferkin, University of Memphis, Memphis, TN, USA
J delBarco-Trillo, Liverpool John Moores University, Liverpool, UK
A Petrulis, Georgia State University, Atlanta, GA, USA
Ó2017 Elsevier Inc. All rights reserved.
This chapter is a revision of the previous edition chapter by R.E. Johnston, J. delBarco-Trillo, volume 1, pp. 395441, Ó2009, Elsevier Inc.
1.10.1 Introduction 286 Brief Overview of Odor Cues and Scent Marking 286 Sources of Scents and Odors 287 The Attractiveness of Scent Marks and Conspecic Odor Preferences 287
1.10.2 Physiological Mechanisms 289 Neural Substrates 289 Mechanisms of Communication 289 Scent Marking 290 Ultrasonic Vocalizations 292
1.10.3 Development and Learning 293 Reproductive States and Odors 293 Individual Discrimination 294 HabituationDishabituation 295 Two Primary Polymorphic and Multigenic Complexes 296 Kin Recognition 297 Mechanisms of Kin Recognition 297 Group Discrimination 298 Species Discrimination: Competitors and Predators 298 Sex Identication 299 Endocrine Effects 300 Social Effects 300 Role of Odors in Social Dominance and Aggression 300 Food and Odors: Diet and Food Deprivation 302 Effect of Health and Environmental Stressors on Odors 303 Age and Odors: From Infancy to Senescence 303
1.10.4 Functions of Odor Communication 304 Scent Marking 304 Hypotheses about Function 304 Over-marking 306 Top- and Bottom-Scent Donors Are Same-Sex Conspecics 306 Mixed-Sex Over-marks and the Effects of Familiarity 307 Ten Hypotheses about the Function of Over-marks 309 An Over-mark versus a Single Scent Mark 310 Self-Grooming as a Form of Scent Marking 311 Mate Attraction 311 Scent Marks Provide Public Information 311
1.10.5 Evolution 312 Basic Concepts 312 Comparative Approaches to the Study of Olfactory Communication 312 Genetic Variation, Natural Selection, and Odors 313 Odors and Sexual Selection 313
1.10.6 Cognition and the Assessment of Scent Marks 314 Processing Scent Marks 315 Cognitive Ability, Numerosity, and Scent Marks 315
1.10.7 Concluding Thoughts 316
Acknowledgments 316
References 316
Hormones, Brain, and Behavior, 3rd edition, Volume 1 285
Hormones, Brain and Behavior, Third Edition, 2017, 285–327
Author's personal copy
1.10.1 Introduction
This chapter focuses on scent marking and odor communica-
tion. We provide examples of odor communication in terres-
trial vertebrates but focus most of our efforts on mammals,
the class of vertebrates most familiar to the authors. Such an
endeavor was previously carried out in a text by Brown and
Macdonald (1985) and by Wyatt (2014). Because of space limi-
tations in this chapter our focus will be less comprehensive.
However, much of the chapter will focus on scent marking
and, to a lesser extent, on other forms of olfactory communica-
tion such as self-grooming in terrestrial mammals, which are
excellent subjects for the study of such topics. This is because
olfaction is a major sensory modality for terrestrial mammals,
most of which display scent marking (Biben, 1980;Brashares
and Arcese, 1999;Drea, 2015;Ferkin, 2015;Hurst, 1990a,b,c;
Heymann, 1998;Johnson, 1973;Johnston and delBarco-
Trillo, 2009;Klailova and Lee, 2014;Lewis, 2005;Macdonald,
1980;McClintock, 2002;Sliwa and Richardson, 1998).
To understand odor communication in terrestrial
mammals, it is necessary to differentiate between studies
that examine proximate causation and ultimate consequences
of scent marking and then integrate the ndings of these
disparate studies into a comprehensive picture of this
behavior. In this chapter, our approach is to discuss advances
and controversies in odor communication among terrestrial
mammals. Our approach is based on Tinbergens (1963)
four levels of behavioral analysis. Specically, we categorized
and discussed studies that deal with proximate causation,
such as the physiological mechanisms that underlie and
mediate odor communication and the role of ontogeny and
learning in the development of odor communication. We
also identify and discuss studies that pertain to ultimate
consequences, such as the function and adaptive signicance
of odor communication, the benets and costs of scent
marking in terms of survival and tness, and its evolution in
terrestrial mammals. We also discuss the role of cognition in
odor communication. We selected papers that are recent
and provide new information and detail and also classics,
many of which provided the initial hypotheses and theoretical
framework for current researchers studying odor communica-
tion in terrestrial mammals. Brief Overview of Odor Cues and Scent Marking
In most terrestrial mammals scent marks are typically depos-
ited on prominent objects or along paths that are shared with
conspecics (Brown and Macdonald, 1985;Clapham et al.,
2013;Sharpe, 2015). Therefore, individuals will enter areas
that contain single scent marks and overlapping scent marks
(over-marks) which are comprised of scent marks from two
or more conspecics (Ferkin et al., 2011a,b). These scent
marks may be a signal that provides particular information
about the donors to individuals that encounter such marks.
These, in turn, may affect how the receivers respond to the
donors (Wyatt, 2014). Typically, individuals spend more
time investigating the scent marks of opposite-sex conspe-
cics than of same-sex conspecics (Ferkin and Seamon,
1987). The response of individuals to scent marks as well
as the location where individuals deposit their own scent
marks may affect subsequent interactions with conspecics.
Presumably, individuals may spend more time investigating
the scent marks of more attractive than of less attractive
opposite-sex conspecics. Similarly, individuals in better
physical condition may scent mark and over-mark more
often than would individuals in worse condition (Ferkin,
2011,2015;Hurst and Beynon, 2004;Johnston and
delBarco-Trillo, 2009).
For many terrestrial mammals, scent marks convey informa-
tion about the donor to nearby conspecics and heterospecics
(Johnston and delBarco-Trillo, 2009;Kaur et al., 2014;Roberts
et al., 2014;Wyatt, 2014). These scent marks are viewed as
honest signals of the donors quality or condition (Ferkin,
2011;Roberts, 2007;Thonhauser et al., 2013a) because
many of the substances used as scent marks are digestive
exudates (Albone, 1984). Thus, scent marks from sources
such as the urine (Drickamer, 1995;Zala et al., 2004;Charlton,
2015), saliva (Block et al., 1981), feces (Tinnesand et al., 2015),
and those from specialized glands, such as the preorbital gland
(Ceacero et al., 2015;Pluhá
cek et al., 2015), anal gland (Bills
et al., 2013), foot pads (Owen et al., 2015), submandibular
glands (Camacho-Arroyo et al., 1999;Mykytowycz, 1965),
and the integument (Martin et al., 2014;McLean, 2014) accu-
rately reect the condition and phenotype (Sabau and Ferkin,
2013a), height (Sharpe, 2015) or genotype of the sender
(Green et al., 2015). Scent marks from these multiple sources
likely provide unique and overlapping information about the
sender (Ferkin and Johnston, 1995a;Hurst and Beynon,
2004;Johnston, 2003,2009;Kaur et al., 2014). As a result,
the response of terrestrial mammals to scent marks and odors
of conspecics has remained of interest to researchers in
anthropology, biology, chemistry, psychology, neuroscience,
and medicine (Apps, 2013;Drea, 2015;Ferkin, 2015;Grimm,
2014;Johnston and delBarco-Trillo, 2009;Logan, 2014;
Petrulis, 2013a,b,2015;Roberts, 2007;Roberts et al., 2014;
Schulte et al., 2013;Wyatt, 2014).
The fact that researchers from different elds are interested
in olfactory communication has led to confusion in the termi-
nology to describe chemical signals. Johnston and delBarco-
Trillo (2009) created a means to identify different types of
signals involved in odor communication. The researchers
argued that chemical signals may be categorized as being
a pheromone, a pheromone blend, or a mosaic. Accordingly,
a pheromone would be a single chemical compound, which
would be sufcient to induce one or more responses in
the receiver. A pheromone blend would be comprised of
several compounds in relatively specic proportions. These
compounds, in the proper amount, would be necessary to stim-
ulate responses in the receiver. A mosaic signal contains a large
number of chemical compounds, which are necessary to evoke
responses in the receiver. It is not known if a mosaic signal
creates an individually distinctive odor, sexually distinctive
odor, or group (species) odors based on differences in the
proportions of these compounds across individuals. It is
possible that most odor signals produced by terrestrial
mammals are mosaic signals as they often involve many
different compounds from multiple sources of scent. The ratio-
nale for this conjecture is that odor signals contain many chem-
ical compounds (Apps, 2013;Schulte et al., 2013) and are
derived from multiple sources of scent, including by-products
286 Communication by Chemical Signals
Hormones, Brain and Behavior, Third Edition, 2017, 285–327
Author's personal copy
of hormone breakdown, excreta, and specialized tissues and
glands in the integument (Johnston and delBarco-Trillo,
2009; Petrulis, 2013). Sources of Scents and Odors
Some sources of scent produce marks that convey sexual infor-
mation about the donor.
For example, meadow voles, Microtus pennsylvanicus, have
a highly localized pattern of sexual information on their bodies
during the breeding season (Ferkin and Johnston, 1995a). The
feces, anogenital area, and urine scent marks of meadow voles
are attractive to opposite-sex conspecics but not to same-sex
conspecics. However, saliva/mouth secretions of female voles
were attractive to male but not to female conspecics, whereas
the posterolateral region scent marks of male voles were attrac-
tive to both male and female conspecics. For meadow voles,
the secretions from the mouth and posterolateral region likely
provide different information than that from the feces, anogen-
ital area, and urine (Ferkin and Johnston, 1995a).
Similarly, the chemical signals for discrimination and recog-
nition of individuals can come from many different specic
sources, such as a variety of specialized scent glands (e.g., seba-
ceous glands, apocrine and eccrine sweat glands) as well as
urine and feces. In Syrian hamsters, Mesocricetus auratus, for
example, there are ve different sources of individually distinc-
tive scents, namely ank gland, vaginal secretions, ear glands
(inside the pinna), urine, and feces (Johnston et al., 1991).
Six other potential sources of odors in Syrian hamsters were
tested but were not found to be individually distinctive, as
measured by habituationdishabituation tests (fur from the
midline ventral surface, fur from the dorsal surface between
the shoulders, saliva, feet, fur behind the ears, and the ank-
gland area from ank-glandectomized males) (Johnston
et al., 1991). Similar results were found with Djungarian
hamsters, Phodopus campbelli. Male hamsters were able to
discriminate individual differences between other males using
scents from the midventral gland, urine, feces, mouth, and
the sacculi from the corner of the mouth (Lai and Johnston,
1994). These different odor sources that provide information
about individual identity may contain some redundant infor-
mation, but the combined information from of all of these
sources may facilitate the creation of stronger memories about
the identity of a given conspecic. The Attractiveness of Scent Marks and Conspecific
Odor Preferences
The attractiveness of scent marks from the above sources can
vary according to, for example, the diet and reproductive condi-
tion of senders (Boonstra, 1994;Wade et al., 1996). This can be
reected in the attractiveness of its scent marks to receivers
(Pierce et al., 2005,2007;Sabau and Ferkin, 2013a;Sabau
et al., 2014). Scent marks also provide honest signals of health
(Zala et al., 2004) and nutritional status of the sender to the
receivers (Hobbs and Ferkin, 2011a,b;Pierce et al., 2005).
Nutritional stresses such as food deprivation and food restric-
tion are an ecological challenge faced by small herbivores
that live in transitional grasslands, where food sources are
patchy and vary in quality across the territories of female
conspecics (Batzli, 1985;Bronson, 1989;Getz, 1985). Food
availability affected the amount of time male and female
meadow voles spent self-grooming in response to the odors
of opposite-sex conspecics. Specically, male and female
meadow voles that were food deprived for either 6 or 24 h
spent less time self-grooming compared to voles that had
continuous access to food (Hobbs and Ferkin, 2012b). It is
not clear at this time why food deprivation and restriction
had different effects on self-grooming and scent marking in
voles. Perhaps, the difference lies in the fact that voles self-
groom when they encounter the scent marks of conspecics
that are in close proximity, whereas voles scent mark when
they encounter the scent marks of conspecics that may be
nearby and when they are distant (Hobbs and Ferkin, 2012b;
Hobbs et al., 2008).
Given the importance of nding potential mates of the
opposite sex and competing with rivals of the same sex, identi-
cation of males and females is probably universal in all
species that reproduce sexually. Chemical signals often provide
crucial information about the sex of conspecics; a few exam-
ples of species in which individuals have been shown to be
more attracted to odors of opposite-sex conspecics than those
of same-sex conspecics include many species of terrestrial
mammals (Eisenberg and Kleiman, 1972;Ferkin and Johnston,
1995a,b;Johnston, 1983). Male mice (Mus musculus) respond
with a surge in luteinizing hormone when exposed to female
mouse odors but not when exposed to male mouse odors or
female hamster odors (Maruniak and Bronson, 1976). Such
hormonal responses indicate that males discriminate sex
from odors even if a behavioral preference cannot be detected.
In most species, there may be several distinct sources of odor
that provide information about sex and/or reproductive state,
although not all odors necessarily contain such information.
For example, in Djungarian hamsters, males showed a prefer-
ence for the odors of receptive females over the odors of males
when the odor source was urine, anogenital secretions, saliva,
or midventral gland secretion but not when the odor source
was feces or secretions from the feet (Lai et al., 1996).
Several studies have shown that the attractiveness of the
scent marks of spontaneous ovulators also varies temporally
(Brown and Macdonald, 1985). The changes in the attractive-
ness of a senders scent mark appear to be concomitant with
changes in the responses of receivers to such marks (Ferkin,
2011;Johnston, 2009;Roberts, 2007;Sabau and Ferkin,
2013a). The scent marks of female house mice and Syrian
hamsters are attractive to male conspecics only when the
female is in estrus (delBarco-Trillo et al., 2009c;Johnston,
1983;Kavaliers et al., 1994;Figure 1). Postpartum estrus
female mammals produced odors and scent marks that were
more attractive to males than were those produced by females
not in postpartum estrus (Ferkin and delBarco-Trillo, 2014;
Ferkin and Johnston, 1995b;Lai et al., 1996;Witt et al.,
1990;Zeigler et al., 1993). Postpartum estrus female voles, rela-
tive to females not in postpartum estrus, also spent more time
and showed increases in scent marking, over-marking, and self-
grooming when they encountered the scent marks of males
(Ferkin and Leonard, 2010;Ferkin, 2011).
In addition to showing a preference for odors of opposite-
sex individuals, male mammals usually show a preference for
odors of females that are sexually receptive over those of
females that are in other reproductive states (Brown, 1985).
For example, male meadow voles prefer the odors of females
Communication by Chemical Signals 287
Hormones, Brain and Behavior, Third Edition, 2017, 285–327
Author's personal copy
in postpartum estrus (when females are in a state of heightened
sexual receptivity) than those of females in induced estrus or
undergoing pregnancy (Ferkin and delBarco-Trillo, 2014).
Sexually experienced female rats, Rattus norvegicus, prefer odors
of normal males over the odors of castrated males (Carr et al.,
1965). Similarly, male rats prefer odors of receptive females
over odors of nonreceptive females. These preferences,
however, may depend on knowledge that males have learned
during previous interactions with females. For example, sexu-
ally experienced male rats prefer odors of receptive females
over odors of nonreceptive females but sexually naïve males
do not show this preference (Carr et al., 1965). Yet, the ability
of males to discriminate between male and female and the
male preference for female odors may be two distinct mecha-
nisms. Experiments with house mice showed that males with
the vomeronasal organ removed did not show a preference
for the odors of estrous females over odors of males (Pankevich
et al., 2004). However, these males were still able to discrimi-
nate between odors of males and estrous females when tested
in a habituationdishabituation method. These results make
the important point that the ability to discriminate between
odors of males and females does not necessarily lead to func-
tionally relevant responses.
The preference for opposite-sex conspecics is also often
dependent on the reproductive state of the receiver. In domestic
rats, males prefer the odors of estrous females to those of
females in other reproductive states but castrated males do
not show such preference (Carr et al., 1965;Randall, 1986).
Similarly, the preference that diestrous, estrous, and lactating
female Syrian hamsters show for the odors of intact male
hamsters over castrated males is not shown by pregnant
females (Johnston, 1979).
Familiarity of a female with a male and/or his odors can
also be a determining factor in her odor preference. A female
repeatedly exposed to odors of a male may show a preference
for that male over a male that is unfamiliar, possibly because
familiarity indicates the ability to defend a territory or living
area for some period of time. Female pygmy loris, Nycticebus
pygmaeus, in captivity were rst exposed to urine marks from
one male over 1420 weeks. When the female was close to
Figure 1 Habituationdishabituation test. A male Syrian hamster was exposed to vaginal secretions of a donor female during four habituation trials
(Hab 1to Hab 4) and consequently to the vaginal secretions of a second donor female during the dishabituation trial (Test). The four panels
represent four scenarios in which: (a) both female donors were in estrus; (b) both female donors were in diestrus-2 (i.e., not in estrus); (c) the rst
female was in diestrus-2 and the second female in estrus; (d) the rst female was in estrus and the second female in diestrus-2. A dishabituation
response (a signicant increase in investigation time in the dishabituation trial compared to the last habituation trial) was observed only when the
second donor female was in estrus. E, estrus; Di, diestrus-2; *p<0.05; **p<0.01; ***p<0.001. Reproduced from delBarco-Trillo, J., LaVenture,
A.B., Johnston, R.E., 2009. Male hamsters discriminate estrous state from vaginal secretions and individuals from ank marks. Behav. Process. 82,
1824, with permission from Elsevier.
288 Communication by Chemical Signals
Hormones, Brain and Behavior, Third Edition, 2017, 285–327
Author's personal copy
ovulation, males conned in small cages were placed in the
much larger living areas of females. Using three different behav-
ioral measures, females showed strong preferences for the
familiar male (Fisher et al., 2003a). In another example, female
collared lemmings were rst housed with a male for 30 days
and were then separated for 1, 12, or 24 days. Females showed
a strong preference for their partnersodors over the odors of an
unfamiliar male 12 days after separation, but they did not
show such a preference after 24 days of separation (Huck and
Banks, 1979). Female rats also show a preference for the odors
of a previous, familiar mate over the odors of a novel male. In
contrast, male rats show a preference for the odors of a novel
female over the odors of a previous mate (Carr et al., 1980).
In socially monogamous species one predicts that mated indi-
viduals will not only be able to recognize their partner from
other conspecics but also that they will show a preference
for the partners odors compared to those of unfamiliar,
opposite-sex individuals (Duarte et al., 2015).
1.10.2 Physiological Mechanisms Neural Substrates
Most social odors are processed by two complimentary chemo-
sensory systems, the main (MOS) and accessory olfactory
(AOS) system (Baum and Kelliher, 2009) as represented in
Figure 2. AOS sensory neurons are located in the vomeronasal
organ (VNO), a blind-ended tube that primarily takes up con-
tacted, nonvolatile, chemosignals (Chamero et al., 2012;
Halpern and Martinez-Marcos, 2003;Meredith, 1994;Zufall
and Leinders-Zufall, 2007). In contrast, sensory neurons of
the MOS (main olfactory epithelium: MOE) are located within
the nasal turbinates and respond primarily to volatile chemo-
signals via snifng (Murthy, 2011;Wachowiak, 2011). These
systems also have separate projection zones within the olfac-
tory bulbs (OB): MOE neurons project to main olfactory bulbs
(MOB) and the VNO neurons project to the accessory olfactory
bulbs (AOB) (Scalia and Winans, 1975). The separation of AOS
and MOS is extended further into the brain systems such that
there are few areas of overlap, primarily in the anterior part
of the medial amygdala (MA) (Fan and Luo, 2009;Kang
et al., 2009;Mohedano-Moriano et al., 2012) or through indi-
rect connections in the cortical amygdala (MOB input, projects
to MA) and posterior bed nucleus of the stria terminalis (BNST;
AOB input, projects to MA) (Maras and Petrulis, 2010a;
Martinez-Marcos, 2009). Indeed, the MOS can alter how the
AOS processes social odors. For example, MOB inputs to the
anterior MA alter how the AOB responds to volatile phero-
mones (Martel and Baum, 2009) and the detection of volatile
odors by the MOS triggers VNO sampling, thereby increasing
chemosignal access to the VNO (Meredith, 1998).
The MA is not only the primary area of chemosensory inte-
gration, it is also a steroid-sensitive and sexually dimorphic
structure (Cooke, 2006) as are its connected structures, such
as the BNST, medial preoptic area (MPOA), anterior hypothal-
amus (AH), and ventromedial hypothalamus (VMH) (Segovia
and Guillamon, 1993;Simerly, 1990;Wood, 1997). Detailed
analysis of MA and its connected structures shows that steroid
and chemosensory information are represented by partially
separate and parallel systems (Maras and Petrulis, 2010a,b;
Newman, 1999;Wood, 1997) that provide a substrate where
steroid hormones and chemosensory systems may interact
(Been and Petrulis, 2011;Maras and Petrulis, 2010a). Impor-
tantly, these interconnected brain regions form the core of
the social behavior neural network(SBNN), a highly
conserved system of interconnected, steroid-sensitive and sexu-
ally dimorphic limbic structures that control multiple social
behaviors (Newman, 1999;OConnell and Hofmann, 2012). Mechanisms of Communication
Chemosensory cues from conspecics guide almost every kind
of mammalian social behavior studied to date, including
aggressive, sexual, and parental behaviors, and the physiolog-
ical control of these behaviors is described elsewhere (Petrulis,
2013). Here we focus on the hormonal and neural control of
behaviors representing productive (scent marking, vocaliza-
tions) aspects of odor communication. The vast majority of
Figure 2 Abbreviated schematic of main olfactory (MOS; gray) and
accessory olfactory or vomeronasal (AOS; white) systems along with
integrative areas (black). For clarity, only unidirectional olfactory bulb
connections are presented and several areas without known relevance
to social odor processing are omitted. AOB, accessory olfactory bulb;
ACo, anterior cortical amygdala; BNST, bed nucleus of stria terminalis;
END, endopiriform nucleus; ENT, entorhinal cortex; HIPP, hippo-
campus; MA, medial amygdala; MDth, mediodorsal thalamus; MOB,
main olfactory bulb; MOE, main olfactory epithelium; MPOA, medial
preoptic area; OFC, orbitofrontal cortex; PIR, piriform cortex; PMCo,
posteromedial cortical amygdala; VMH, ventromedial hypothalamus;
VNO, vomeronasal organ; VPH, ventral premammilary hypothalamus.
Reproduced from Petrulis, A., 2013. Chemosignals, hormones and
mammalian reproduction. Horm. Behav. 63, 723741, with permission
from Elsevier.
Communication by Chemical Signals 289
Hormones, Brain and Behavior, Third Edition, 2017, 285–327
Author's personal copy
detailed information on this topic comes from the study of
only a few species: the Syrian hamster, the Mongolian gerbil,
the house mouse, the Norway rat, and the rabbit. Conse-
quently, our coverage will be constrained to these species, but
it should be noted that neural studies with similar ndings
have been reported in other species, such as domestic cats
(Hart and Voith, 1978;Lischka et al., 2008) and dogs (Hart,
1974;Gadbois and Reeve, 2014;Jia et al., 2014).
When rodents of either sex investigate conspecic odor cues,
they often deposit their own odor cues through stereotyped
scent-marking responses and/or vocalize, often in the ultra-
sonic range (Brown and Macdonald, 1985; Petrulis, 2013;
Pierce et al., 1989;Wyatt, 2014). Both of these behaviors serve
to facilitate localization of mates over different temporal and
spatial ranges (Pfaff et al., 2008). Although the details may
vary as to exact dominant vocal frequencies or the postural
and chemical nature of deposited scent marks, there is remark-
able consistency in the chemosensory, hormonal, and neural
substrates of these behaviors across model species. Indeed,
the aggregate evidence, described below, suggests that both
scent marking and vocalizations are driven by a steroid-
dependent neural network composed of the chemosensory
inputs to the extended medial amygdala (MA, BNST), its
outputs to basal forebrain (MPOA, AH, VMH) and connected
midbrain elements (periaqueductal gray, PAG). Scent Marking Syrian Hamster: Flank Marking
Unlike species discussed below, both male and female
hamsters deposit products of dorsal ank sebaceous scent
glands on vertical surfaces, a behavior termed ank marking
(Johnston, 1975a,1977). Flank marking is thought to reect
territorial or competitive motivation as it occurs primarily in
response to same-sex odors and following establishment of
dominantsubordinate relationships (Johnston, 1975b,
1977). This behavior has been extensively investigated at the
neural level following the accidental discovery that MPOA
injections of arginine-vasopressin (AVP) injections elicit excep-
tional levels of ank marking in both males and females
(Albers, 2015;Ferris et al., 1984). Both AVP- and odor-
induced ank marking is blocked by MPOA injections of AVP
V1a receptor antagonists (Albers et al., 1986). The key locations
within MPOA that trigger AVP-induced marking are primarily
in the ventral parts of the anterior hypothalamus (AH) (Ferris
et al., 1986a) and the source of the AVP appears to be from
nonpituitary projecting nucleus circularis (NC) cells within
AH and the medial supraoptic nucleus (mSON) (Delville
et al., 1998;Ferris et al., 1990b,1992;Mahoney et al., 1990).
Indeed, subordinate hamsters (that do not mark) have lower
levels of AVP in NC (Ferris et al., 1989) and these cells (and
mSON) show increased activity during aggressive interactions
(Delville et al., 2000). Further studies have indicated that
AVP action within the AH requires excitatory glutamatergic
neurotransmission (Bamshad et al., 1996) and that it is
antagonized by serotonin action (Albers et al., 2002). Input
to the AH from the broader AVP-sensitive brain also regulates
ank marking as LS/BNST lesions reduce marking and AVP
injections in lateral septum (LS), or in nearby BNST, trigger it
(Ferris et al., 1990a). Lesions of AH block the induction of
ank marking by AVP injections in LS/BNST. In contrast, LS
lesions have no effect on the ability of AVP injections in AH
to drive ank marking (Ferris et al., 1994). This strongly
suggests that AH is downstream of LS control and is the
primary regulator of AVP-driven ank marking, perhaps via
midbrain structures (Hennessey et al., 1992).
Initial reports demonstrated that both ank marking and
V1a receptors (but not AVP itself) in MPOA/AH are dependent
on gonadal steroids, especially androgens (Albers and
Rowland, 1989;Albers and Prishkolnik, 1992;Albers et al.,
1988,1996;Huhman and Albers, 1993;Young et al., 2000).
However, more recent work suggests a more complicated
picture. Although male hamsters in a nonbreeding photope-
riod have regressed testes and low levels of circulating testos-
terone, ank marking levels are testes-independent and not
reduced during social encounters, despite steroid-dependent
reductions in V1a receptors (Caldwell et al., 2008).
The AVP manipulations within AH alter ank marking irre-
spective of social status, but they do not have lasting effects on
dominance interactions (Ferris et al., 1986b), suggesting that
AH AVP primarily regulates behavioral output rather than stim-
ulus evaluation or dominance/subordination itself. So, what
then is the source of social odor information to the AH?
Initially, this information comes from both the MOS and
AOS representations in the olfactory bulb as olfactory bulb
removal eliminates ank marking (Kairys et al., 1980). Periph-
eral lesions of the MOS reduce male and female ank marking
in arenas scented with the odors of both male and female
conspecics (Johnston, 1992;Johnston and Mueller, 1990).
Although VNO removal does not reduce ank marking in
this context, excising the VNO in females reduces preferential
targeting of this behavior toward other females, without
altering their low baseline response to male odors (Petrulis
et al., 1999). As the MA serves as the main point of convergence
between MOS and AOS, it is perhaps unsurprising that
complete lesions of MA virtually eliminate ank marking in
females (Petrulis and Johnston, 1999); smaller lesions of either
anterior or posterodorsal MA result in reductions rather than
outright elimination of this behavior (A.P., unpublished obser-
vations). As the MA has substantial inputs to MPOA/AH, it is
likely that odor modulation of ank marking is mediated by
MA (and perhaps cortical amygdala; (Petrulis et al., 2000) effer-
ents to AVP-sensitive regions of AH; this awaits further
investigation. Syrian Hamster: Vaginal Marking
Female hamsters scent mark (vaginal mark) to advertise
impending sexual receptivity (Johnston, 1977) similar to rats,
house mice, and rabbits (Birke, 1984;Chang et al., 2000;
González-Mariscal et al., 1990;Rich and Hurst, 1999). Vaginal
marking is a stereotyped behavior that deposits sexually attrac-
tive vaginal uid on the surface (Been and Petrulis, 2008;Been
et al., 2012) and is specically targeted toward male odors
(Johnston and Brenner, 1982;Johnston, 1977;Petrulis and
Johnston, 1997). This behavior changes substantially across
the females estrous cycle (Figure 3) such that females mark
most on the night prior to sexual receptivity (during high circu-
lating estrogen levels) and then show no vaginal marking
during sexual receptivity (during high progesterone levels)
(Johnston, 1977;Lisk and Nachtigall, 1988). This remarkable
hormone-dependent shift in vaginal marking is likely regulated
290 Communication by Chemical Signals
Hormones, Brain and Behavior, Third Edition, 2017, 285–327
Author's personal copy
by the steroid action on MPOA and VMH: E implants in these
areas restore vaginal marking in OVX females (Takahashi et al.,
1985). However, the VMH may not be critical for vaginal
marking, as VMH lesions do not eliminate the behavior
(Floody, 2002). The role of the MPOA is presently unclear as
large, ber-damaging MPOA lesions disrupt vaginal marking
(Malsbury et al., 1977), whereas smaller, ber-sparing lesions
limited to MPOA do not (Martinez and Petrulis, 2013). This
discrepancy is likely due to the concomitant damage to BNST
present in the earlier study, as ber-sparing lesions conned
to the posterior BNST impair vaginal marking (Martinez and
Petrulis, 2011) without altering the cyclicity of this behavior
(Figure 3).
As is the case for ank marking, olfactory bulbectomy
reduces vaginal marking (Kairys et al., 1980) and the disruptive
effects of MOS peripheral lesions on vaginal marking are much
more pronounced than those following VNO removal (John-
ston, 1992;Petrulis et al., 1999). MOS inputs to the MA, rather
than to other MOS targets (Petrulis et al., 1998,2000), appears
to regulate vaginal marking as MA lesions lead to signicant
reductions in vaginal marking (Petrulis and Johnston, 1999;
Takahashi and Gladstone, 1988). Interestingly, females with
MA lesions still show differentially high marking rates to
male odors and still retain normal cyclic variation in vaginal
marking. This suggests that MA is not critical for hormonal
control of marking (likely mediated by hypothalamic struc-
tures) or differential marking responses. In contrast, the BNST
is critical for differential marking to male odors (Martinez
and Petrulis, 2011). This effect is likely mediated by oxytocin
action because blockade of oxytocin receptors in BNST and
MPOA reduces male-directed vaginal marking (Martinez
et al., 2010,2013). House Mouse: Urine Marking
Urine marking by socially dominant male mice is generally
used in the context of territorial defense (Arakawa et al.,
2008a,b;Desjardins et al., 1973;Lumley et al., 1999). However,
male urine marking does not appear to be differentially tar-
geted toward males versus females (Hurst, 1990a,b,c;Labov
and Wysocki, 1989), suggesting a more general advertising
function. Nevertheless, males do urine mark at much greater
levels than do females, an effect largely driven by sex differ-
ences in adult and developmental exposure to testosterone
and its estrogenic metabolites (Juntti et al., 2010;Kimura and
Hagiwara, 1985;Wu et al., 2009). These steroid effects appear
to be exerted on MPOA/AH, BNST, or VMH neurons as testos-
terone or estradiol implants near or in these structures can
maintain urine marking in gonadectomized males (Matochik
et al., 1994;Nyby et al., 1992), perhaps through interactions
with midbrain dopamine systems (Sipos and Nyby, 1996).
The available evidence suggests that the MA is not a substrate
for steroid effects on urine marking (Sipos and Nyby, 1998;
Unger et al., 2015). Recent evidence suggests that central
oxytocin decreases urine marking and thereby provides
a possible chemical mechanism through which marking can
be decreased in subordinate animals (Arakawa et al., 2015).
Surprisingly, little is known as to how chemosensory informa-
tion is transmitted to steroid-sensitive structures. What
evidence exists suggests that urine marking is partially medi-
ated by the AOS (MOS contribution not yet assessed) as evi-
denced by decreased marking in males following VNO
removal (Labov and Wysocki, 1989;Maruniak et al., 1986).
As in many other social contexts, removal of VNO has a greater
effect in sexually naïve animals than in males with sexual expe-
rience (Clancy et al., 1984;Labov and Wysocki, 1989;Maru-
niak et al., 1986). Mongolian Gerbil: Ventral Marking
Dominant male gerbils engage in more scent marking than do
subordinate male gerbils. Much of this scent marking involves
the ventral gland. Briey, individuals rub odors from this
ventral sebaceous scent gland onto the substrate (Thiessen
et al., 1968,1970;Whitsett and Thiessen, 1972;Yahr, 1977).
Adult gonadectomy reduces both male and female marking
in gerbils and can be reinstated with administration of testos-
terone or its metabolites in males (Yahr and Stephens, 1987;
Yahr et al., 1979) or estradiol plus progesterone in females
(Owen and Thiessen, 1974;Yahr and Thiessen, 1975). The
sex difference in overall levels of marking is due to organiza-
tional effects of gonadal steroids as neonatal testosterone and
subsequent adult T in females will increase female marking
to male levels (Ulibarri and Yahr, 1988,1996). Testosterone
appears to exert its effect via the MPOA; implants near the
most rostral or caudal limits of MPOA are most effective in
restoring marking in gonadectomized males (Thiessen and
Yahr, 1970;Thiessen et al., 1973;Yahr et al., 1982). Indeed,
Diestrous day 2
Number of vaginal marks
30 Male
(a) (b)
Figure 3 Mean (SEM) number of vaginal marks to male and female odors. SHAM, but not BNST-X, females preferentially marked at higher levels
to male odors compared to female odors on both (a) diestrous day 2, and (b) proestrus. *p<0.01, male versus female odor condition.
Communication by Chemical Signals 291
Hormones, Brain and Behavior, Third Edition, 2017, 285–327
Author's personal copy
large MPOA lesions that include these areas reduce ventral
marking in males (Yahr et al., 1982) particularly if these lesions
remove the sexually dimorphic cell groups in caudal MPOA
and BNST (Commins and Yahr, 1984). However, small ber-
sparing lesions of these cell groups had no effect on ventral
marking (Yahr and Gregory, 1993) indicating that previous
effects were due to damage to passing axon bers. Further
attempts at identifying brain regions that regulate ventral
marking have not been successful: small, ber-sparing lesions
of posterodorsal MA, medial preoptic nucleus, or posterome-
dial BNST were without effect on ventral marking (Heeb and
Yahr, 2000;Sayag et al., 1994). Rabbits: Chinning
Studies have examined the mechanisms underlying chinning,
a type of scent marking in rabbits. For example, González-
Mariscal et al. (1990) found that the frequency of chinning
varied with the females reproductive cycle. Moreover, during
estrous there were more acini/eld than during pregnancy
and lactation (Cerbón et al., 1996). Gonadectomy reduced
the number of acini/eld and increased their diameter in
females but had the opposite effect in males (Cerbón et al.,
1996). Hudson et al. (1990) found that estradiol treatment
was necessary to increase chinning in ovariectomized rabbits.
Interestingly, progesterone treatment given to estradiol-
primed, ovariectomized females induced an immediate
suppression of chin marking and lordosis, but progesterone
therapy alone had no effect on these measures in ovariecto-
mized female rabbits (Hudson et al., 1990). González-
Mariscal et al. (1993) found that chinning in males depends
on testicular steroids, as castration decreased the frequency of
chinning. This behavior was nearly restored to precastration
levels by treatment with testosterone propionate, whereas the
administration of estradiol and dihydrotestosterone increased
chinning above precastration levels (Gonzalez-Mariscal et al.,
1993). Melo et al. (2008) examined the brain sites where
steroids act to stimulate chinning. They did so by implanting
testosterone propionate or estradiol benzoate into the ventro-
medial hypothalamus (VMH) or the medial preoptic area
(MPOA) of gonadectomized male and female rabbits, respec-
tively. Melo et al. (2008) found that, for female rabbits,
chinning was stimulated by EB implants into the VMH or
MPOA. In contrast, for males, chinning was stimulated by
testosterone implants into the MPOA, but not into the VMH
(Melo et al., 2008). Camacho-Arroyo et al. (1999) discovered
progesterone and estrogen receptors in the nucleus of acini
cells of the submandibular (chin) glands of rabbits of both
sexes. Intact males had a smaller number of estrogen
receptor-immunoreactive cells and progesterone receptor-
immunoreactive cells than did estrous females. Progesterone
receptor-immunoreactive cells were more numerous in
estrous than they were in ovariectomized rabbits. Estradiol
therapy increased the number of progesterone receptor-
immunoreactive cells in ovariectomized female rabbits
(Camacho-Arroyo et al., 1999). Ultrasonic Vocalizations House Mouse
In mice, reproductively active and territorial/dominant adult
males produce high amounts of ultrasonic vocalizations
(USV) specically during investigation of novel females or their
urine; males or their urine do not elicit USVs from other males
or females (Guo and Holy, 2007;Nyby et al., 1977;Nyby,
1983b;Wang et al., 2008;Warburton et al., 1989;White
et al., 1998) and so these USVs are often considered to be
male-typical promoters of opposite-sex interactions (Chabout
et al., 2015;Musolf et al., 2010;Pomerantz et al., 1983;Sugi-
moto et al., 2011). Female mice do produce a modest amount
of USVs during femalefemale mutual investigation (Moles
et al., 2007) but, unlike female-directed calling by males, this
female-directed calling habituates rapidly (Maggio and
Whitney, 1985;Warburton et al., 1989).
Removal of olfactory bulbs in males eliminates USVs to
female odors and similar or lesser effects are also observed after
AOS damage, especially in sexually naïve animals (Bean, 1982;
Wysocki et al., 1982). Females with genetically induced VNO
dysfunction display increased USVs to males suggesting that
the VNO may normally suppress male-typical behavior in
female mice (Kimchi et al., 2007;Stowers et al., 2002, but
see Pankevich et al., 2004). Data on the role of the MOS in
USV generation is more equivocal: peripheral lesions of MOS
are reported to either have no effect (Bean, 1982) or to reduce
USVs (Sipos et al., 1995).
Given the sex difference in USV production, it is perhaps not
surprising that male USVs are strongly steroid-dependent. Male
USVs are eliminated by adult gonadectomy and restored, and
even induced in females, by circulating testosterone or its estro-
genic/androgenic metabolites (Bean et al., 1986;Juntti et al.,
2010;Matochik and Bareld, 1991;Nunez et al., 1978;
Pierman et al., 2008;Wu et al., 2009). The male-biased pattern
of USV production does not appear to require organizational
hormone action, as neonatal castration does not alter male-
typical patterns of calling (Warburton et al., 1989). The identi-
ed brain sites of gonadal steroid action on USVs (MPOA/AH,
BNST, or VMH) are the same ones identied for urine marking
as testosterone or estradiol implants around these structures
maintains USVs in gonadectomized males (Matochik et al.,
1994;Nyby et al., 1992), perhaps interacting with midbrain
dopamine systems (Sipos and Nyby, 1996). However, small
lesions to MPOA do not change USVs (Bean et al., 1981) and
so suggests that MPOA steroid implants are effective because
of steroid spread to nearby areas, such as BNST. The available
evidence suggests that the MA is not a substrate for steroid
effects on USVs (Sipos and Nyby, 1998;Unger et al., 2015),
but MAs broader role in regulating USVs has not been assessed. Syrian Hamster
Both estrous female and male hamsters (Floody et al., 1977,
1979a;Johnston and Kwan, 1984) produce USVs in response
to opposite-sex conspecics or their odors; this response is
steroid-dependent in both sexes (Floody and Petropoulos,
1987;Floody et al., 1979a,b,1987). Hamster USVs to odors
requires intact olfactory bulbs (Kairys et al., 1980) and either
a functioning MOS or AOS (Johnston, 1992). Lesions of the
MA leads to decits in USV production by female hamsters,
suggesting that MOS/AOS chemosensory information
converge in this structure (Kirn and Floody, 1985). It is likely
that MA projections to MPOA/AH or BNST are critical for this
behavior as damage to MPOA/AH impairs USVs (Floody,
1989) as do knife cuts that damage BNST connections
292 Communication by Chemical Signals
Hormones, Brain and Behavior, Third Edition, 2017, 285–327
Author's personal copy
(Miceli and Malsbury, 1982). Similar to studies in other
species (Jürgens, 2009), hamster vocalizations require
processing by the periaqueductal gray (PAG) region of
the midbrain (Floody and DeBold, 2004;Floody and
ODonohue, 1980) via a steroid-sensitive mechanism (Floody
et al., 1986). Norway Rat
Both male and estrous female rats emit USVs (50 kHz) during
interactions with opposite-sex individuals or their odors (Geyer
and Bareld, 1978;White et al., 1991). Male USVs are depen-
dent on both androgen- and estrogen-receptor activity
(Matochik and Bareld, 1991;Vagell and McGinnis, 1998).
Testosterone action within the AH and/or VMH appears to be
necessary for USVs in males (Harding and McGinnis, 2004)
even though it is not sufcient (Harding and McGinnis,
2003). Indeed, small lesions in VMH inhibit the ability of testos-
terone to restore USVs in gonadectomized male rats (Harding
and McGinnis, 2005) suggesting that the VMH is one of the crit-
ical steroid-sensitive brain regions for USVs. Other structures
may also be involved in regulating USVs: pharmacological stim-
ulation of the BNST or MPOA/AH increase 50 kHz USVs in
males (Burgdorf et al., 2007;Fu and Brudzynski, 1994;Wintink
and Brudzynski, 2001), likely via their direct connections with
midbrain vocal centers (Kyuhou and Gemba, 1998).
1.10.3 Development and Learning Reproductive States and Odors
In many mammalian species females approaching or in estrus
may release odors or increase their scent marking to advertise
their sexual receptivity and thus attract mates. For example,
female Syrian hamsters increase their vaginal and ank
marking immediately prior to sexual receptivity, concentrating
those scent marks around the dominant male territories and
also creating a trail of scent marks between the burrow of
a dominant male and her own burrow (Huck et al., 1985a).
Several mechanisms may be used for advertising sexual recep-
tivity. The quantity of secretions produced and available for
scent marking may be increased, the frequency of scent
marking may increase, the composition of odors may change
with the females reproductive state, or a combination of these
mechanisms may be employed. With the rst two mechanisms
there need not be a change in the quality of the odor, but in
most cases both the quality and the quantity of odors may
change. Indeed, given that metabolic products of sexual
hormones are likely to be contained in the females odors,
such odors can be a direct reection of the reproductive state
of the female and the attractiveness of female odors.
As a further example of how females advertise their sexual
receptivity through odors, we can consider those species in
which females undergo postpartum estrus. Postpartum estrus
is a relatively short period of heightened sexual receptivity
that occurs immediately after parturition. Females that mate
and become pregnant during postpartum estrus gestate a set
of offspring while they are nursing the previous litter. As
a consequence, postpartum estrus allows females to be simul-
taneously pregnant and lactating, maximizing the number of
litters that they can produce during the breeding season.
Postpartum estrus occurs in many species but it is especially
prevalent in small-sized rodents (Ferkin and delBarco-Trillo,
2014) and in lagomorphs (i.e., rabbits, hares, pikas). Female
meadow voles in postpartum estrus are more sexually receptive
and likely to become pregnant than females in other estrous
states (Ferkin and delBarco-Trillo, 2014). These females in
postpartum estrus also deposit more scent marks, advertising
their heightened sexual state to males, who show a preference
for the odors of postpartum-estrus females compared to those
of females in other estrous states (Ferkin and delBarco-Trillo,
2014). In contrast, in mice and prairie voles (Microtus
ochrogaster), females do not increase scent marking when in
postpartum estrus (Coquelin, 1992;Ferkin and delBarco-
Trillo, 2014), possibly because females in these species nor-
mally mate with the same partner, who may be familiar with
the females state of sexual receptivity.
Different sources of female odor may contain information
about her sexual receptivity. Male Syrian hamsters show high
levels of copulatory behavior when all of a females odors are
present but they show declining levels of copulatory behavior
as specic odor sources are removed (Johnston, 1986). Elimi-
nation of vaginal secretions signicantly reduced male copula-
tory behavior and removal of ank, ear, and Harderian
secretions caused additional decreases in copulatory behavior.
Therefore, these four different odor sources carry information
about the attractivity of females, inuencing male sexual
arousal and performance (Johnston, 1986). Similarly, male
Djungarian hamsters show a preference for the urine, saliva,
and vaginal secretion of estrous females over similar odors of
diestrous females, whereas a preference is not shown when
males investigate other types of body odor, such as feces and
midventral gland secretions, even though these odors are
important in other aspects of communication (Lai et al.,
1996). Also in Djungarian hamsters, the attractiveness of three
different odors of females varied across the estrous cycle, each
odor being maximally attractive at a different stage of the cycle
(Lai and Johnston, 1994). That is, different odors from the
same female may be most attractive at different times during
the estrous cycle. This type of pattern is especially interesting
because it might allow males to accurately track and predict
the timing of estrous.
It has already been mentioned that males show a preference
for the odors of sexually receptive females. Such preference for
odors of receptive females may be universal and has indeed
been shown to exist in many species, including the house
mice (Hayashi and Kimura, 1974), brown lemmings, Lemmus
trimucronatus, and collared lemmings, Dicrostonyx groenlandicus
(Huck and Banks, 1984), meadow voles (Ferkin and Johnston,
1995a), Indian desert gerbils, Meriones hurrianae (Kumari and
Prakash, 1984), Mongolian gerbils (Block et al., 1981), wood-
rats, Neotoma lepida (Fleming et al., 1981), Columbian ground
squirrels, Spermophilus columbianus (Harris and Murie, 1984),
dogs, Canis lupus familiaris (Beach and Gilmore, 1949;Dunbar,
1977), rams, Ovis aries (Lindsay, 1965), and pygmy marmosets,
Cebuella pygmaea (Converse et al., 1995). A lack of a preference
for odors of estrous females over odors of nonestrous females is
difcult to interpret and may be due to methodological short-
comings, but it has been reported in some species such as
deer mice, Peromyscus maniculatus (Dewsbury et al., 1986), and
guinea pigs, Cavia porcellus (Nyby, 1983a). Indeed, in species
Communication by Chemical Signals 293
Hormones, Brain and Behavior, Third Edition, 2017, 285–327
Author's personal copy
that have been studied by many investigators, such as mice, rats,
and Syrian hamsters, some studies report male preferences for
estrous females (Carr et al., 1965;Johnston, 1980;Lydell and
Doty, 1972), whereas other studies report a lack of such a pref-
erence (Brown, 1977;Kwan and Johnston, 1980;Landauer
et al., 1978;Taylor and Dewsbury, 1990). Such differing results
are probably due to differences in methodology (delBarco-Trillo
et al., 2009c;Taylor and Dewsbury, 1990). For example, in one
set of experiments with female Syrian hamsters, six different
methods were compared using measures of investigation of
a single stimulus and ve similar methods were used to measure
preference for one of two odor stimuli. Results showed that tests
in the subjects home cage or in a moving air stream were rela-
tively poor, whereas tests in which the stimulus (or stimuli)
were in the center of a relatively large arena or were outside
the arena yielded data that often demonstrated differences in
attraction (Johnston, 1981).
In addition to showing a preference for the odors of recep-
tive females, males may increase their scent-marking rates when
in areas marked by sexually receptive females. For example,
male meadow voles over-mark more the scents of females in
a heightened state of sexual receptivity, i.e., females in post-
partum estrus, than those of females in other estrous states
(Ferkin and delBarco-Trillo, 2014). Males can also respond
physiologically to odors of receptive females; for example,
more erections were elicited by exposing males to odors of
receptive females than to odors of unreceptive females (Sachs,
1999). Males can also use female odors to discriminate
between mated and unmated females (Thomas, 2011).
In several species, sexually experienced males show a prefer-
ence for odors of receptive females but sexually naïve males do
not (Carr et al., 1965;Hayashi and Kimura, 1974;Huck and
Banks, 1984;Johnston, 1980;Lydell and Doty, 1972). In other
studies, however, both sexually naïve and sexually experienced
males show a similar preference for odors of estrous females
over odors of nonestrous females, e.g., in male mice (Rose
and Drickamer, 1975).
Changes in seasonality may result in changes in the compo-
sition of odors and the scent-marking rates of individuals,
which may affect the preference of individuals for the odors
of conspecics. When meadow voles were tested in short-day
conditions, females showed a preference for the odors of other
short-photoperiod females over those of males, probably asso-
ciated with the habit of often nesting with other females in
winter to maximize warmth (Ferkin and Johnston, 1995a).
Short-photoperiod males did not show a preference for the
odors of short-period females over those of other short-
period males. Furthermore, short-photoperiod voles (both
sexes) did not show sex-specic preferences for long-
photoperiod conspecics, and vice versa (Ferkin and Johnston,
1995a). Some of these seasonal changes in odor preferences
may depend on the endocrine state of individuals. For
example, during long photoperiods, the attractiveness of
male scent to females depends on high titers of both testos-
terone and prolactin (Ferkin et al., 1997a).
Mate attraction is not universally driven by females. There
are species in which males produce odors to attract females.
For example, female frogs of the genus Hymenochirus are
attracted to secretions produced by male breeding glands
(Pearl et al., 2000). Females are attracted to water containing
males, but not to water containing females or males without
breeding glands. Males are not attracted to any of these odors,
indicating that the attractant is specic to females and not
a general signal that promotes aggregation (Pearl et al.,
2000). Sexually mature, terrestrial female toadlets, Pseudo-
phryne bibronii, tested in a two-choice Y-maze, were attracted
to the odors deposited by males compared to the clean arm
of the Y-maze (Byrne and Keogh, 2007). Similarly, male red-
bellied newts, Cynops pyrrhogaster, produce a peptide
pheromone in a cloacal gland that has a female-attracting
function (Kikuyama et al., 1995). In contrast, female palmate
newts, Triturus helveticus, are equally attracted to male and
female chemical cues (Secondi et al., 2005). In this species,
males and females aggregate on breeding sites at high
densities. Attraction to odors of conspecics, irrespective of
sex, may serve to locate a breeding site and then short-range
chemical or visual signals may be used to locate mates
(Secondi et al., 2005). In terrestrial salamanders, Plethodon
jordani, the mental gland of males produces a proteinaceous
secretion that operates as a courtship pheromone,
increasing female receptivity and easing insemination
(Feldhoff et al., 1999). There are also several mammalian
examples, including mice, Syrian hamsters, dogs, and pigs,
Sus scrofa, in which females show a preference for the odors
of intact males over those of castrated males, indicating that
male attractivity may be important during sexual interactions
across many species (Brown, 1977;Johnston, 1981;Lisberg
and Snowdon, 2009;Signoret, 1976). Individual Discrimination
Individual discrimination involves the ability of a subject to
detect the odors of different donors as being different. Recogni-
tion does not only involve the ability to discriminate between
the cues/signals from one individual and another, but it also
implies that the subject has some previous knowledge about
the stimulus animal (Halpin, 1986). At the simplest level,
this knowledge may just involve familiarity with one or more
signals from another individual and the ability to categorize
a stimulus into one of two categories (familiar or unfamiliar).
For example, using the knowledge about the odors of ones
own group may be sufcient to accept a given individual into
the nest or not. True individual recognition, however, implies
more thorough knowledge of the characteristics of other
Individual recognition requires the existence of distinct
signals and the ability to detect and discriminate between these
signals. Most animals may combine a broad spectrum of
signals to communicate individual identity, including a combi-
nation of chemical, visual, and auditory cues, producing multi-
modal signals (Higham and Hebets, 2013). Individuals are the
fundamental units of social interaction and social organiza-
tion. Thus discrimination and recognition of individuals, or
classes of individuals, are fundamental to the understanding
of animal behavior. Even though individual recognition medi-
ated by odors is not restricted to vertebrates (Caldwell, 1985),
most of such research has been conducted on vertebrates,
including many mammalian species (Kent and Tang-
Martínez, 2014).
294 Communication by Chemical Signals
Hormones, Brain and Behavior, Third Edition, 2017, 285–327
Author's personal copy
Studies on rodents such as Mongolian gerbils, Meriones
unguiculatus, ground squirrels, Syrian hamsters, meadow voles,
otters (Lontra canadensis), koalas (Phascolarctos cinereus),
raccoons (Procyon lotor), and house mice have also provided
evidence that these animals can distinguish between the scent
marks of different individuals (Charlton, 2015;Ferkin et al.,
2016;Halpin, 1986;Kean et al., 2015;Kent and Tang-
Martínez, 2014;Thom and Hurst, 2004), responding preferen-
tially or displaying a selective memory for the scent mark of
a familiar conspecic compared to that of an unfamiliar
conspecic(Heth et al., 1998;Johnston, 1993;Kaur et al.,
2014;Mateo and Johnston, 2000a,b). Recent work on house
mice has shown that females can learn the individual identity
of males from their scent marks and females can use this infor-
mation to select potential mates (Roberts et al., 2014).
In a series of elegant experiments, Johnston and colleagues
(Heth et al., 1999;Johnston, 1993,2003;Johnston and
Bullock, 2001;Mayeaux and Johnston, 2002;Todrank et al.,
1998) used a habituation/dishabituation paradigm to show
that Syrian hamsters could distinguish between scent marks
of two same-sex conspecics that were similar in condition,
treating each scent mark as if it were individually distinct. John-
ston and coworkers found that the amount of time that Syrian
hamsters spent snifng the scent mark of an individual donor
decreased with each successive exposure to that individuals
marks, indicating that the hamsters habituated to the scent
marks, and viewed them as being familiar. The hamsters were
then exposed to a scent mark of a different conspecic and
that of a familiar donor. During this exposure, the hamsters
spent more time snifng the novel scent mark than the familiar
scent mark, suggesting that scent marks convey individually
distinct information to conspecics (Johnston, 2009). HabituationDishabituation
Different methodological approaches can be used to demon-
strate individual discrimination and recognition. The rst
studies to demonstrate discrimination of odors in mammalian
species used training procedures with food reinforcement to
demonstrate such abilities (Bowers and Alexander, 1967;
Rasa, 1973). During positive reinforcement the subject is pre-
sented with odors from two different donors but rewarded
only when responding to only one of those two odors. If
over a series of trials the subject animal learns to discriminate
between the two odors we can assume that the animal has
the ability to distinguish between them. Positive reinforcement
is a useful approach to determine the discriminating potential
of animals but it has a limited use to understand individual
recognition under normal, nontaught conditions.
The most commonly used approach to study individual
discrimination involves variations of the habituationdishabi-
tuation technique (Halpin, 1986). A subject animal is rst pre-
sented with the odor of one donor (odor A) over several trials
(this is the habituation phase), and then the same subject is
presented with the odor of a second donor (odor B; this is
the discrimination phase). In each trial, the time the subject
spends investigating the odor is recorded. During the habitua-
tion phase, with repeated presentations of new samples of the
same odor stimulus, the behavioral response declines signi-
cantly, indicating that habituation has occurred. In this case,
individuals habituated to odor A at the end of the habituation
phase should investigate odor B longer than odor A during the
last dishabituation trial. This result would suggest that the indi-
vidual views odor A and odor B as being different from one
another. The habituationdishabituation technique is much
more naturalistic and much easier to implement than reinforc-
ing techniques. A variation on this method is to present two
stimuli on the test trial after the habituation phase, namely,
the stimulus that the subject has been habituated to and a novel
stimulus. Simultaneous presentation of the familiar and novel
stimuli is an easier task and may allow the visualization of
differences in response to the novel stimulus when a single-
stimulus test does not (Brown et al., 1987). The optimal dura-
tion of the interval between trials will depend on the species. In
hamsters, varying the intertrial interval from 1 s to 2 days led to
similar results (Johnston, 1993). In contrast, dogs and captive
wolves would not even approach the stimulus on the second
trial after an interval of 1530 min, apparently because they
could determine from a distance that the odor presented on
the second trial was the same as that on the rst trial; when
the interval between trials was shifted to 24 h, both dogs and
wolves showed the typical habituationdishabituation
response (Brown and Johnston, 1983).
Some studies have used across-odor habituation approaches
to determine if animals can remember familiar individuals
using several different distinctive signals and integrate these
separate memories into integrated representations of others.
For example, male hamsters rst had a series of brief interac-
tions with females on four successive days. During these
4 days males had the opportunity to investigate several odor
sources from those females. After habituation to one individu-
ally distinctive odor of a stimulus animal (e.g., that from vaginal
secretions), males were also habituated to other odors from the
same individual (e.g., sebaceous ank glands). That is, subjects
showed an across-odor habituation (Johnston and Jernigan,
1994). Since the odors themselves are composed of very
different chemical compounds, it is not likely that this effect
was due to chemical similarities in the two odors. Additional
evidence for this conclusion comes from results of control
experiments in which subjects were not familiar with the stim-
ulus animals. If there was similarity in odor quality across
different odor sources, subjects should show across-odor habit-
uation without familiarity with the scent donor. However, if
subjects had not previously interacted with the scent donors,
no cross-odor habituation was observed (Johnston and Bullock,
2001;Johnston and Jernigan, 1994). Subsequent experiments
showed that the same effects were observed with other pairs
of odors (Johnston and Bullock, 2001).
In virtually all laboratory studies aimed at individual recog-
nition, it has proven difcult to provide evidence for true
individual recognition. That is, the unique signicance of an
individual, not just the signicance of categories of individuals
such as familiar versus unfamiliar or dominant versus subordi-
nate (Halpin, 1986). In particular, it has been difcult to
demonstrate the emotional signicance of another individual
without possible confounding interpretations, such as
differences in familiarity or dominance status (Martin and
Beauchamp, 1982). In one attempt to disentangle the effect
of familiarity during individual discrimination, male hamsters
were exposed to two different males. One stimulus male beat
the subject in a series of three brief ghts, whereas the subject
Communication by Chemical Signals 295
Hormones, Brain and Behavior, Third Edition, 2017, 285–327
Author's personal copy
became familiar with a second male by interacting across
a wire-mesh screen. This latter type of exposure is sufcient
for males to develop multicomponent memories of the stim-
ulus animal (Johnston and Peng, 2008). After both of these
experiences, male subjects were then tested for their responses
in a Y-maze with odors and other cues from the two stimulus
animals. Subjects were attracted to the odors and other cues
from a familiar neutral male but were hesitant to approach
the odors of the familiar winner or stay near him (Lai et al.,
2004,2005). These results indicate specic types of response
to two different individuals that have different signicance to
the subject but have equivalent (or at least very similar)
levels of familiarity. These results are based on a relatively
simple set of procedures and experiences compared to the
number and potential complexity of experiences in the wild.
One disadvantage of habituation tasks is that a failure to
show dishabituation to a novel stimulus may have two
different interpretations: (1) the animal did not discriminate,
or (2) the animal did not investigate the stimulus because of
other motivational/emotional reasons, for example, if the
novel stimulus came from an animal that had beaten the
subject in a ght the subject might be reluctant to investigate
an odor from the familiar winner. Thus, failure to discriminate
in habituationdishabituation tasks may be difcult to
Both trained discrimination and the basic habitua-
tiondishabituation tests are primarily useful for determining
whether animals discriminate between and remember indi-
vidual signatures. If one wants to determine other types of
information, such as the content of the memory or the
emotional salience of a memory for an individual, other
methods of testing are necessary. After individuals interact
with one another, what do they remember about each other?
Two types of information may be valuable to remember: (1)
memories that incorporate several different characteristics of
the individual, such as several separate odors or other cues
(sound of voice, visual features) and (2) the emotional or func-
tional signicance of the individual to the subject. Two Primary Polymorphic and Multigenic
There are two primary polymorphic and multigenic complexes
that are important in studies of olfactory communication that
contribute to or determine individual differences in odors.
These complexes are the major histocompatibility complex
(MHC) and the major urinary proteins (MUPs). MHC genes
produce highly polymorphic glycoproteins involved in immune
system function (Yamazaki et al., 1980). MUPs are mostly
produced in the liver and become concentrated in the urine in
mice. MUPs are nonvolatile molecules, but they bind smaller,
volatile molecules and release them slowly (Hurst and Beynon,
2004), thus prolonging the effectiveness of volatile compounds
contained in scent marks (Hurst et al., 1998). Without such
binding the volatile compounds might be lost from the scent
mark relatively quickly (in minutes), which would decrease
the information contained in the scent mark (Hurst and
Beynon, 2004). By being bound to MUPs, these signaling vola-
tiles can emanate from the scent mark for up to 24 h
(Humphries et al., 1999). MUPS also contain individually
distinctive information on their own (Hurst et al., 2001).
In fact, the components involved in individual discrimination
either are the MUPs themselves or the MUPligand complexes,
rather than the volatiles emanating from a scent mark (Nevison
et al., 2003).
Studies have also shown that some subsets of MUPs
found in scent marks of male house mice and rats provide
information about the identity of the sender, essentially
abar code(Gómez-Baena et al., 2014;Hurst et al., 2001).
This information is stable over time and individual as well
as species-specic and appears to be based on the individ-
uals genotype and not its phenotype (Cheetham et al.,
2007;Green et al., 2015;Hurst et al., 2001). Recently, Stow-
ers and colleagues discovered that wild male house mice
expressed their own unique combination and ratio of
MUPs in their urine and that the subsets of MUPs were indi-
vidually distinct (Kaur et al., 2014). For example, Kaur et al.
(2014) discovered that wild male house mice learn their own
subset of MUPs that provide individually distinct informa-
tion and distinguish it from those found in the urine of other
male house mice. Male house mice used this information to
avoid countermarking their own scent marks and those of
male conspecics whose subset of MUPs is similar to their
own. In contrast, male house mice countermarked the scent
marks of conspecics whose composition of MUPs was
unlike their own (Kaur et al., 2014). Male house mice,
however, did not countermark the scent marks of male
conspecics that they had learned to be socially dominant
to them (Kaur et al., 2014). In addition, in a recent study
Green et al. (2015) found that MUPs, which act as a genetic
marker, can also be used by female house mice to select
closely related females as nest partners to help look after
their offspring and that they can do so without being familiar
with those close relatives.
Mice were shown to spontaneously notice differences in the
odors of conspecic mice with different MHC types (Penn and
Potts, 1998b). Norway rats also discriminated body odors
based on slight genetic differences in the MHC (Brown et al.,
1987,1989,1990). These results demonstrate that genes in
the MHC complex do inuence body odors and mice and
rats can discriminate between the odors of individuals with
different MHC types. However, later studies that specically
tested whether mice can use MHC-related odors to recognize
the owner of a given scent found negative results (Cheetham
et al., 2007;Hurst et al., 2005). In these studies, MUPs were
involved in individual discrimination but the MHC was not
(Cheetham et al., 2007). It is important to notice that MHC-
related odors are affected by such factors as status and diet
(Schellinck et al., 1997) and thus are not a good candidate to
offer a stable individual chemical ngerprint (Hurst et al.,
2005). Therefore, there seems to be no clear evidence that the
MHC of an animal offers an individual signature that is
perceived by conspecics. In addition, Green et al. (2015)
found that the vertebrate-wide MHC was not involved in kin
recognition in house mice.
MUPs appear to be a better candidate for individually
distinctive information, at least in mice (Hurst and Beynon,
2004), because the pattern of MUPs expressed by an individual
is unchanged and thus provides a constant individual signature
not affected by factors such as diet or social status (Nevison
et al., 2003). For example, when two individuals share the
296 Communication by Chemical Signals
Hormones, Brain and Behavior, Third Edition, 2017, 285–327
Author's personal copy
same pattern of MUPs, odors of one mouse do not trigger
competitive behaviors in the other (Hurst et al., 2001). MUPs
from other conspecics have also been shown to elicit more
countermarking than self-MUPs (Kaur et al., 2014). Also,
when a puried MUP is added to the urine of a male, he treats
his own modied urine as if it belonged to an intruder male
(Hurst et al., 2001).
A recent study challenges the ngerprint properties of MUPs
(Thoß et al., 2015). In this study the MUP proles were more
similar between brothers than between unrelated males.
However, the MUP prole of a given individual changed over
the course of several weeks, supporting more a dynamic
expressionhypothesis than a barcodehypothesis. Further
studies will need to determine if mice in natural conditions,
i.e., having the ability to investigate the scent marks of neigh-
boring conspecics in a daily basis, adjust their memories of
individual neighbors according to the slowly changing compo-
sition of their scent marks. That is, it is possible that animals
recognize individuals by investigating relatively dynamic
signals instead of xed ones.
Although MUPs have been described and thoroughly
studied in the house mice, similar proteins can also have equiv-
alent semiochemical functions in other species (Gómez-Baena
et al., 2014). A particular MUP, Darcin, also found in the urine
of male house, plays a particular role in chemical communica-
tion. Roberts et al. (2010) found that Darcin stimulated very
rapid learning of spatial cues associated with its location in
male and female house mice. Female mice recalled the location
where they encountered Darcin and later returned to that area,
presumably to locate that male donor. Similarly, male mice
also rapidly learn and remember the location of rival male
scents and drive off these rivals or over-mark their scent marks
(Roberts et al., 2010). Mice that encountered Darcin scent only
once would return to that location 2 weeks later even if the Dar-
cin scent was absent (Roberts et al., 2010).
Some species of male mammals do not produce large quan-
tities of MUPs. Instead, some animals such as Syrian hamsters
produce a proteinaceous compound referred to as Aphrodisin
(Singer et al., 1986). Exposure to Aphrodisin from the vaginal
secretions of female hamsters was sufcient to increase copula-
tory behavior among male hamsters (Macrides et al., 1986). In
a recent paper, Stopková et al. (2010) found that Aphrodisin
like proteins are produced by prostate, preputial, and salivary
glands, and liver and uterus in male and female bank voles,
Myodes glareolus.Stopková et al. (2010) also identied three
novel odorant-binding proteins, Obp1, Obp2, and Obp3, in
urine and saliva. Stopková et al. (2010) suggested that due to
their high sequence homology (i.e., Obp1, Obp2, and Obp3
with Aphrodisin) Obps may have capacity to bind ligands
similar to those bound by Aphrodisin like proteins. If so,
Aphrodisin-like proteins and their ligands binding proteins
are likely to play a role in chemical communication similar
to that of MUPs in attracting mates (Humphries et al., 1999;
Kaur et al., 2014;Roberts et al., 2010). Kin Recognition
The main functions of kin recognition involve nepotism and
inbreeding avoidance. Kin recognition allows an individual
to focus its cooperative behavior toward extended kin and to
focus aggressive behavior toward nonkin as a means of
increasing its inclusive tness. Many animals use chemical
signals from conspecics to differentiate between kin and non-
kin. For example, odors of kin elicit lower levels of agonistic
scent marking (ank marking) in hamsters than did odors of
nonkin (Heth et al., 1998). Recognizing kin also allows an indi-
vidual to avoid mating with closely related individuals,
achieving optimal outbreeding, and thus reducing inbreeding
depression in its offspring (Nunes, 2007).
A special aspect of kin recognition is that of parent
offspring recognition. Mutual recognition between a mother
and her offspring is likely to be benecial to both parties under
most scenarios. Most if not all senses may be involved in paren-
toffspring communication, but olfaction is especially impor-
tant, particularly in rodents (Porter, 1983) but also in other
mammalian groups such as ungulates (Grau, 1976) and carni-
vores (Pitcher et al., 2011). In altricial species, neonates remain
blind and deaf during the rst days, whereas they can already
process scents from the mother and from other pups (Porter,
1983). Rodent pups may rely on olfaction to locate the mothers
nipples as shown by olfactory bulbectomy of pups impairing
nipple location (Porter, 1983). In rabbits, a pheromone
emanating from the mothers ventrum triggers a stereotyped
behavior in the pups that guides them toward the mothers
nipples and suckling (Hudson and Distel, 1983). This volatile
pheromone is sensed by the main olfactory system of the young
(Hudson and Distel, 1986) and it is also involved in reinforced
learning (Coureaud et al., 2006;Schaal et al., 2003). The emis-
sion of such a pheromone is hormonally controlled (González-
Mariscal et al., 1994;Hudson et al., 1990). The source of the
pheromone 2-methylbut-2-enal is milk (Schaal et al., 2003).
Yet, even females that are not producing milk, i.e., estrous
or pregnant, can still produce an olfactory cue that triggers
the same behaviors as those seen following exposure to
2-methylbut-2-enal (González-Mariscal et al., 2016).
Mothers can identify the odors produced by their offspring.
For example, human mothers can distinguish the odors on T-
shirts of their children from those of unfamiliar children of
the same age and sex (Porter and Moore, 1981). Using
a similar methodological approach mothers, but not fathers,
were able to recognize the odor of their children better than
by chance (Ferdenzi et al., 2010). Many studies on goats and
sheep, two precocial species, have established that mothers
use olfactory information to discriminate between their own
newborns and those of other females, although olfaction is
not the only sensory modality involved in such discrimination
(Lévy et al., 2004;Romeyer et al., 1994;Viviers et al., 2014).
Mothers may also mark their offspring for later recognition.
Female Mongolian gerbils use their ventral gland to scent
mark their own pups. Females will scent mark their pups if
they have been licked and groomed, and females will
preferentially retrieve pups marked with their own secretion
compared to unmarked pups (Wallace et al., 1973). Mechanisms of Kin Recognition
There are at least three mechanisms by which an individual can
use odors to recognize kin: familiarity-based recognition,
phenotype matching, and self-referent phenotype matching.
In familiarity-based recognition, animals learn the characteris-
tics of others that they grow up with and later in life treat only
Communication by Chemical Signals 297
Hormones, Brain and Behavior, Third Edition, 2017, 285–327
Author's personal copy
these individuals as kin. This type of learned recognition can
only be useful for actual kin recognition in species in which
litters of one females pups are physically separated from the
litters of other females so that developing pups are only
exposed to kin, and in which future kin interactions are
restricted to the mother and siblings. This type of recognition
can be demonstrated by using a cross-fostering design in which
pups grow up with siblings or foster siblings from another
litter. When these individuals are adult, results showing that
individuals treat nonsiblings that shared their nest like kin indi-
cate that the mechanism underlying such recognition is the
association in the nest.
In phenotype matching, animals learn the phenotype of the
mother and siblings and later compare those phenotypes with
the phenotypes of unfamiliar individuals, treating unfamiliar
individuals as kin if there is a close match between phenotypes.
Phenotype matching in which features of mothers can be used
as a template occurs in Beldings ground squirrels, Spermophilus
beldingi (Mateo, 2009). Young were exposed to odors from an
unrelated mother during early stages of development and
before natal emergence. These young chose as play partners
not only their littermates but also juveniles from the unrelated
mother, indicating that the odors of the unrelated mother were
incorporated into the kin template of the subject young.
A third mechanism is self-referent phenotype matching, in
which an individual compares its own phenotype (e.g., odors)
to that of other conspecics. If there is a high correlation
between these characteristics, other individuals are treated as
kin. This method of kin recognition may be found in species
with multiple paternity or maternity, where full-siblings and
half-siblings may share a nest. It also may occur in species in
which pups of unrelated females can mix early in life. In this
type of situation, recognition by association may not be a reli-
able method to establish kinship. Self-referent phenotype
matching can also facilitate optimal inbreeding and
outbreeding. An example of self-referent phenotype matching
has been shown in female Syrian hamsters. Estrous females
that had been cross-fostered shortly after birth were more sexu-
ally attracted to unfamiliar nonkin than to unfamiliar kin
(Mateo and Johnston, 2000a).
Unless using a complex experimental design, it may be