Current Biology 16, R726–R735, September 5, 2006 ª2006 Elsevier Ltd All rights reservedDOI 10.1016/j.cub.2006.07.068
ReviewEvolution of Plant Breeding Systems
Breeding systems are important, and often ne-
glected, aspects of the natural biology of organisms,
affecting homozygosity and thus many aspects of
theirbiology, includinglevels and patterns ofgenetic
diversity and genome evolution. Among the different
plant mating systems, it is useful to distinguish two
types of systems: ‘sexsystems’, hermaphroditic ver-
sus male/female and other situations; and the ‘mat-
ing systems’ of hermaphroditic populations, in-
breeding, outcrossing or intermediate. Evolutionary
changes in breeding systems occur between closely
related species, and some changes occur more of-
ten than others. Understanding why such changes
occur requires combined genetical and ecological
approaches. I review the ideas of some of the most
important theoretical models, showing how these
are based on individual selection using genetic prin-
ciples to ask whether alleles affecting plants’ out-
crossing rates or sex morphs will spread in popula-
tions. After discussing how the conclusions are
affected by some of the many relevant ecological
factors, I relate these theoretical ideas to empirical
data from some of the many recent breeding system
studies in plant populations.
Breeding systems have attracted interest for many
reasons. Their evolution and change is particularly ev-
ident in plants. Flower size differences are easily visi-
ble, and it has long been noticed that small-flowered
plants that quickly produce many seeds without polli-
nator visits are often closely related to species with
more conspicuous flowers that either set seed only
after pollinator visits or are self-incompatible. The
change from outcrossing to inbreeding is a repeated
evolutionary transition, occurring in many unrelated
genera. Many self-incompatible species have self-
fertile relatives (for example [1–4]). For instance, the
closest relatives of the inbreeder Arabidopsis thaliana
are self-incompatible (see Figure 1, top), and there are
many similar examples in the Brassicaceae  and
other plant families.
The directionality of such changes is often evident
from the relationships of the species. Phylogenetic
analyses often infer outcrossing systems in ancestral
eration the low likelihood of de novo evolution of sys-
tems such as self-incompatibility or separate sexes
(see below). Thus, the bottom part of Figure 1 shows
frequent loss of outcrossing systems and rarer evolu-
However, adaptations promoting outbreeding, includ-
ing the evolution of separate sexes, have also evolved
many times, which is briefly discussed below.
Breeding systems affect many important aspects
of a population’s ecology and evolution, including
whether individuals are homozygous or heterozygous.
A change to inbreeding rapidly increases homozygote
frequencies, and thus individuals will express reces-
sive or partially recessive deleterious mutations, and
suffer lower survival probabilities and reduced fertility:
‘inbreeding depression’. In the long term, these effects
on survival and fertility lower mutation frequencies,
and thus lead toreduced inbreeding depression (purg-
ing) . Another effect of homozygosity is to allow ex-
pression of recessive advantages. A larger range of
new mutations is therefore affected by selection, com-
pared with outcrossing populations in which some de-
also lowers effective population size — a population
of homozygotes has an effective size half that of an
outcrossing diploid with the same number of individ-
uals — implying greater effects of genetic drift, poten-
tially reducing the ability of natural selection to elimi-
nate disadvantageous mutations and increase the
frequency of advantageous ones. Inbreeding popula-
tions are thus predicted slowly to deteriorate by fixing
mildly deleterious mutations [10,11]. They will also
tend to lose neutral genetic diversity.
Understanding mating system evolution is thus an
important part of evolutionary biology. In addition to
the ecological effects of breeding systems just men-
tioned, ecological factors must evidently be among
the major influences on mating system evolution. For
example population fragmentation might cause low
availability of conspecific potential mates, or low avail-
ability of pollinators to ensure cross-fertilisation be-
tween individuals, and a similar situation may exist in
species with a weedy or colonising lifestyle (which,
like low density, may affect the effective availability
of mates). Mating system evolution is also evidently
likely to be affected by resource availability. Along
with the supply of suitable pollinators, resources may
limit plants’ ability to produce fruits and seeds, with
important effects on life history evolution and conse-
quently on the mating system.
For instance, annual plants must produce seeds be-
fore dying, and even low quality offspring make some
contribution to fitness, defined as the representation
of an individual’s genes in the progeny generation. It
is thus not surprising that, even though self-fertilisa-
tion often leads to offspring of low quality because
of inbreeding depression, many annuals have low
This articleisdedicated tothememoryofDavid Lloyd, whodied
on May 30, 2006. He made many important contributions to
botany, and his theoretical work established clearly the
importance of individual selection for understanding breeding
Institute of Evolutionary Biology, School of Biological Sciences,
University of Edinburgh, Ashworth Lab. King’s Buildings, West
Mains Road, Edinburgh EH9 3JT, UK.
outcrossing frequencies in nature, especially as many
are weeds, for which self-fertilisation provides the
potential advantages of freedom from need for con-
specifics, and from pollinating animals (‘reproductive
sibility of producing offspring that will contribute to fit-
allocation of resources to reproduction, relative to an
size, possibly affecting pollinator behaviour so that
transfer of pollen between different individuals, and
thus outcrossing, becomes less likely. The evolution
of large plant size may thus be accompanied by other
female stages of flowering, or even unisexuality of the
whole plant — or self pollen acceptance — including
chemical recognition systems (‘self-incompatibility’)
that reject pollen of the same ‘incompatibility type’ as
the maternal plant, and thus prevent self-fertilisation
and also some matings between close relatives.
The evolution of mating systems raises some partic-
ularly intriguing questions. It is clear that both genetic
and ecological factors are important, but there has
been disagreement about their relative importance.
The issues are greatly clarified by understanding that
selection acts on individuals in the short term, causing
breeding system changes, probably often in relation to
environmental changes — for example, reproductive
assurance is likely to be important in colonising
plants — and also has long-term consequences of
such changes affecting population survival. Fitness
differences between genotypes with different breed-
ing systems, which depend on their ecological circum-
stances, lead to evolutionary changes (short-term, or
individual selection, effects). Because a population’s
breeding system influences its evolution and genetic
diversity, the long-term consequences outlined above
may also be important. Mating system patterns in
present day populations must therefore reflect the
combined effects of the kinds of changes that evolve
in different ecological situations — the input of breed-
ing system changes — together with the survival or
systems, again strongly influenced by their ecological
Theories for Breeding System Changes
From the outline of some of the factors affecting mat-
ing systems given above, it is clear that, even ignoring
long-term consequences, a complete theory of breed-
ing system changes will be very difficult, because it
must include the selective forces that affect mating
systems themselves, along with those affecting other
aspects of plant fitness. How resources devoted to
imal pollinators — will interact with how outcrossing
versus self-fertilisation affect offspring quality (which
also depends on resources affecting fruit and seed
output and quality). At the same time, resource alloca-
tion to reproduction evolves in competition with de-
mands for allocation to functions increasing survival
and growth, in a life-history evolution context, and
this may also involve aspects of competition between
conspecific seedlings and further inter-species inter-
actions in the process of seed dispersal. No theory
combining life-history and resource allocation is avail-
able, and a complete evolutionary model would prob-
ably be too complex to be useful. Thus a general pre-
dictive theory may be an unrealistic aim.
Instead, simplifying approaches have given us
a fairly good understanding of several major factors
in mating system evolution, despite this complexity.
The advantage of this approach is that it generates
testable predictions about single observable factors,
for example, that low pollination levels tend to select
for an ability to self-fertilise. Such predictions can be
tested in the field and experimentally. A particularly
Arabidopsis lyrata ssp. petraea
Inbreeding, for example
by loss of self-incompatibility
Advantages: gene transmission, reproductive assurance
Figure 1. Evolution of outcrossing and in-
Closely related outcrossing and inbreed-
ing species in the genus Arabidopsis
(upper part of figure). The lower part of
the figure shows the different frequencies
of transitions between outcrossing and
favourable circumstance for testing for selection (fit-
ness differences) occurs when a population is poly-
morphic, and, remarkably, breeding systems are often
polymorphic. Comparative tests, based on correla-
tions between breeding systems and particular eco-
logical situations, are also possible, because similar
breeding system changes have occurred repeatedly.
As will be seen below, the evolution of inbreeding
versus outcrossing, and even the initially puzzling evo-
lution of separate sexes (dioecy) from hermaphrodit-
onthe sameplant)are bothsimpler tounderstand than
the evolution or maintenance of sexuality. A first, his-
torically very helpful, insight was that mating systems
are properties of individuals, not of populations. Thus
an individual selection approach can be used to think
about mating system evolution. Itis particularly helpful
as homozygotes can then be ignored, greatly simplify-
ing the genetics, yet dealing with parts of the allele fre-
quency range that tell us whether a population will
change (whether the mutation will spread in the popu-
lation). The mutation’s fate is often determined by
whetherthe phenotype of the rare heterozygotes gives
higher fitness than the type initially present in the pop-
ulation; whether it will become fixed can similarly be
studied by focusing on populations in which it is so
common that homozygotes for the initial allele can be
neglected . These ‘phenotypic selection models’
are based on genetical principles, and are correct —
they give identical results to those obtained by the
more laborious calculations of genotype frequency
changes — only under certain restrictions [14–16].
When changes evolve, the effects on the population
outlined above are slow, because they rely on disad-
as will be seen, the individual selection increased fit-
ness through reproduction of a selfing mutation can
be very large, so that the long-term effects cannot pre-
vent its spread throughout the population (‘fixation’).
We can thus study separately the input of breeding
system changes, using an individual selection ap-
proach, and consider effects on the population’s
long-term survival and measures of ‘population fit-
ness’ (such as whether it can produce enough seeds
to persist) only as potentially affecting the mainte-
nance of established breeding systems.
The Major Categories of Mating Systems
Mating systems descriptions involve three important
aspects: first, whether sexual reproduction occurs at
all; second, whether individuals have both sex func-
tions (‘cosexual’, including hermaphroditic and mon-
oecious plants) or whether some or all are unisexual
males or females (dioecy) (reviewed in ); and third,
whether cosexual individuals are self-compatible or
not and, if compatible, what natural outcrossing rates
are inferred using genetic markers or other ap-
proaches [18,19]. Hermaphrodite and monoecious
species are ‘sexually monomorphic’, while dioecious
populations are ‘sexually polymorphic’, having sepa-
rate males and females (or other situations mentioned
below, see Figure 2) determined by a genetic sex-
determination system , or sometimes environ-
mentally [21,22], or partially so (for example [23,24]).
As shown in Figure 2, the genus Silene includes co-
sexuals and dioecious species, with many hermaphro-
dite species and also ‘gynodioecious’ species having
cies have monoecious relatives , among many
other examples [27–29]. There are also a few sexually
polymorphic species with cosexuals and males (‘an-
drodioecious’), all with close dioecious relatives
([23,30], reviewed in ).
flower morphology is sometimes misleading, because
flowers of plants that have recently evolved dioecy
may appear cosexual . Functional mating systems
in cosexuals are much more difficult to determine in
the wild, as matings cannot be directly observed
when pollinators, or wind or water pollination, are
involved, and often only female reproduction is
Males + Females
of male fertility
hermaphrodite by loss of
Figure 2. Evolution of two separate sexes
from hermaphrodites, showing that at
least two evolutionary steps are required.
Wider lines indicate commoner transi-
tions, and dotted lines show changes
that can occur once dioecy has evolved
(reversion to hermaphroditism, and evolu-
tion of androdioecy). The figure illustrates
the sex systems known within the genus
Silene. The bladder campion (S. vulgaris)
is an example of cytoplasmic male sterility
(CMS) while the white campion, S. latifolia,
rility (hermaphrodites carrying a Y chro-
mosome) occasionally arise in S. latifolia;
an example is shown at the bottom left-
hand side of the figure. Androdioecy is
not known in the genus Silene.
accessible. Even functional tests of male fertility can
be misleading, because seed set does not guarantee
that viable progeny will be produced in natural condi-
tions . Inbreeding can, however, be quantified
using genetic markers. Markers can be used to esti-
mate frequencies of homozygote and heterozygote
genotypes (which, assuming a population at equilib-
rium under a defined mating system, such as a given
frequency of ovule self-fertilisation, can yield an esti-
mate of this ‘selfing rate’).
Recent changes in frequencies of inbreeding can
also be detected . Alternatively, sets of seed par-
ing rate estimates without assuming that equilibrium
has been reached . From a large body of such es-
timates, as well as from earlier careful observations
in the field, it has been established that many annual
plants have low outcrossing rates (as predicted
above), while longer-lived plants tend to be more
outcrossing . Different outcrossing frequencies
among cosexuals can be caused by timing differences
in the male and female phases of flowers (or of whole
plants in monoecious species) or by presence or ab-
sence of self-incompatibility. It is also clear that a sub-
stantial minority of flowering plants have intermediate
outcrossing rates [13,18,36].
Genetic Models of Mating System Evolution
Models of mating system evolution have emphasised
genetic effects, even though, as already explained,
ecological circumstances, such as pollinator abun-
dance or plant density, must often be important. Their
complexity and variety, however, creates difficulties in
developing any general theories. In contrast, there are
some generally applicable strong genetic effects. Al-
though many organisms are outcrossing, theory tells
us that, like asexuality, inbreeding has a large advan-
tage due to the increased transmission of gametes to
the next generation — it gives an advantage to alleles
that increase the rate of self-fertilisation or other in-
breeding . For selfing, the effect is smaller than
the two-fold advantage of asexual reproduction, un-
less the allele causing increased selfing also causes
reproductive resources to be re-allocated from pollen
production to the extent that female fertility is in-
creased. Even with no such effect, however, an allele
causing complete selfing has an initial 50% advantage
when introduced into an outcrossing population. This
advantage arises because selfing individuals transmit
two gametes to their own seeds, and may in addition
contribute gametes if their pollen fertilises ovules of
other plants. These simple models predict that com-
plete selfing should evolve, unless some strong selec-
tive force acts against inbreeding.
Models involving individuals with different selfing
rates must take into account this genetic force, along-
side the influences of ecological circumstances that
may affect the evolutionary outcome. If the density of
conspecific plants, or of pollinators, is low, selfing
has an additional potentially strong advantage, as
of producing progeny either through its own seeds, or
via its pollen generating outcrossed seeds on other
maternal plants. This makes the maintenance of out-
crossing even harder to explain.
Other circumstances can, however, reduce the
advantage of mutations causing a change from out-
crossing to inbreeding. A highly selfing individual
may sire no outcrossed seeds on other plants (com-
plete ‘pollen discounting’). This complete loss of
male fitness via outcrossing abolishes the advantage
to selfing ; lesser discounting reduces the advan-
tage . If selfing occurs by pollinators moving from
flower to flower on the same plant (‘geitonogamous’
pollination), each pollinator visit causing self-pollina-
tion of a flower is a missed potential outcross pollina-
tion event. Pollen discounting is likely when selfing is
due to a mutation making the flowers small, so that
anther–stigma separation is reduced and self pollen is
deposited. Such flowers may produce less pollen, or
pollen may be less available to pollinators (sometimes,
flowers remain closed or only partially open). Smaller
flowers will generally also attract fewer pollinator
visits. The reduced outcrossing lowers contributions
to the individuals’ fitness through pollen; they may
then have only a small fitness advantage relative to
outcrossing individuals, or no advantage. Similarly,
‘competing self-fertilisation’, in which pollinators can
bring either self pollen or pollen from other plants, is
less advantageous than ‘delayed self-fertilisation’, in
which ovules are fertilised by self pollen only after out-
crossing opportunities have ended [13,39].
The general conclusion that selfing often has a large
genetic advantage is important because it highlights
the need for a strong disadvantage, to account for the
disadvantage. Theoretical models show that strong in-
breeding depression — with, in the simplest models,
outbred offspring having more than twice the survival
rate of inbred ones — can prevent the invasion of an
outcrossing population by a selfing mutant [13,40,41].
These results have prompted many experiments that
versity of plants, that inbred progeny often have low
survival and fertility, with a tendency for lesser effects
in more inbred populations (reviewed in ). In some
plant populations, inbreeding depression can be se-
vere throughout life (for example [42–45]), as is notori-
ous in some conifers . Inbreeding depression is,
, while pollen discounting adds a disadvantage to
selfing, effectively increasing inbreeding depression.
More Realistic Models and Intermediate
The simple phenotypic models for the evolution of
outcrossing outlined above ignore many ecological
details that may be important in particular situations.
For instance, many different situations may lead to
evolution of intermediate outcrossing rates (reviewed
in [19,48]). These include situations with competing
self-fertilisation when higher self-fertilisation lowers
the fraction of outcrossed ovules  and the some-
what similar mass action model , seedling com-
petition , biparental inbreeding , trade-offs
between female and male functions [52,53], and pol-
len limitation, which can lead to a low, but non-zero,
outcrossing, as commonly observed .
Since the development of these models, they have
been successively improved by introducing greater
genetic realism, including models of the genetic basis
tion of a fixed relative fitness of progeny produced by
selfing [55–57]. For instance, intermediate outcrossing
rates evolve if inbreeding depression either increases
in successive generations, as a result of a contribution
from overdominant loci , or varies temporally .
The genetic details of precisely how outcrossing
systems can change are also important, because
they determine whether the necessary genetic and
phenotypic changes are likely to evolve, or difficult to
evolve, thus affecting the rates of changes from out-
crossing to inbreeding, and vice versa, which may be
flower to one with much higher self-fertilisation, such
as the example of small flower size just mentioned,
but even this case is not simple, as it must be remem-
bered that there will also be other effects on fertilisa-
tions achieved; changes in anther–stigma separation
can also probably be selected to best manage pollen
removal by pollinators, and delivery when they visit
the next flower, and to reduce interference between
male and female functions . This is an example of
how selection probably very often acts on flower mor-
phology — including the timing of the phases of male
and female functions — to increase outcrossing suc-
cess via pollen, by avoiding pollen discounting and
minimising interference between male and female
functions. Flowers represent the integrated outcome
of selection of this kind, along with selection affecting
the proportions of outcrossed and selfed seeds.
Genetic details are evidently important in the evolu-
tion of self-incompatibility systems. An initial incom-
patibility allele, causing all pollen grains of its carrier
plants to be recognised and rejected by its stigmas
(as when the allele is expressed sporophytically in the
plant producing the pollen), will have the simple effect
of causing self-incompatibility, much as in the simpli-
fied models above that assume a mutation altering
allele expressed by the male gametophytes (pollen
grains) will have different evolutionary dynamics. In
a plant heterozygous for the mutant (incompatible)
allele, half the pollen will carry the other allele. Thus
ible, but, at best, increases its outcrossing rate. This
advantage to the diploid parent is at the cost of a
great disadvantage to the self-incompatible pollen ge-
notype. Consequently, for such an allele to invade a
population, inbreeding depression must be very large
[55,61,62]. Interestingly, these models predict that
non-functional alleles may persist alongside alleles
crossing rates may be intermediate, and indeed self-
compatible individuals may coexist with incompatible
ones in plant populations .
tems evolve when several different genetic changes
are necessary. Despite their name, self-incompatibility
systems do not involve a self-recognition system, but
are based on genes encoding proteins with receptors
and ligands that recognise one another (reviewed in
). The cross-incompatibility between some individ-
incompatibility systems, which maintains many alleles
in populations of plants with the homomorphic type of
system, in which flower morphology is similar for all in-
in a species that lacks incompatibility is likely to be
a very rare event. Separate genes encode the pollen
and pistil recognition proteins in both types of self-
incompatibility system whose genes have been identi-
fied: the sporophytic system in Brassica and other re-
lated species [65–67], and the gametophytic systems
with pistil S-RNases and pollen F-box proteins in the
Solanaceae, Rosaceae and Antirrhinum (reviewed in
Homostyle, with anthers
and stigma close together
a homostyle species
Primrose (Primula vulgaris)
Figure 3. Heterostylous systems in the
genus Primula, and the breakdown of het-
Heterostyled flowers of the primrose
Primula vulgaris, showing the self-com-
patible ‘homostyle’ flower type (left-hand
side of the figure). On the right is shown
a homostyled species whose distribution
of which is confined to regions where
pollination is often unreliable, due to bad
). The problem of how new alleles can arise, given
patibility loci, is not yet solved [69,70].
The evolution of heterostylous systems with two or
three floral morphs (Figure 3), often with cross-incom-
patibility among plants of the same morph [71,72] in-
volves coordinated development of different flower
parts. Again, successive changes at linked loci must
have been involved, to avoid producing a high propor-
tion of progeny with incorrect arrangements of the
anthers and stigmas [73,74].
The trend to include greater realism has also led to
study of relevant ecological factors. For example,
population subdivision can lead to intermediate out-
crossing rates . Under certain conditions, interme-
diate outcrossing may also arise when low amounts
of compatible pollen are received, which is likely to
be important for the evolution and maintenance of
self-incompatibility . The importance of reproduc-
tive assurance has already been mentioned, and evi-
dence for its action is accumulating. In populations
of Collinsia verna, a self-compatible plant with gener-
ally low outcrossing rates and delayed selfing, out-
crossing rates were lower in years when pollinator ser-
vice was poor , and within-flower self-pollination
was higher, leading to progeny suffering some
inbreeding depression, but not of high enough magni-
tude to negate the advantage of selfing.
Heterostylous systems have repeatedly broken
down to yield more inbreeding populations [71,78].
Reproductive assurance is probably sometimes the
of Primula vulgaris, pollinator service, reflected in
numbers of seeds per fruit, and fruits per plant, varied
from year to year (Figure 4). In these populations, as
shown in Figure 3, some plants are self-compatible
‘homostyles’, which can self-fertilise within flowers,
unlike the usual two incompatible Primrose ‘morphs’
[79,80]. In two good pollinator years, all plant morphs
had fairly uniform, high seed output, but in one year,
when the weather was unusually rainy at the flowering
season, the self-fertile morph had much higher seed
output than the incompatible morphs .
Evolution of Unisexuality
Like the maintenance of outcrossing, the evolution of
unisexuality presents difficult puzzles. First, like the
evolution of self-incompatibility systems, at least two
successive stages are required, first evolving females
or males — generating gynodioecious or androdioe-
cious populations, respectively — and then changing
the cosexuals into males or females, in one or more
Batcombe 1Batcombe 2
Wanstrow 1, 2
Eastcombe 1 Eastcombe 2
Mean seeds/capsule Mean seeds/capsuleMean seeds/capsule
Batcombe 1Batcombe 2
Wanstrow 1, 2
Eastcombe 1Eastcombe 2
Batcombe 1 Batcombe 2
Wanstrow 1, 2
Eastcombe 1Eastcombe 2
flowers of the same morph experimentally pollinated to ensure excess pollen receipt (supplemented), and with naturally pollinated
flowers of homostyle plants. Only the long-styled morph supplementation results are shown, because supplementation of flowers
of the short-styled morph is difficult, but the few results were very similar to those shown. In 1982 and 1984, seed numbers per fruit
under natural pollination were slightly below those achieved by supplemented flowers (the grey bars are only slightly below 1), indicat-
ing good pollination levels. Homostyles nevertheless often had somewhat higher seed set per fruit (and per plant) than the self-incom-
patible long-styled morph (black bars often higher than 1). In 1983, the flowering season was unusually rainy. Pollination was evidently
poor, as pin plants’ seed numbers under natural pollination were low, relative to flowers with pollen added (grey bars often less than 1).
In that year, homostyle plants’ seed output greatly exceeded that of pin plants (black bars much higher than 1). Thus the homostyle
flowers benefited from reproductive assurance in 1983.
ual females and males in dioecious populations, with
no male fertility, must have a strong disadvantage rel-
ative to cosexual plants, unless something compen-
sates for the fertility loss.
The second difficulty is less severe for cytoplasmic
male-sterility (CMS) than nuclear sterility factors. The
spread of CMS mutations is not affected by the loss
of male fertility, so these mutations can invade cosex-
ual populations, given only a slight female fertility ad-
vantage; CMS factors are thus classic ‘selfish’ genetic
likelyif unisexuals reallocate resources to female func-
tions, which is expected whenever developmental
trade-offs occur between different functions that
draw on the same pool of resources. Evidence for
this has accumulated from many ‘gynodioecious’ pop-
ulations, in which females and hermaphrodites are
polymorphic and can be compared . Populations
with CMS polymorphisms may then be invaded by nu-
clear genes restoring male fertility, leading to complex
genetic polymorphisms with female frequencies that
vary greatly among populations , as theoretically
The chance that unisexual females receive pollen is
evidently important for breeding system evolution, so
population subdivision interacts with the genetic basis
ofmalesterility. Recolonisationafterextinctionof local
populations can either increase or decrease female
frequencies in systems with mixed cytoplasmic/nu-
clear male-sterility, depending on the extinction rate
and dominance of male fertility restorers; although fe-
males are more often absent by chance from small
populations , restorer genes are also often absent,
erate [89,90]. Some populations are polymorphic for
hermaphrodites and males (androdioecy), probably
due to females of dioecious species evolving some
ability to produce pollen in response to pollination lim-
itation in colonising situations [91–93].
Two possible factors can compensate for loss of
male functions caused by nuclear sterility genes: in-
breeding depression favouring unisexuals, and trade-
offs increasing the remaining sex function. Recent
work is starting to uncover negative genetic correla-
tions (trade-offs) in gynodioecious species between
hermaphrodites’ pollen and fruit production . Real-
ised seed or pollen production often depends on envi-
ronmental quality. Many examples are known in which
female fertility of cosexual plants appears more sus-
ceptible to limitation by low environmental quality
than male fertility . In many plants male reproduc-
tion begins at a younger age than seed production
, or males grow larger , though differences are
not invariably seen. In monoecious species, seed pro-
duction may be more labile than pollen production
, and this is also found in ‘subdioecious’ species,
such as the spindle tree Euonymus europeus , in
which females coexist with cosexuals that often have
low fruiting ability (‘inconstant males’, see ), partic-
often yield males while plants develop as females only
in good conditions [21,22].
Animal pollinators are also likely to be a very impor-
tant part of the environmental influences. If pollen is an
important reward, females may be less attractive than
males, and less re-visited. On the other hand, male fit-
ness relies on pollinators visiting female plants, so that
selection may favour lower rewards in the flowers of
male plants. An evolutionary balance is to be expected
between pressures increasing male attractiveness, in-
creasing the fitness of males in competition with con-
specific males, and decreasing it, inducing pollinators
to leave a male plant, and, ideally next visit a female.
Given the complexity of factors that can influence the
success of different mating systems, a general predic-
tive theory seems unlikely. We can nevertheless iden-
tify the major variables, outcrossing rates and their
consequences for the genetic phenomenon of in-
breeding depression, and allocation of resources to
pollination, including attracting and rewarding polli-
nating animals, and nourishing and protecting seeds.
Ecological forces are clearly important, often deter-
mining whether a species’ mating system is selected
to change. Outcrossing rates depend on plant density,
and on the density and nature of pollinators in animal
pollinated populations, and seed production in many
plants is limited by pollen supplies, since more seeds
are often produced when natural pollination is supple-
mented experimentally, as in the two studies cited
above [77,81]. The mode of pollination affects the se-
above, many ecologically plausible situations can
allow intermediate selfing rates.
It is also firmly established that the evolutionary ef-
fects that drive mating system evolution are short-
term, and that characteristics that lower population
survival chances often cannot be opposed. The study
of mating system evolution has uncovered several ex-
amples. Self-fertilisation has often evolved, despite
the inbreeding depression it causes. It is not yet clear
whether this leads to extinction of selfing species or
populations. Although some evidence suggests a defi-
cit ofold selfinglineages, it isnotconclusive about this
question . It is difficult to determine how long self-
ing lineages can persist, because phylogenetic analy-
ses that compare numbers of transitions from out-
crossing to selfing, and vice versa, mostly ignore the
evidence that the former is a simple change that may
readily occur (see above), whereas the evolution of
an outcrossing mechanism where none previously ex-
isted must often be extremely unlikely, particularly
when several successive changes are required (for in-
stance, evolution of self-incompatibility or of dioecy,
The evidence that self-incompatibility systems are
maintained over long evolutionary times implies that
changes to selfing rarely reverse . This is partly
because of the difficulty of evolving incompatibility
systems, but also because inbred populations may of-
ten evolve low inbreeding depression (if they persist
long enough). It is difficult to determine how long ago
present-day selfers changed from outcrossing to self-
ing. This could be much more recent than the time of
separation from the closest extant relative species,
as is thought to be true for A. thaliana’s change to
two kinds of disadvantage. Unisexual females are evi-
dently exposed to a risk of failure to receive pollen, so
that fruit-set may fail. This can explain the many exam-
ples of reversion to cosexuality, including the andro-
dioecious species mentioned earlier. It may also lead
to extinction of dioecious species, perhaps partially
explaining why dioecious taxa include fewer species
than sister taxa . A longer-term problem may
come from the evolution of sex chromosomes. As se-
lection eliminates X–Y recombination, Y-linked genes
mayaccumulate deleteriousmutations, and eventually
be deleted , leading to reduced male survival and
lower success of pollen carrying Y chromosomes.
These examples show very clearly that it is simplistic
to conclude that competition leads to beneficial out-
comes, and that the theory of evolution makes no
We now have a clear body of individual selection
based theories and models, well integrated with ap-
proaches and techniques developed for testing ideas
in experiments, field studies, and comparative analy-
ses. It is to be hoped that more of the most important
assumptions and predictions will be tested in the next
few years. The many plants with intermediate out-
crossing rates offer excellent opportunities for com-
paring the selective advantages and disadvantages
of outcrossing. Studies of pollination, including com-
abundance, should help to clarify cases of thresholds
when outcrossing cannot be maintained, and show
how competition for pollinators, or the presence of
other species flowering at the same time, may influ-
ence selection for inbreeding versus outcrossing, in
turn affecting flower attractiveness and the allocation
to rewards versus other plant functions. More tests
of whether inbreeding depression is stonger under
competition are needed, as this is assumed in some
The puzzle of how new self-incompatibility alleles
arise remains to be solved, as do the genetic details
of loss of incompatibility, and whether (and how) it
can beregained once lost byaspecies. More evidence
is also needed on the extent to which increased male
function in cosexuals comes at a cost of lowered fe-
male fertility (and how much allocation to pollinator at-
traction may reduce pollen and ovule production); this
is likely to illuminate the evolution of pollen/ovule ra-
tios (which may affectpollination), as well asthe evolu-
tion of dioecy and of sex chromosomes.
1. Lloyd, D.G. (1965). Evolution of self-compatibility and racial differ-
entiation in Leavenworthia (Cruciferae). Contrib. Gray Herbarium
Harv. Univ. 195, 3–134.
2. Brauner, S., and Gottlieb, L.D. (1987). A self-compatible plant of
Stephanomeria exigua subsp. coronaria (Asteraceae) and its rele-
vance to the origin of its self-pollinating derivativeS. malheurensis.
Syst. Bot. 12, 299–304.
its presumed progenitor M. guttatus. New Phytol. 112, 269–279.
4. Wyatt, R., Evans, E.A., and Sorenson, J.C. (1992). The evolution of
ceae). VI. Electrophoretically detectable genetic variation. System-
atic Botany 17, 201–209.
III. Cruciferae. Heredity 9, 52–68.
6.Goodwillie,C. (1999).Multiple origins ofself-compatibilityin Linan-
from internal-transcribed-spacer sequence data. Evolution 53,
7.Schoen, D.J., L’Heureux, A.-M., Marsolais, J., and Johnston, M.O.
(1997). Evolutionary history of the mating system in Amsinckia
(Boraginaceae). Evolution 51, 1090–1099.
8.Byers, D.L., and Waller, D.M. (1999). Do plant populations purge
their genetic load? Effects of population size and mating history
on inbreeding depression. Annu. Rev. Ecol. and Systematics 30,
9.Charlesworth, B. (1992). Evolutionary rates in partially self-fertiliz-
ing species. Amer. Nat. 140, 126–148.
a large metapopulation. Genetics 160, 1191–1202.
Gle ´min, S. (2003).How aredeleteriousmutationspurged?Drift ver-
sus nonrandom mating. Evolution 57, 2678–2687.
12.Morgan, M.T., Schoen, D.J., and Bataillon, T. (1997). The evolution
of self-fertilization in perennials. Am. Nat. 150, 618–638.
13.Lloyd, D.G. (1979). Some reproductive factors affecting the selec-
tion of self-fertilization in plants. Am. Nat. 113, 67–79.
14.Lloyd, D.G. (1977). Genetic and phenotypic models of natural se-
lection. J. Theoret. Biol. 69, 543–560.
15. Charlesworth,B.,andCharlesworth,D.(1978).A modelfor theevo-
lution of dioecy and gynodioecy. Am. Nat. 112, 975–997.
16.Maurice, S., Couvet, D., Charlesworth, D., and Gouyon, P.-H.
(1993). The evolution of gender in hermaphrodites of gynodioe-
cious populations: a case in which the successful gamete method
fails. Proc. Royal Soc. Lond., B. 251, 253–261.
17. Barrett, S.C.H. (2002). The evolution of plant sexual diversity. Nat.
Rev. Genet. 3, 274–284.
18. Vogler, D.W., and Kalisz, S. (2001). Sex among the flowers: the dis-
tribution of plant mating systems. Evolution 55, 202–204.
19.Goodwillie, C., Kalisz, S., and Eckert, C.G. (2005). The evolutionary
enigma of mixed mating systems in plants: occurrence, theoretical
explanations, and empirical evidence. Annu. Rev. Ecol. and Sys-
tematics 36, 47–79.
20.Charlesworth, D., and Guttman, D.S. (1999). The evolution of di-
oecy and plant sex chromosome systems. In Sex Determination
in Plants, C.C. Ainsworth, ed. (Oxford: BIOS), pp. 25–49.
21. Condon, M.A., and Gilbert, L.E. (1988). Sex expression of Gurania
And Psiguria (Cucurbitaceae) - neotropical vines that change sex.
Am. J. Bot. 75, 875–884.
22.Zimmerman, J.K. (1991). Ecological correlates of labile sex expres-
sion in the orchid Catasetum viridiflavum. Ecology 72, 597–608.
24.Yamashita, N., and Abe, T. (2002). Size distribution, growth and
inter-year variation in sex expression of Bischofia javanica, an
invasive tree. Ann. Bot. 90, 599–605.
Roy. Soc. Lond. B. 263, 409–414.
evolution of dioecy from monoecy in Sagittaria latifolia (Alismata-
ceae). Proc. Roy. Soc. Lond. B. 271, 213–219.
27.Renner, S.S., and Ricklefs, R.E. (1995). Dioecy and its correlates in
the flowering plants. Am. J. Bot. 82, 596–606.
28.Renner, S.S., and Won, H. (2001). Repeated evolution of dioecy
from monoecy in Siparunaceae (Laurales). Systematic Biol. 50,
29.Heilbuth, J.C. (2000). Lower species richness in dioecious clades.
Am. Nat. 156, 221–241.
30.Rieseberg, L.H., Philbrick, C.T., Pack, P.P., Hanson, M.A., and
tions of Datisca glomerata (Datiscaceae). Am. J. Bot. 80, 757–762.
31.Pannell, J.R. (2002). The evolution and maintenance of androdio-
ecy. Annu. Rev. Ecol., Evol. Systematics 33, 397–425.
Same Species (London: John Murray).
33. Verdu, M., Montilla, A.I., and Pannell, J.R. (2004). Paternal effects
on functionalgender account for cryptic dioecy in a perennial plant
Proc. Roy. Soc. Lond. Series B: Biological Sciences 271, 2017–
34. Enjalbert, J., and David, J.L. (2000). Inferring recent outcrossing
rates using multilocus individual heterozygosity: application to
evolving wheat populations. Genetics 156, 1973–1982.
35. Ritland, K. (1993). Estimation of mating systems. In Plant Genetics
and Breeding, Vol. Part A, S.D. Tanksley and T.J. Orton, eds.
(Amsterdam: Elsevier), pp. 289–301.
Baker, H.G. (1959). Reproductive methods as a factor in speciation
in flowering plants. Cold Spring Harb. Symp. Quant. Biol. 24, 177–
Fisher, R.A. (1941). Average excess and average effect of a gene
substitution. Ann. Eugen. 11, 53–63.
Nagylaki, T. (1976).A modelfor the evolutionofself fertilizationand
vegetative reproduction. J. Theoret. Biol. 58, 55–58.
Lloyd, D.G. (1992). Self- and cross-fertilization in plants. II. The se-
lection of self-fertilization. Int. J. Plant Sci. 153, 370–380.
Lande, R., and Schemske, D.W. (1985). The evolution of self-fertil-
ization and inbreeding depression in plants. I. Genetic models.
Evolution 39, 24–40.
Charlesworth, B. (1980). The cost of sex in relation to mating sys-
tem. J. Theoret. Biol. 84, 655–671.
Willis, J.H. (1993).Effectsofdifferentlevels ofinbreedingon fitness
components in Mimulus guttatus. Evolution 47, 864–876.
Herlihy, C.R., and Eckert, C.G. (2002). Genetic cost of reproductive
assurance in a self-fertilizing plant. Nature 416, 320–323.
Ramsey, M., Vaughton, G., and Peakall, R. (2006). Inbreeding
avoidance and the evolution of gender dimorphism in Wurmbea
biglandulosa (Colchicaceae). Evolution 60, 529–537.
Schneller, J.J., and Holderegger, R. (1997). Vigor and survival of in-
bred and outbred progeny of Athyrium filix-femina. Int. J. Plant Sci.
Savolainen, O., Ka ¨rkka ¨inen, K., and Kuitinien, H. (1992). Estimated
numbers of embryonic lethals in conifers. Heredity 69, 308–314.
Porcher, E., and Lande, R. (2005). Reproductive compensation in
the evolution of plant mating systems. New Phytologist 166, 673–
Johnston, M.O. (1998). Evolution of intermediate selfing rates in
plants: pollination ecology versus deleterious mutations. Genetica
Holsinger, K.E. (1991). Mass action models of plant mating sys-
tems: the evolutionary stability of mixed mating systems. Am.
Nat. 138, 606–622.
Lloyd, D.G. (1980).Demographicfactorsandmatingpatternsinan-
giosperms. In Demography and Evolution in Plant Populations,
O.T. Solbrig, ed. (Oxford: Blackwell), pp. 67–88.
Uyenoyama, M.K. (1986). Inbreeding and the cost of meiosis: the
Evolution 40, 388–404.
infinite size. II. Protectedness of a biallelic polymorphism. J. The-
oret. Biol. 96, 689–705.
of partial male-sterility and the evolution of monoecy and dioecy.
Heredity 41, 137–153.
Porcher, E., and Lande, R. (2005). The evolution of self-fertilization
and inbreeding depression under pollen discounting and pollen
limitation. J. Evol. Biol. 18, 497–508.
Porcher, E., and Lande, R. (2005). Loss of gametophytic self-in-
compatibility with evolution of inbreeding depression. Evolution
Latta, R., and Ritland, K. (1994). Models for the evolution of selfing
under alternative models of inheritance. Heredity 71, 1–10.
Charlesworth, D., Morgan, M.T., and Charlesworth, B. (1990). In-
rates in a multi-locus system with no linkage. Evolution 44, 1469–
Charlesworth, D., and Charlesworth, B. (1990). Inbreeding de-
pression with heterozygote advantage and its effect on selection
for modifiers changing the outcrossing rate. Evolution 44, 870–
Cheptou, P.O., and Mathias, A. (2000). Can varying inbreeding de-
pression select for intermediary selfing rates? Am. Nat. 157, 361–
Routley, M.B., Bertin, R.I., and Husband, B.C. (2004). Correlated
evolution of dichogamy and self-incompatibility: A phylogenetic
perspective. Int. J. Plant Sci. 165, 983–993.
Uyenoyama, M.K. (1988). On the evolution of genetic incompatibil-
ity systems. II. Initial increase of strong gametophytic self-incom-
patibility under partial selfing and half-sib mating. Am. Nat. 131,
Charlesworth, D., and Charlesworth, B. (1979). The evolution and
breakdown of S-allele systems. Heredity 43, 41–55.
Charlesworth, D., Vekemans, X., Castric, V., and Gle ´min, S. (2005).
Plant self-incompatibility systems: a molecular evolutionary
perspective. New Phytologist 168, 61–69.
64. Vekemans, X., and Slatkin, M. (1994). Gene and allelic genealogies
at a gametophytic self-incompatibility locus. Genetics 137, 1157–
Schopfer, C.R., Nasrallah, M.E., and Nasrallah, J.B. (1999). The
male determinant of self-incompatibility in Brassica. Science 286,
Takayama, S., Shiba, H., Iwano, M., Shimosato, H., Che, F.-S., Kai,
N., Suzuki, G., Hinata, K., and Isogai, A. (2000). The pollen determi-
nant of self-incompatibility in Brassica campestris. Proc. Natl.
Acad. Sci. USA 97, 1920–1925.
Kusaba, M., Dwyer, K., Hendershot, J., Vrebalov, J., Nasrallah,
J.B., and Nasrallah, M.E. (2001). Self-incompatibility in the genus
Arabidopsis: characterization of the S locus in the outcrossing A.
lyrata and its autogamous relative, A. thaliana. Plant Cell 13, 627–
ses of S-RNase-based self-incompatibility. Plant Cell 16, S72–S83.
of self-incompatibility haplotypes: transition through self-compat-
ible intermediates. Genetics 157, 1805–1817.
Chookajorn, T., Kachroo, A., Ripoll, D.R., Clark, A.G., and Nasral-
lah, J.B. (2003). Specificity determinants and diversification of the
Brassica self-incompatibility pollen ligand. Proc. Natl. Acad. Sci.
USA 101, 911–917.
Barrett, S.C.H. (1992). Heterostylous genetic polymorphisms:
model systems for evolutionary analysis. In Evolution and Function
of Heterostyly, A. Omary, ed. (Heidelberg: Springer-Verlag), pp.
Barrett, S.C.H., and Harder, L.D. (1996). Ecology and evolution of
plant mating. Trends Ecol. Evol. 11, 73–79.
Charlesworth, D.,andCharlesworth,B. (1979).A modelfortheevo-
lution of distyly. Amer. Nat. 114, 467–498.
Lloyd, D.G., and Webb, C.J. (1992). The evolution of heterostyly. In
Evolution and function of heterostyly, S.C.H. Barrett, ed. (Heidel-
berg: Springer-Verlag), pp. 151–178.
Holsinger, K.E. (1986). Dispersal and plant mating systems: the
evolution of self-fertilization in subdivided populations. Evolution
Vallejo-Marin, M., and Uyenoyama, M.K. (2004). On the evolution-
sation due to pollen limitation. Evolution 58, 1924–1935.
Kalisz, S., Vogler, D.W., and Hanley, K.M. (2004). Context-depen-
dent autonomous self-fertilization yields reproductive assurance
and mixed mating. Nature 430, 884–887.
Husband, B.C., and Barrett, S.C.H. (1993). Multiple origins of self-
ferences from style morph and isozyme variation. J. Evol. Biol. 6,
Crosby, J.L. (1949). Selection of an unfavourable gene-complex.
Evolution 3, 212–230.
Piper, J., Charlesworth,B., andCharlesworth,D. (1984).A high rate
of self-fertilization and increased seed fertility of homostyle prim-
roses. Nature 310, 50–51.
Piper, J., Charlesworth, B., and Charlesworth, D. (1986). Breeding
system evolution in Primula vulgaris and the role of reproductive
assurance. Heredity 56, 207–217.
cio, M. (2003). Effects of male sterility on reproductive traits in
gynodioecious plants: a meta-analysis. Oecologia 135, 1–9.
Nilsson, E., and Agren, J. (2006). Population size, female fecundity,
Biol. 19, 825–833.
Delannay, X., Gouyon, P.-H., and Valdeyron, G. (1981). Mathemat-
ical study of the evolution of gynodioecy with cytoplasmic inheri-
tance under the effect of a nuclear restorer gene. Genetics 99,
Charlesworth, D. (1981). A further study of the problem of the
maintenance of females in gynodioecious species. Heredity 46,
Gouyon, P.H., Vichot, F., and Damme, J.v. (1991). Nuclear-cyto-
plasmic male sterility: single point equilibria versus limit cycles.
Am. Nat. 137, 498–514.
Frank, S.A. (1989). The evolutionary dynamics of cytoplasmic male
sterility. Am. Nat. 133, 345–576.
Jacobs, M.S., and Wade, M.J. (2003). A synthetic review of the the-
ory of gynodioecy. Am. Nat. 161, 837–851.
Couvet, D., Ronce, O., and Gliddon, C. (1998). The maintenance of
nucleocytoplasmic polymorphism in a metapopulation: the case of
gynodioecy. Am. Nat. 152, 59–70.
Manicacci, D., Couvet, D., Belhassen, E., Gouyon, P.-H., and Atlan,
A. (1996). Founder effects and sex-ratio in the gynodioecious Thy-
mus vulgaris L. Mol. Ecol. 5, 63–72.
91. Pannell, J. (1997). Widespread functional androdioecy in Mercuria-
lis annua L. (Euphorbiaceae). Biol. J. Linnean Soc. 61, 95–116.
Pannell, J. (1997). The maintenance of gynodioecy and androdio-
ecy in a metapopulation. Evolution 51, 10–20.
Pannell, J.R., and Barrett, S.C.H. (1998). Baker’s law revisited: Re-
productive assurance in a metapopulation. Evolution 52, 657–668.
Ashman, T.-L. (2003). Constraints on the evolution of males and
sexual dimorphism: Field estimates of genetic architecture of re-
productive traits in three populations of gynodioecious Fragaria
virginiana. Evolution 57, 2012–2025.
Allen, G.A., and Antos, J.A. (1993). Sex ratio variation in the dioe-
cious shrub Oemleria cerasiformis. Am. Nat. 141, 537–553.
Rocheleau, A.F., and Houle, G. (2001). Different cost of reproduc-
tion for the males and females of the rare dioecious shrub Corema
conradii (Empetraceae). Am. J. Bot. 88, 659–666.
Dorken, M.E., and Barrett, S.C.G. (2004). Phenotypic plasticity of
vegetative and reproductive traits in monoecious and dioecious
populations of Sagittaria latifolia (Alismataceae): a clonal aquatic
plant. J. Ecol. 92, 32–44.
Lloyd, D.G., and Bawa, K.S. (1984). Modification of the gender of
seed plants in varying conditions. Evol. Biol. 17, 255–338.
Takebayashi, N., and Morrell, P.P. (2001). Is self-fertilization an
evolutionary dead end? Revisiting an old hypothesis with genetic
theories and a macroevolutionary approach. Am. J. Bot. 88,
Igic, B., Bohs, L., and Kohn, J.R. (2006). Ancient polymorphism re-
veals unidirectional breeding system shifts. Proc. Natl. Acad. Sci.
USA 103, 1359–1363.
Shimizu, K.K., Cork, J.M., Caicedo, A.L., Mays, C.A., Moore, R.C.,
Olsen, K.M., Ruzsa, S., Coop, G., Bustamante, C.D., Awadalla, P.,
et al. (2004). Darwinian selection on a selfing locus. Science 306,
Charlesworth, B., and Charlesworth, D. (2000). The degeneration of
Y chromosomes. Phil. Trans. Roy. Soc. Lond. B. 355, 1563–1572.