Second-order moments of segregating sites under variable population size.

Abstract

The identification of genomic regions that have been exposed to positive selection is a major challenge in population genetics. Since selective sweeps are expected to occur during environmental changes or when populations are colonizing a new habitat, statistical tests constructed on the assumption of constant population size are biased by the co-occurrence of population size changes and selection. To delimit this problem and gain better insights into demographic factors, theoretical results regarding the second-order moments of segregating sites, such as the variance of segregating sites, have been derived. Driven by emerging genomewide surveys, which allow the estimation of demographic parameters, a generalized version of Tajima's D has been derived that takes into account a previously estimated demographic scenario to test single loci for traces of selection against the null hypothesis of neutral evolution under variable population size.

Full text

Page 1

Second order moments of segregating sites under variable
population size
DanielˇZivkovi´ c, Thomas Wiehe
Institut f¨ ur Genetik, Universit¨ at zu K¨ oln, 50674 K¨ oln, Germany
1

Genetics: Published Articles Ahead of Print, published on August 20, 2008 as 10.1534/genetics.108.091231

Page 2

Running Head: Coalescent with variable population size
Key Words: Coalescent trees, variable population size, second order moments, Tajima’s D,
Drosophila melanogaster
Corresponding Author:
Thomas Wiehe
Institute for Genetics, University of Cologne,
Z¨ ulpicher Straße 47, 50674 Cologne, Germany
Tel.: +49 221 470 1588
Fax: +49 221 470 1630
Email: twiehe@uni-koeln.de
2

Page 3

Abstract
The identification of genomic regions which have been exposed to positive selection is a
major challenge in population genetics. Since selective sweeps are expected to occur during
environmental changes or when populations are colonizing a new habitat, statistical tests
constructed on the assumption of constant population size are biased by the co-occurrence
of population size changes and selection. In order to delimit this problem and gain better
insights into demographic factors, theoretical results regarding the second order moments
of segregating sites, such as the variance of segregating sites, have been derived. Driven by
emerging genome-wide surveys, which allow the estimation of demographic parameters, a
generalized version of Tajima‘s D has been derived which takes into account a previously
estimated demographic scenario to test single loci for traces of selection against the null
hypothesis of neutral evolution under variable population size.
3

Page 4

INTRODUCTION
The analysis and interpretation of patterns of DNA variation is the ‘main obsession’
(Gillespie 2004) of molecular population geneticists. It is of particular interest to deter-
mine whether a locus is evolving neutrally or as a target of selection (Hey 1999). Adaptive
processes are generally thought to be associated with selective sweeps (Charlesworth
1992). Models predict that selective sweeps result in a local reduction of DNA polymor-
phism around the target site of selection. However, it is well known that a pattern of
reduced polymorphism may also result from demographic changes, particularly from recent
population size bottlenecks.Therefore, a reliable identification of the cause of reduced
variability is difficult; especially when DNA fragments of only a few loci are examined.
Recently, several new methods and statistical tests were developed to analyze polymorphism
data which presumably have been produced by a complex evolutionary history, during which
selective and demographic forces acted simultaneously. Nielsen et al. (2005) developed
a composite likelihood method, which is an extension of the test proposed by Kim and
Stephan (2002), where the null hypothesis, rather than being fixed a priori, is derived from
the pattern of background variation in the data itself. Li and Stephan (2006) devised a
maximum likelihood method, which allows inference of demographic changes and detection
of recent positive selection in populations of varying size. Alternatively, variability patterns
at multiple loci may be considered jointly. The reasoning here is that the traces of selective
sweeps may be distinguished from those of recent population bottlenecks, since only the
former have a local effect, while the latter should have a chromosome wide effect. In a test
tailored for multiple microsatellite loci, Wiehe et al. (2007) found that this is often the
case, but, for a certain (population genetically relevant) range of demographic parameters
the false positive rate to detect selective sweeps is still unacceptably high. Other recently
popularized methods to detect traces of selection are centered around the analysis of linkage
disequilibrium (Kim and Nielsen 2004, Stephan et al. 2006, McVean 2007) or the
4

Page 5

structure and frequency spectrum of haplotypes (Sabeti et al. 2002,2007). In contrast to
variability based methods, the latter require knowledge or inference of multilocus haplotypes
for data analysis. While the human HapMap project (The International HapMap
Consortium 2005) furnishes such data, they are rarely available in population genetical
studies of other species.
Initiated by the seminal paper of Kingman (1982), coalescent models have become the
textbook standard to study the properties of alleles that are generated by common ancestry
(Hein et al. 2005). Although it is well known that demographic changes such as population
size expansions and bottlenecks, admixture and population subdivision affect present
day polymorphism patterns, popular statistical tests for deviation from neutral evolution
(Tajima 1989b, Fu and Li 1993, Fay and Wu 2000) posit a model of constant population
size. At the same time, researchers emphasize that an adequate interpretation of observed
polymorphism patterns, for instance in Drosophila, cannot neglect the demographic history
of a population (Glinka et al. 2003, Haddrill et al. 2005). There is a fairly long history of
population genetical models considering the effects of variable population size upon genetic
variability. For example, Nei et al. (1975), Maruyama and Fuerst (1984), Watterson
(1984), Tajima (1989a) and Slatkin and Hudson (1991) studied the consequences of
population size reductions or expansions. The different scenarios have been consolidated
into a more general framework by Griffiths and Tavar´ e (1994).
Here, we study the standard coalescent approximation to the Wright-Fisher model for a
sample of n genes, without recombination and with population size varying in time. We
first revisit joint densities of waiting times, as defined by Griffiths and Tavar´ e (1994),
and their means (Griffiths and Tavar´ e 1998). Next, we present new results concerning
their second order moments. We extend the results of Fu (1995), regarding size and type
of mutations, to general coalescent trees (Griffiths and Tavar´ e 1998,2003) and in
particular to coalescent trees under variable population size. Based on these results, we
establish a generalized version of Tajima´s D (Tajima 1989b), which is applied to test
5