Selection in the Lab: A Network Approach

Abstract

We study the dynamics of the selection problem in economic experiments. We show that adding dynamics significantly complicates the effect of the selection problem on external validity and can explain some contradictory results in the literature. We model the dynamics of the selection problem using a network model of diffusion in which agents’ participation is driven by the two channels: the direct channel of recruitment and the indirect channel of agent interaction. Using rich recruitment data from a large public university, we find that the patterns of participation and biases are consistent with the model. We find evidence of both short- and long-run selection biases between student types. Our empirical findings suggest that network effects play an important role in shaping the dynamics of the selection problem. We discuss the implications of our results for experimental methodology, design of experiments, and recruitment procedures.

Full text

Selection in the Lab: A Network Approach∗
Aleksandr Alekseev†Mikhail Freer‡
August 12, 2019
Abstract
We study the dynamics of the selection problem in economic experiments. We show that
adding dynamics significantly complicates the effect of the selection problem on external valid-
ity and can explain some contradictory results in the literature. We model the dynamics of the
selection problem using a network model of diffusion in which agents’ participation is driven by
the two channels: the direct channel of recruitment and the indirect channel of agent interaction.
Using rich recruitment data from a large public university, we find that the patterns of partici-
pation and biases are consistent with the model. We find evidence of both short- and long-run
selection biases between student types. Our empirical findings suggest that network effects play
an important role in shaping the dynamics of the selection problem. We discuss the implications
of our results for experimental methodology, design of experiments, and recruitment procedures.
Keywords: selection problem, experiments, external validity, networks, diffusion, peer effects
JEL codes: C32, C90, D85
∗We thank Daniel Houser for providing access to the data and Arthur Dolgopolov for helping with the extraction
and preparation of the data. This paper has greatly benefited from suggestions by Glenn Harrison, Robert Slonim,
Nat Wilcox, and David Rojo-Arjona, as well as participants at the Economic Science Association World Meetings
(2017) and Southern Economic Association Meetings (2018).
†Economic Science Institute, Chapman University, One University Drive, Orange, CA, 92866, e-mail:
alekseev@chapman.edu, phone: +1 (714) 744-7083, ORCID: 0000-0001-6542-1920.
‡ECARES, Universit´e Libre de Bruxelles, Av. Franklin D. Roosevelt 50, CP 114/04, 1050 Bruxelles, Belgium.

1 Introduction
Experimental methods have become an indispensable tool in economic science (Smith,1989;Falk
and Heckman,2009). Lab and field experiments provide a steady source of behavioral insights for
new models and serve as a convenient test bed for existing models. With increasing reliance on
experimental methods, however, comes the need to confirm the robustness of their procedures. An
important issue that has gained considerable attention in recent years is external validity, or whether
the results obtained with experimental subjects generalize to a relevant reference population.1A
major threat to external validity is the selection problem, i.e., the fact that subjects represent a
non-random sample of a reference population.
A large and growing literature has emerged to address the selection problem. This literature
has produced contradictory results.2A major limitation of the existing literature, which could also
serve as a potential source of contradictory results, is a view that the selection problem is static
and individual in nature. In this paper we attempt to overcome this limitation by studying, both
theoretically and empirically, how the dynamic and network effects shape the selection problem.
We begin our analysis by showing that the presence of dynamics significantly complicates the
effect of the selection problem on external validity. Conditions under which selection problem does
not pose a threat to external validity in the static case are no longer valid in the dynamic case. In
particular, even if treatment effects are homogeneous across types (a sufficient condition for external
validity in the static case), the estimated treatment effects may still deviate from the population
treatment effect in the dynamic case. The presence of dynamics is likely to create a treatment
effect bias even a subject pool does not have a selection problem initially.
Having established the importance of dynamic effects in the selection problem, we proceed
by developing a dynamic network model of participation in a subject pool. Our model is based
on a classic Bass (1969) network model of diffusion. The distinctive feature of our model, as
1For example, Levitt and List (2007) and List (2009) argue that social preferences, which remain one of the most
actively studied topic in behavioral and experimental economics, are grossly over-estimated in laboratory experiments.
Henrich et al. (2010) argue that the behavior and characteristics of student subjects in Western countries are somewhat
atypical when compared with other societies across the world. G¨achter (2010), on the other hand, argues that student
subjects are a great sample for the research questions economists usually ask.
2Some papers (Cleave et al.,2013;Exadaktylos et al.,2013;Falk et al.,2013) report that the behavior of students
who sign up for economic experiments is not significantly different from the behavior of student or general population
in social and risky settings. Other papers (Slonim et al.,2013;Cappelen et al.,2015) do find significant differences
between student participants and reference populations.
1

opposed to the models typically used in the literature (e.g., Abeler and Nosenzo (2015), Slonim
et al. (2013)), is that it recognizes the dynamic and social (via networking) nature of the decisions
to participate in a subject pool. The central assumption of the model is that agents can sign
up for participation via two channels. The first channel is a direct channel represented by an
experimenter’s recruitment efforts, such as e-mail invitations or class visits. The second channel
is an indirect channel represented by the interactions between agents within their network.3This
channel highlights the idea that an agent may be convinced to participate by her peers. In each
period, the agents who are not in a subject pool may arrive into the pool via either direct or indirect
channels at given type-specific rates.4
In our first specification of the model, an agent’s type is a time-invariant characteristic. The
model generates three key features of the participation dynamics: rapid growth immediately after
recruitment, a slowdown in growth later on, and steady growth towards an upper limit in subsequent
periods. A natural implication of this model is that over time the participation of each type
asymptotically approaches a value that we refer to as a potential proportion. Differences in potential
proportions, which may result from differences in preferences, such as risk preferences (Harrison
et al.,2009), is what drives the long-run selection biases between types in this version of the model.
Additionally, if types have different initial participation levels or have different participation growth
rates, a subject pool will also exhibit short-run selection biases. Simulations show that a typical
dynamic pattern is a high initial bias that declines steadily in subsequent periods. Such a pattern
is consistent with the results in two recent studies (Slonim et al.,2013;Cleave et al.,2013) that
have almost identical designs but differ in the points in time at which biases were evaluated.
In our second specification of the model we allow types to be time-variant. We consider a case
when types are cohorts of agents. An important insight from this version of the model is that
long-run biases are possible even when potential proportions are the same across types that is not
the case with time-invariant types. The model predicts oversampling of older cohorts in a subject
pool relative to younger cohorts. The participation levels among types will be asymptotically lower
than their respective potential proportions, unlike in the model with time-invariant types, due to
3Recruitment through an indirect channel is related to a recruitment technique in sociology known as snowball
sampling (Biernacki and Waldorf,1981).
4Our decision to introduce two separate channels is motivated by two considerations. First, it is a more accurate
description of an actual recruiting process. Second, having two channels leads to different dynamics of the selection
problem, and hence to different policy implications, as compared to having only one channel.
2

the constant outflow of agents. The dynamic path and magnitude of the selection problem between
cohorts will depend on the network structure of a reference population.
We use recruitment data from the Interdisciplinary Center for Economic Science (ICES) at
George Mason University (GMU) and GMU’s registrar enrollment data to construct daily time-
series of student participation in the subject pool. Our observation period is from the Fall semester
of 2014 until the Spring semester of 2017. We have data on when each subject entered and left the
subject pool, as well as subjects’ characteristics. We compare the number of students of each type
who are in the subject pool to the number of students of that type in the student population to
identify selection biases.
While our theoretical model allows for any reference population, in the empirical analysis we
use the student population as our reference population. Our choice of the reference population is
motivated by convenience and the fact that the GMU registrar provides high-quality data on the
student population. Using alternative reference populations would have changed the scales of our
time-series but not the dynamic trends, which are the focus of the present study.
Similarly, even though our theoretical model allows for types to be unobservable, as well as ob-
servable, we choose to use observables in our empirical analysis. This choice is dictated by necessity:
the analysis of the dynamics of selection requires high-frequency data on student types. We are
aware of no recruitment system that currently features such high-frequency data on unobservable
characteristics. The analysis of the dynamics of observable types, however, does provide insights
into the dynamics of the unobservable types. If a decision to participate in a subject pool reflects
a subject’s preferences, then the trends in observable types should reflect these preferences. Put
differently, if the dynamics of unobservable types are flat and there is a mapping between observable
and unobservable characteristics, the dynamics of observable types must also be flat.
Our empirical analysis yields three main findings. First, we show that the dynamics of the
subject pool is consistent with the dynamics predicted by the model. In particular, we confirm the
existence of potential proportions for each type. These patterns are observed regardless of whether
we define gender (time-invariant) or cohort (time-variant) as a type. Second, we find evidence of
both short- and long-run biases between types. Males tend to be consistently oversampled relative
to females both in the short- and long-run, though the size of the bias is small. Cohorts data reveal
that younger cohorts are initially oversampled in the subject pool relative to older cohorts but over
3

time a reversal of this ranking occurs. Finally, we conduct the structural estimation of the model
and document the heterogeneity between types in terms of the strengths of the direct and indirect
channels. The counterfactual analysis shows that the indirect channel accounts for one-third of the
total participation gain and plays a substantial role is shaping the selection bias.
Three policy recommendations emerge from our analysis of the dynamics of the selection prob-
lem. First, future studies of the selection problem should recognize its dynamic nature. The mea-
surements of the selection problem, for example, could be done at various points in time throughout
a recruitment cycle. Ignoring the dynamic nature of the selection problem could lead to seemingly
contradictory results about the presence of the selection problem in a subject pool. Second, stud-
ies using a between-subject design would benefit from randomizing treatments on a subject-level
within a session. That would minimize the potential treatment effect bias caused by the dynamic
component the selection problem. If randomization to treatment within a session is infeasible,
keeping sessions temporally close (or even running them simultaneously at different places) would
be the second-best option. Third, it would make sense to leave a short (one to two weeks) burn-in
period after a main recruitment event. Selection biases tend to stabilize in the long-run. Therefore,
leaving a burn-in period would minimize the selection biases caused by short-run fluctuations.
Our paper contributes to the growing literature on the selection problem in economic exper-
iments by providing a theoretical framework that captures the dynamic and network nature of
participation in a subject pool, as well as empirical evidence on the dynamics of student participa-
tion and the selection problem. Other papers have proposed theoretical models of participation in a
subject pool. Abeler and Nosenzo (2015) assume that the aggregate utility of participation consists
of three parts: monetary utility, pro-social utility, and a fixed cost. Monetary and pro-social parts
have different weights, and these weights determine which motive is stronger. The model assumes
that recruitment changes potential participants’ priors about the expected monetary reward and
the need for social approval. Slonim et al. (2013) introduce a more general utility-of-participation
function that has four components: monetary reward, leisure time, intellectual curiosity, and so-
cial preferences. Their model predicts that students with lower income, more leisure time, higher
curiosity, and higher pro-social preferences are more likely to participate in a subject pool.
Our theoretical analysis highlights the mechanisms that drive short- and long-run selection bi-
ases and helps to explain some contradictory results in the existing literature. Falk et al. (2013)
4

look at the entire student population of the University of Zurich from 1998 to 2004 and compare
social preferences of students who signed up for participation in economic experiments with social
preferences of the general student population. As a measure of social preferences, they use informa-
tion from the school about students’ donation choices. The main finding is that pro-social behavior
does not differ between the two groups. Cleave et al. (2013) study the student population at the
University of Melbourne by conducting experiments in classes and also inviting students to partici-
pate in economic experiments. Five months later, they compare the choices of students in the class
experiments to the choices of those students who agreed to participate in experiments, registered
in the database and showed up for a laboratory experiment. Their analysis reveals that the choices
are very similar across the two groups, as are their demographic characteristics. Slonim et al.
(2013), using a design similar to that of Cleave et al. (2013) but a shorter waiting period of several
weeks, do find significant differences in some characteristics and preferences of participants and
non-participants in lab experiments. Taken at face value, the results from the two former papers
contradict each other, however, viewing these results from the perspective of our dynamic model
alleviates the contradiction. It is likely that the results in Slonim et al. (2013) reflect the short-run
biases in the data, while the results in Cleave et al. (2013) reflect the long-run biases, which are
typically smaller. None of the existing models of participation can account for this pattern.
The remainder of the paper is organized as follows. Section 2defines the selection problem
and illustrates how the selection problem affects external validity. Section 3introduces the model
with time-invariant types and presents theoretical results for the dynamics of participation and
selection bias. Section 4extends the model by allowing for time-variant types. Section 5describes
the data and presents the reduced-form results for participation and selection. Section 6contains
the procedure and results of the structural analysis of the data. Section 7concludes and discusses
directions for further research.
2 Selection Problem
Let N={1, . . . , N }be the set of types of agents in a population based upon a certain characteristic.
We call the characteristics that do not change over time time-invariant characteristics. Characteris-
5

tics that change over time are called time-variant characteristics. Characteristics can be observable
(such as demographic characteristics) or unobservable (such as preferences and personality traits).
Let ˜
xt≡(˜x1
t,...,˜xN
t)0be a vector of proportions of the number of agents of each type in
a subject pool relative to the total number of agents in a reference population at time t. A
reference population can be the population of people in a school that hosts a subject pool, the
population of a local area in which the school is located, the general population of a country, or
any other population deemed relevant by a researcher. For example, if we use risk aversion as a
characteristic and assign Type I to risk averse/neutral people and Type II to risk loving people,
then ˜
xt=10 = (0.1,0.2)0means that at the 10th period of observation a subject pool contained
10% of people in the reference population who are risk averse/neutral and 20% of people in the
reference population who are risk loving people. If we denote Stto be the total number of people
in a reference population at time t, then a subject pool can be defined by the number of agents
of each type in it: St˜
xt. For example, if there are S10 = 10,000 people in a reference population
at time 10, a subject pool would be composed of 1,000 risk averse/neutral subjects and 2,000 risk
loving subjects.
While there are more risk loving subjects than risk averse/neutral subjects in this subject pool,
this does not necessarily cause a selection problem, since the shares of each type in a reference
population may also be different. For instance, if there are twice as many risk loving people in
a reference population than risk averse/neutral people, then the composition of the subject pool
perfectly matches the composition of the reference population, and there is no selection problem.
Let m≡(m1, . . . , mN)0,ı0m= 1,(ıis a sum vector of order N), be a vector of shares of each type
in a reference population. In the previous example, if there are twice as many risk loving people as
risk averse/neutral people, the reference population is 33% risk averse/neutral and 66% risk loving,
and so m= (1/3,2/3)0. Let M≡diag(m) be a diagonal matrix with the population proportions
of each type on the diagonal. Then xt≡M−1˜
xt= (˜x1
t/m1,...,˜xN
t/mN)0is a vector of proportions
of the number of agents of each type in a subject pool relative to the shares of agents of these
types in a reference population. This quantity will be called a vector of relative proportions and its
dynamics is the main object of study in this paper.
6

What does it mean, then, to have a selection problem? A selection problem occurs when the
composition of a subject pool does not match the composition of a reference population. We can
formally define the selection problem as follows.
Definition 1 (Selection Problem).A subject pool is said to have a selection problem at time t
(asymptotically) if there is no constant ¯α∈[0,1], such that the vector of relative proportions can
be written as xt= ¯αı(limt→∞ xt= ¯αı).
Suppose that for some subject pool xt6= ¯αıfor any ¯αso that there is a selection problem at
time t. How bad is this problem? To answer this question, we can look, for instance, at pairwise
biases between types and define bij
t≡xi
t/xj
t, i, j ∈ N . This number determines the ratio of a
relative proportion of agents of type ito a relative proportion of type jat time t. If bij
t>1, there
are more agents of type iin a subject pool than agents of type jrelative to their population shares,
and the bigger is bij
t, the stronger is the selection bias between these two types. On the flip side, bji
t
in this example will be less than one. The strength of the bias is therefore determined by how far
a pairwise bias is from 1. If all the pairwise biases are equal to one, there is no selection problem.
On the other hand, if there is at least one pairwise bias that is not equal to one, there is a selection
problem, and a subject pool is biased.
Proposition 1. A subject pool does not have a selection problem at time t(asymptotically) iff for
any two types i, j ∈ N the pairwise bias bij
t= 1 limt→∞ bij
t= 1.
The existence of a selection problem makes it harder to extrapolate the results obtained from a
subject pool to a reference population. Two inference problems arise because of the selection prob-
lem. First, the estimated shares of each type will not be representative of the reference population.
Consider the example with two types of risk preferences. The share of Type I (risk averse/neutral
people) in the subject pool is given by5
˜mI≡˜xI
˜xI+ ˜xII =mI
mI+mIIbII,I.
5To simplify the exposition, below we focus on the case with only two types. However, the logic carries over to
a more general case.
7

The existence of a selection bias between the two types (bII,I6= 1) will drive the wedge between the
shares of each type in a subject pool and the actual shares in a reference population. The following
proposition follows.
Proposition 2. The shares of each type in a subject pool, ˜mi, will be identical to a reference
population shares, mi, iff a subject pool does not have a selection problem.
If a research question is to measure the prevalence of a certain type in the population (e.g.,
based on risk aversion or altruism), then the presence of a selection problem will bias the results.
A selection problem will also lead to a second inference problem: biased estimates of treatment
effects. To see this, consider again the case of two types. The average treatment effect in a reference
population, ∆Y, is given by
∆Y=mI∆YI+mII∆YII = ∆YII +mI∆YI−∆YII ,
where ∆YIand ∆YII are the treatment effects for the two types. In a subject pool, the average
treatment effect, ∆ ˜
Y, is given by
∆˜
Y= ˜mI∆YI+ ˜mII∆YII = ∆YII + ˜mI∆YI−∆YII .
The following proposition follows.
Proposition 3. A treatment effect in a subject pool will coincide with a treatment effect in a
reference population iff either (or both) condition holds: a) treatments effects are homogeneous
across all types; b) a subject pool does not have a selection problem.
This implies that if neither condition holds, i.e., treatment effects are heterogeneous and there is
a selection problem, a treatment effect in a subject pool will be different from a treatment effect in
a reference population. The existence of heterogeneous treatment effects across types is a common
scenario in many experimental settings (l’Haridon et al.,2018;Castillo and Freer,2018). Hence,
selection problem will be sufficient to cause biases in estimated treatment effects.
If a selection problem is dynamic, then the inference problems become more complicated. Con-
sider the case with two types when the relative proportions of types vary in time and treatments
8

are assigned in different moments in time. It is a common practice for experimental economists to
assign treatments on a session level in a between-subject design and then run different sessions in
sequence, which creates a temporal separation of treatments (Charness et al.,2012). Assume that
at time tboth types receive the baseline condition that produces response Yi
0in Type iand that at
time t+ 1 both types receive the treatment condition that produces response Yi
1in Type i. Then
an average treatment effect in a subject pool is given by
∆˜
Y= ∆YII + ˜mI
t∆YI−∆YII+ ∆ ˜mIYI
1−YII
1,
where ˜mI
tis share of Type I in a subject pool at time t, ∆ ˜mI≡˜mI
t+1 −˜mI
tis the change in a subject
pool share of Type I from time tto t+ 1. The difference between the treatment effects in a subject
pool and a reference population, a treatment effect bias, is then
∆Y−∆˜
Y=∆YI−∆YIImI−˜mI
t
|{z }
Static component
−∆ ˜mIYI
1−YII
1
| {z }
Dynamic component
,
where the first term represents the static component of the bias in an average treatment effect, and
the second component represents the dynamic component of the bias. The following proposition
then holds.
Proposition 4. In the presence of a dynamic selection problem, an estimated treatment effect in a
subject pool will coincide with a treatment effect in a reference population iff the static component
equals the dynamic component.
In contrast to the previous case with no time-varying selection problem, it is possible to have
a bias in an estimated average treatment effect even if treatment effects are homogeneous across
types. On the other hand, even if there is a selection problem at time tand treatment effects are
heterogeneous across types, it does not necessarily imply that there will be a bias in an average
treatment effect. It is possible that the change in the selection bias will be just right to compensate
for the differences between treatment responses.6Such a fortunate scenario, however, is unlikely.
If we further assume that types differ in their treatment effects and treatment responses, which is
6The precise condition is YI
0−YII
0
YI
1
−YII
1
=mI
−˜mI
t+1
mI−˜mI
t
, assuming that the denominators are not zero.
9

reasonable to expect, the presence of the dynamic component will create a treatment effect bias
even a subject pool does not have a selection problem initially. If a subject pool has a selection
problem initially, it is unclear a priori whether the dynamic component will make a treatment
effect bias smaller or larger. It is clear, however, that the dynamic component complicates the
interaction between a selection problem and related inference problems and thus deserves a closer
consideration.
3 Model For Time-Invariant Types
In order to guide our empirical analysis in the next section and make testable predictions, we build
a simple model of the participation in a subject pool and potential bias among types with time-
invariant characteristics. We use Bass (1969) model of diffusion as a starting point and adapt it
to the environment of a subject pool.7Suppose that at the start of the observation period, t= 0,
there are S˜
x0agents in a subject pool. We allow the relative proportions xtto have upper limits
given by a vector of potential proportions α= (α1, . . . , αN)0, αj∈[0,1], j ∈ N . These potential
proportions will allow for the existence of asymptotic biases.
New subjects can arrive into a subject pool via two channels. The first channel is a direct
channel of recruitment efforts, such as emails, posters, and class visits. The growth rate at which
agents who are not a pool will join it immediately after a recruitment event is given by a vector of
spontaneous rates p≡(p1, . . . , pN)0, pj>0, j ∈ N . These rates can be different across types. We
allow for a decay in the spontaneous rates, since recruitment events are typically infrequent and
short-lived. Let δ≡(δ1, . . . , δN)0be a vector of decay rates and let dt≡exp(−δt) be a vector of
decay factors.8Then at each time t, the vector of spontaneous rates is defined as pt≡diag(dt)p.
The second channel is an indirect channel of agent interactions: agents who are already in a
subject pool can induce their peers to join the pool. For example, a person who earned a decent
amount of money in an experiment or just enjoyed being a part of it may share this experience with
her friends. This channel is captured by a matrix of imitation rates Q≡ {qij }, i, j ∈ N . Each
7We are certainly aware of more advanced network models with types, e.g., Jackson and L´opez-Pintado (2013).
However, we chose in favor of a simple model of diffusion to make the exposition more intuitive. Despite its simplicity,
the model is flexible enough to incorporate the relevant features of the student pool.
8Research in psychology suggests that exponential decay is a reasonably close approximation of the so-called
“forgetting curve” (Murre and Dros,2015;Averell and Heathcote,2011;Ebbinghaus,1885/1974).
10

element qij >0 represents the growth rate in the participation of type ibecause of the interaction
with agents of type j. The diagonal elements of Qare own-imitation rates (imitation of the own
type), and the off-diagonal elements are cross-imitation rates (imitation of other types). Matrix
Qcan be thought of as a network structure of a reference population. The own-imitations rates
are allowed to differ between types. It is natural to assume that the own-imitation rates are higher
than the cross-imitation rates, but this is not required. The strength of the indirect channel will
depend on the relative proportions of agents of each type in a pool: there has to be someone to
imitate, and the more agents of a given type are in the pool, the more likely they will influence
the decision of their peers. We assume that imitation rates are constant, but this can be relaxed
by assuming that they depend on the activity of an experimental laboratory. In this case, running
more experiments would result in a higher likelihood that participants will share their experience
with their friends.
Our decision to introduce two separate channels, as opposed to the direct channel only, is
motivated by two considerations. First, having two channels is a realistic description of an actual
recruiting process. For example, in some schools the majority of the agent inflow into a subject pool
occurs via word-of-mouth (indirect channel). Ignoring the indirect channel thus would considerably
limit the model’s descriptive ability. The second, and more pragmatic, consideration is that the
model with two distinct channels produces very different dynamics of the selection problem as
compared to the model with the direct channel alone. Since different dynamic patterns of the
selection problem will have different policy implications, entertaining the possibility of the two
distinct channels and comparing their relative strength is relevant for policy recommendations.
In the practice of laboratory experiments, agents arrive into a subject pool from a student
population. However, a student population does not have to be a reference group. If a reference
group is chosen to be the general population, the rates in the model will simply reflect the joint
rate of transitioning from the general population into a student population and transitioning from
a student population into a subject pool. The choice of a reference group does not affect the intra-
semester or intra-year dynamics of a subject pool (measured at a daily frequency in our application),
since the transition into a student population occurs at a much lower frequency (typically, yearly).
The choice of a reference group will, however, affect the levels of time series.
11

Let the spontaneous and imitation rates be defined over a small time period ∆t>0. The dy-
namics of the subject pool participation can then be described by the following system of equations:
˜xi
t+∆t= ˜xi
t+pi
t∆t(αimi−˜xi
t) + qi1˜x1
t
m1+. . . +qiN ˜xN
t
mN!∆t(αimi−˜xi
t), i ∈ N .(1)
This equation attributes the growth of a subject pool participation to the two channels described
above. The second term on the right-hand side represents the inflow of subjects due to recruitment
efforts. The spontaneous rate is applied to the proportion of agents who are not currently in the
subject pool. Note that we multiply the share mby the potential proportion αto allow for non-full
participation, as typically only a fraction of a reference population participates in a subject pool.9
The third term reflects the inflow due to the imitation of the behavior of own types and other
types. The own- and cross-imitation rates are scaled by the relative proportions of agents: the
more agents participate in the subject pool, the stronger is the indirect channel.
Dividing both parts of equation (1) by the share mi, rearranging the terms, and taking the limit
∆t→0, we obtain the following system of differential equations:
˙xi= (αi−xi)(pi
t+q0
i·xt), i ∈ N ,(2)
where q0
i·is an i-th row of Q, or in matrix form,
˙
x= diag(α−xt)(pt+Qxt).(3)
Since the second terms on the right-hand side are positive, the steady state of the system is
simply the vector of potential proportions α. Starting from any initial value x06α, the relative
proportions will converge to the potential proportions: limt→∞ xt=α. The data on the relative
proportions at the end of a semester can then be used to infer the potential proportions. If the
potential proportions are different across types, long-run biases will occur. The differences in the
potential proportions can be caused by differences in preferences. For example, since women are
9In the case when the reference population is different from a student population, the potential proportion reflects
the joint long-term rate of transitioning from the reference population into a student population and from a student
population into a subject pool.
12

typically found to be more risk-averse than men (Falk et al.,2018), women may have lower potential
proportions than men. We summarize this result in the following proposition.
Proposition 5. A subject pool has a selection problem asymptotically iff there is no constant ¯α,
such that the vector of potential proportions can be written as α= ¯αı.
The short-run dynamics of the relative proportions and biases are determined by the starting
values x0, the potential proportions and the rate parameters of the model. The time path of relative
proportions can be found analytically in some special cases. The first case arises when there are
only own-imitation and no cross-imitation between types, and there are no subjects in the pool at
the beginning of the observation period. The formula below also assumes that the direct channel is
always present, i.e., there is no decay in the spontaneous rates. This case would apply in a situation
when, for example, recruitment posters are posted on a school’s announcement board and remain
there until the end of a semester.
Example 1. Assume that there is no cross-imitation between types, i.e., all the off-diagonal ele-
ments of the imitation matrix Qare zero, x0=0, and there is no decay, δ=0. Then the relative
proportions of each type evolve according to
xi
t=αi1−e−(pi+αiqii)t
1 + αiqii
pie−(pi+αiqii)t, i ∈ N .
The second case arises when there is no direct channel, and the indirect channel works only
through own-imitation. For example, some time might have passed after recruitment and sponta-
neous rates decayed to almost zero. In order for the subject pool to grow, some initial amount of
subjects has to be there. A subject pool will exhibit logistic growth in this case.
Example 2. Assume that there is no cross-imitation between types, (all the off-diagonal elements of
the imitation matrix Qare zero), that own-imitation rate is non-zero, that there is no direct channel
(pt=0), and that the subject pool starts with Sx0subjects in it. Then the relative proportions of
each type evolve according to
xi
t=αi
1 + αi−xi
0
x0e−αiqiit, i ∈ N .
13

The model, in general, generates a time path of relative proportions that is characterized by a
quick growth immediately after a recruitment event followed by a slow growth towards potential
proportions, as illustrated on Figure 1(panel A). Three features of the model produce this pattern.
First, a quick growth immediately after a recruitment event is caused by spontaneous rates. In the
model, spontaneous rates enter the growth equation directly, while imitation rates are scaled by the
relative proportions of imitated types. Therefore, imitation rates cannot account for initial rapid
growth. Figure 1(panel B) illustrates this point. It simulates the dynamics of the relative propor-
tions of two types when the direct channel is shut down (spontaneous rates are zero). The only
source of growth in the simulation is the indirect channel, which works because of non-zero starting
values. The relative proportions show steady but slow growth, reaching potential proportions only
by the end of the simulation period.
A sharp decrease in the growth rate of relative proportions is produced by the presence of
decay rates that dampen the effect of recruitment and reduce spontaneous rates over time. In the
absence of decay, the relative proportions would very quickly converge to the potential proportions.
Figure 1(panel C) shows simulation results for the case of zero decay rates and when only the direct
channel is present. Relative proportions show a steep growth immediately after the beginning of
the simulation period and quickly reach potential proportions.
Slow growth of relative proportions in the subsequent periods is produced by imitation rates.
In the absence of imitations rates, the growth rate would plummet to almost zero, and relative
proportions would never reach potential proportions. Figure 1(panel D) shows the simulation
results featuring the direct channel with decaying rates and completely no indirect channel. After
initial rapid growth, relative proportions flatten out and converge to steady states that are well
below potential proportions.
The model can produce several distinct dynamic patterns of biases. On Figure 2we focus on
the case with two types, in which there is a high initial bias and no asymptotic bias. Panel A
shows the simulation that features both direct and indirect channels, as well as decay, and all
the rates are identical across types. The only difference between types is in the starting values
of potential proportions. The initially high bias produced by different starting values quickly
converges, following a recruitment event, to the bias implied by the ratio of potential proportions.
14

A. All Features Present
B. Zero Spontaneous Rates
C. Zero Decay
D. Zero Imitation Rates
0 50 100 150 0 50 100 150 0 50 100 150 0 50 100 150
0.02
0.03
0.04
0.05
Time
Figure 1: Simulated Relative Proportions Under Different Parameter Values
Differences in spontaneous rates between types can alter the picture in two distinct ways.10
In particular, if the initial spontaneous rate of a type that starts low is higher than the initial
spontaneous rate of a type that starts high, a dip in the bias is observed, as Panel B illustrates.
A different pattern occurs if the type that starts high also has a higher initial spontaneous rate,
as illustrated on Panel C. After an initially high level, the bias increases even further following a
recruitment event, after which the bias drops and converges to the level implied by the ratio of
potential proportions.
A. Drop
B. Dip
C. Hill
0 50 100 150 0 50 100 150 0 50 100 150
1.0
1.2
1.4
1.6
Time
Figure 2: Simulated Biases Under Different Parameter Values
4 Model For Time-Variant Types
In the previous section we assumed that the population is static in that it consists of only time-
invariant types. The natural result of this assumption is that all the agents who can potentially
10 We focus on the effect of spontaneous rates on the dynamics of the bias since these rates have the most impact.
Adding differences in imitation rates can produce even more complex patterns.
15

be in a subject pool eventually end up there. In order to model low overall participation and
asymptotic biases between types, we had to set potential proportions exogenously and make them
different across types. In this section we are relaxing the assumption about a static population by
introducing cohorts of agents as time-variant types and look at how this changes our results. One
important consequence of this change is that non-full participation and biases between types will
arise endogenously, even when potential proportions are the same across types.
We assume that a refernce population consist of cohorts indexed by elements of a set N=
{0, . . . , N −1}, N >2. For simplicity, we assume that the shares of each cohort in a population are
identical: m≡(m0, . . . , mN−1)0= (1/N, . . . , 1/N)0. Agents who belong to cohort iin one period
move to cohort i+ 1 in the next period. Agents of cohort N−1 in the next period automatically
drop out of a subject pool (if they were there). At the same time, a new cohort, n= 0, arrives
every period. Each cohort represents a different type of agents. For example, in the case of college
students, as students advance through their programs, their type (e.g., risk or time preferences)
changes due to new experiences and acquired knowledge.11 To make the point about endogenous
biases and non-full participation stronger, we assume that a fixed proportion α∈[0,1] of each
cohort can potentially be recruited for a subject pool.12
As before, we assume that there are two channels through which agents may be induced to
participate in a subject pool. The direct channel is given by a vector of spontaneous rates p≡
(p0, . . . , pN−1)0, pi∈[0,1], i ∈ N . Importantly, the spontaneous rates are defined over the same
period as cohorts, and thus we do not consider decay in these rates within a period. In other words,
the rates in this version of the model are defined over academic years (or semesters) rather than
days.
The indirect channel is given by a matrix of imitation rates Q≡ {qij}, i, j ∈ N . It is natural
to assume that the probability to imitate own type is at least as big as that of other types, qii >
qij,∀i, j ∈ N . For instance, imitation rates can decline with the distance between cohorts, qij =
qii/(1 + d(i, j )), where d:N ×N 7→ R+is some distance function, such as |i−j|. This specification
11 For example, Dohmen et al. (2017) show, using representative panel data from Germany and Netherlands, that
people become more risk averse as they age. In particular, the willingness to take risks declines almost linearly with
age.
12 As we will show shortly, the equality of the potential proportions does not lead to the asymptotic equality of
relative proportions in the case of time-variant types. Differences in the potential proportions, therefore, will only
make this result stronger.
16

captures the idea that, for example, sophomores are more likely to interact with freshmen and
juniors than with seniors, and are even more likely to interact among themselves.
Let ˜
xt≡(˜x0
t,...,˜xN−1
t)0be a vector of proportions of the total number of agents in each cohort,
who are in a subject pool at time t, to the total number of agents, S. Given some initial number
of people in a subject pool, S˜
x0, the dynamics of these proportions are described by the following
system of equations:13
˜xi+1
t+1 = ˜xi
t+piαmi−˜xi
t+X
j∈N
qij ˜xj
t
mjαmi−˜xi
t, i ∈ N /{N−1}.(4)
These equations state that the number of agents of cohort i+1 who are in the subject pool at time
t+ 1 consists of agents who belonged to a previous cohort one period ago plus those who arrived
at the subject pool via direct and indirect channels. Equations (4) can be viewed as a natural
extension of the system (1), with the difference being that types change over time. We assume that
the newly arrived cohort does not provide any agents for a subject pool, since recruitment efforts
and agents’ interactions happen after cohorts are formed, so that ˜x0
t+1 = ˜x0
t= 0,∀t>0.
The proportion of all agents who are in a subject pool at time trelative to the total number of
agents is
˜xt=X
i∈N
˜xi
t.
Using this equation and equation (4), we find that the dynamics of the proportion of all agents
relative to a total number of agents in a population are given by
˜xt+1 = ˜xt−˜xN−1
t+
N−2
X
i=0
piα
N−˜xi
t+
N−2
X
i=0
N−1
X
j=0
qij ˜xj
t
mjα
N−˜xi
t.(5)
Note that we have to subtract the agents who were in cohort N−1 at time tsince they leave the
subject pool in the next period.
Denote xt≡(˜x0
t/(αm0),...,˜xN−1
t/(αmN−1)) to be a vector of the proportions of agents of
each cohort who are in a subject pool at time trelative to their potential shares in a reference
population. We will call xta vector of relative proportions. Note that it is defined differently than
in the previous section. We scale ˜
xtnot only by the shares m, but also by the potential proportions
13 In this version of the model we are focusing on the discrete time, so ∆t= 1.
17

α. In the context of time-invariant types, this definition would imply the absence of selection bias
in the long-run. In the context of time-variant types, however, this will not be the case. Similarly,
define xt≡˜xt/α to be the proportion of all agents in a subject pool relative to their potential
share. We will call this quantity a total relative proportion.
We now show that the selection problem occurs in the model with time-variant types even if the
types are a priori identical. We consider a symmetric case with no heterogeneity in spontaneous
or imitation rates across cohorts, i.e., pi=p, qij =q, ∀i, j ∈ N , and (without loss of generality) the
potential share α= 1. We bound the rate parameters by the condition p+qxt<1,∀tto ensure
that cohorts do not immediately reach their potential proportions. Under these assumptions, the
dynamics of the relative proportions can be rewritten as follows.
xi+1
t+1 =xi
t+p(1 −xi
t) + qxt(1 −xi
t), i ∈ N /{N−1}.(6)
Similarly, the dynamics of the total relative proportion are given by
xt+1 =xt−xN−1
t
N+p
N−1
N− xt−xN−1
t
N!
+qxt
N−1
N− xt−xN−1
t
N!
.
First, consider the asymptotic behavior of the subject pool. We are interested in the steady
state of the system when xi
t+1 =xi
t=xiand xt+1 =xt=x. Plugging these values into (6) yields
xi+1 =xi1−(p+qx)+p+qx.
This is a first-order linear difference equation, whose solution is xi= 1 −1−(p+qx)i. Using
this solution, we can find the stead-state equation for the total relative proportion:
x−1 + 1−1−(p+qx)N
N(p+qx)= 0.(7)
While a closed-form solution for xcannot be obtained, note that it will depend on the rate param-
eters of the model, as well as on the number of cohorts. Therefore, not everyone ends up in the
subject pool among those who can potentially be there, even if we wait long enough. Moreover,
18

this result has implications for the nature of the selection problem that would arise in a subject
pool.
Proposition 6. A subject pool has a selection problem asymptotically in the presence of cohorts.
The agents from older cohorts are oversampled in a subject pool, while agents from younger cohorts
are undersampled. The pairwise bias between two subsequent cohorts decreases in the cohort number.
The further cohorts are apart from each other the larger is the bias between them.
The intuition for this result is straightforward: agents in older cohorts have more chances
to be recruited over time. In the example with college students, if recruiting occurs every year,
juniors experience two recruitment periods (at the beginning of a school year), while sophomores
experienced only one. The problem will also exist for any time t. Thus we have shown that even
in the simplest symmetric case and potential proportion equal to 1, non-full participation and
selection problem arises endogenously.
We now consider possible asymmetric cases and compare them with the baseline results. We
consider four cases:
1. Baseline. Spontaneous, own- and cross-imitation rates are the same (pi=p, qij =q, ∀i, j ∈
N). This case is one from Proposition 6.
2. Constant. Spontaneous and own-imitation rates are constant for every cohort (pi=p, qii =
q, ∀i∈ N ), but the cross-imitation rates decline with the distance between the cohorts ac-
cording to
qij =qii
1 + |i−j|(8)
This represents the case when the cohorts are identical in terms of their interaction with their
own members and their sensitivity to recruitment, but their interaction with other cohorts
declines with the distance between the cohorts.
3. Increasing. Spontaneous and own-imitation rates increase exponentially with the cohort num-
ber (pi=p1/(i+1), qii =q1/(i+1),∀i∈ N ), and the cross-imitation rates decline with the dis-
tance between the cohorts according to (8). This pattern of rates would occur when not only
the interaction with other cohorts is different from the interaction within a cohort, but also
the interaction within a cohort becomes more pronounced, and the propensity to be recruited
19

increases. In the example with college students, students might get to know each other bet-
ter and build stronger bonds among themselves over time. Also, students can become less
risk-averse and more willing to engage in new activities, as they overcome the initial stress of
first college years and feel more comfortable in school.
4. Decreasing. Spontaneous and own-imitation rates decrease exponentially with the cohort
number (pi=p(i+1), qii =q(i+1) ,∀i∈ N ), and the cross-imitation rates decline with the
distance between the cohorts according to (8). In the example with college students, this case
reflects the possibility that as students progress through their programs, they become more
focused on their major, which leads to less interaction with their peers. Students also become
less willing to engage in new activities such as participation in experiments, because a larger
chunk of their time is now dedicated to completing thesis papers and finding jobs.
A. Baseline
B. Constant
C. Increasing
D. Decreasing
0 2 4 6 8 10 0 2 4 6 8 10 0 2 4 6 8 10 0 2 4 6 8 10
0.00
0.25
0.50
0.75
1.00
Time
Cohort First Second Third
Figure 3: Simulated Relative Proportions for Cohorts
Figure 3shows the simulated time-series of relative proportions xi
tfor each cohort, as well as the
total relative proportion xt(thick dashed line), for the four cases described above. The figures do
not show the results for cohort i= 0 since it never participates in the subject pool by assumption.
Comparing the Constant (panel B) and the Baseline (panel A) cases, we see that the reduced
interaction between the cohorts leads to lower long-run participation in the subject pool. We can
20

also see that the long-run participation increases with the cohort number since older cohorts have
more chances to end up in a subject pool, in line with the theoretical prediction.
Panel C shows that the increasing interaction within cohorts can more than offset the reduc-
tion caused by lower interaction between cohorts. The oldest cohort almost reaches its potential
maximum of participation in the long-run. In the Decreasing case (panel D), the participation is
even lower than in the Constant case, and thus decreasing rates is another important factor that
could drive low overall participation in a subject pool. Interestingly, the participation is no longer
greater for older cohorts at each time t: initially, younger cohorts are more represented in a subject
pool than older cohorts, but over time the older cohorts catch up and start to dominate the subject
pool.
A. Baseline
B. Constant
C. Increasing
D. Decreasing
0 2 4 6 8 10 0 2 4 6 8 10 0 2 4 6 8 10 0 2 4 6 8 10
0
1
2
3
Time
Bias 3rd and 2nd 3rd and 1st 2nd and 1st
Figure 4: Simulated Biases Between Cohorts
This point is worth developing further by looking at the graphs of pairwise biases over time on
Figure 4. In all the cases, except for the Decreasing one, we observe the growth of the bias over
time. Older cohorts are oversampled in a subject pool relative to younger cohorts in the long-run.
The bias increases in the distance between cohorts, e.g., the bias between cohorts 3 and 1 is higher
than the bias between cohorts 3 and 2. As we depart from the Baseline, the selection problem
becomes worse. It is particularly strong in the Increasing case. The Decreasing case is a notable
exception since the pattern of the spontaneous and imitation rates is offsetting the oversampling of
21

older cohorts. Even though the asymptotic bias is still present in the Decreasing case, its magnitude
is much lower than in other cases.
5 Empirical Results
5.1 Data Description
The data we use come from an online recruitment system (the recruiter, henceforth) used at the
ICES at GMU. Our observation period is from the beginning of the Fall semester of 2014 until
the end of the Spring semester of 2017.14 Two types of recruitment events are conducted at the
ICES: email invitations sent to the entire student population at GMU and selected class visits
(typically large undergraduate classes). Both types of events encourage students to sign up for
participation in economic experiments. In order to sign up, a student has to create an account and
fill out personal information, such as gender, ethnicity, age, starting year at college, and major.
Moreover, the recruiter records when an account was created, as well as the first and the last dates
of participation in a session for each subject.
We use an account creation date as the date when a subject enters the subject pool. Upon en-
tering the pool, a subject’s account becomes active. Only subjects with active accounts are allowed
to participate in experiments. All accounts are deactivated at the beginning of each semester, and
a corresponding email notification is sent to the entire subject pool. The email notifies subjects
that they will not receive any invitations for experiments until they reactivate their accounts. In
order to reactivate an account, a subject only needs to login to the recruiter. Subjects who do not
reactivate their accounts become non-active and are assumed to have dropped out of the subject
pool. We use the last day of a semester in which a non-active subject last participated in a session
as the date when they exited the subject pool. By the end of Spring 2017 the ICES subject pool
consisted of around 1,000 active accounts with complete demographic information.
We concentrate our analysis on two subject characteristics: gender (time-invariant type) and
year in college, or cohort, (time-variant type). Gender is self-reported by a subject upon registering
14 The ICES recruiter has been launched in Fall 2014. Unlike the previously used ORSEE system, the ICES
recruiter allows us to collect the data necessary for our analysis. We exclude Fall 15/Spring 16 academic year from
the analysis, however, due to a temporary shift in the recruitment procedures used. We also include summer months
in the Spring semester, since there are some changes in the subject pool during summer, but these changes are not
enough to have a separate Summer semester.
22

in the recruiter. Year in school is imputed from the self-reported starting year at college (the year
in which a subject started their program at GMU). In our analysis we use only undergraduate
student population since most economic experiments are conducted with undergraduate students.
We use year in college instead of a “class year” (freshman, sophomore, junior, senior), because the
information about the class year is not regularly updated by the subjects.
While our model allows the types to be observable or unobservable, we opt for using observables.
The primary reason for our choice is data availability. The analysis of the dynamics of selection
requires high-frequency data on students’ types, which are simply not available for the unobservable
characteristics.15 And even if data on unobservables were available, it would have likely been much
noisier than the data on observables (Gillen et al.,2019), which in turn would have made uncovering
the dynamic trends harder. We do believe, however, that the dynamics of the observable types are
still informative of the dynamics of the unobservable types. If a decision to participate in a subject
pool reflects a subjects’ preferences, as the literature suggests (Abeler and Nosenzo,2015;Slonim
et al.,2013;Krawczyk,2011), then the trends in the observable types, which we uncover in our
data, should reflect some of these unobservable preferences.
We use the data from the Office of Institutional Research and Effectiveness (OIRE) at GMU to
construct the characteristics of the reference population, which in our case is the student popula-
tion. Each semester OIRE publishes an overview of the entire student population, which includes
gender and ethnic composition, the total number of students enrolled, the number of undergraduate
students enrolled,16 as well as how many new students were admitted and what are the retention
rates for every class starting from 2007. Retention rate is a share of students who are still in college
after a given number of years. Students are considered to be in college only if they are still pursuing
a bachelor degree. For instance, for the cohort who started their program in 2007, 83.5% of the
students remain in college after one year, while only 5% of them remain in college after six years.
15 It is conceivable, and even advisable, that a recruitment system supplements basic demographic questions
with preference elicitation instruments. It will be impractical, of course, to deploy fully incentivized risk elicitation
tasks at the recruitment stage. However, survey questions, such as the ones proposed by Falk et al. (2018), can be
easily incorporated into a recruitment system and will not add too much burden on potential student participants
during registration. While arguably being crude, these instruments will shed at least some light on the unobservable
characteristics of entire subject pools.
16 The OIRE demographic composition data combines both undergraduate and graduate students. We use the
gender ratio for the total student population to impute the undergraduate gender composition.
23

Using these data in combination with the initial number of students in every cohort allows us to
impute the number of students who spend between 1 and 6+ years in college.17
While our model allows for any reference population, in our empirical analysis we opt for using
the student population at GMU as a reference. This choice is motivated by convenience and the
fact that the university’s registrar provides high-quality data on the student population. There is
little loss of generality from using the student population at GMU as a reference, as opposed to the
general population of the city of Fairfax, the state of Virginia, or the United States. Using these
alternative reference populations would change the scales of our time-series but not the dynamic
trends, the focus of this study. At the same time, the distributions of types in these alternative
reference populations are arguably measured with more noise than in the GMU registrar’s data.
Admitted Freshmen Total Undergraduate % Female
2014-2015 3,113 21,672 54%
Fall 2015 3,226 22,304 53%
Fall 2016 3,254 23,174 53%
Table 1: Description of the Population by Semester
Table 1provides a brief description of the student population at GMU by semester. Overall we
observe around 22,000 students every academic year of whom there are slightly more females than
males. About 3,000 new undergraduate students are admitted every Fall.
5.2 Results
5.2.1 Gender Types
We begin the analysis by looking at gender as a time-invariant type and present the dynamics and
summary statistics of the relative proportions of male and female participants in the subject pool
in Figure 5and in Table 2(see table notes for variables definitions). The shaded vertical bars in
Figure 5indicate the start and end of recruitment. The long-run participation in the subject pool
is very limited among both types, rarely exceeding 5%. Apart from the Fall 2014 semester, there
is a constant gap between the relative proportions of males and females, with males participating
at higher rates than females. The average potential proportion of males is slightly higher than the
17 We lump together the students who spend 6 and more years in college to increase the sample size in this small
group of students.
24

average potential proportion of females indicating the existence of a small but persistent long-run
bias.
Table 2: Summary Statistics
Spontaneous sign-up Imitation sign-up Drop-out Potential
proportion
Absolute Relative Absolute Relative Absolute Relative
Gender types
Female 0.014 1.018 0.006 0.190 0.009 0.213 0.043
Male 0.015 0.839 0.006 0.201 0.009 0.191 0.046
Cohort types
First-years 0.002 0.851 0.009 0.897 0.011 0.328 0.022
Second-years 0.037 1.800 0.010 0.150 0.034 0.359 0.075
Third-years 0.020 0.800 0.006 0.108 0.026 0.355 0.066
Fourth-years 0.013 0.419 0.004 0.083 0.024 0.356 0.067
Fifth-years 0.018 0.517 0.004 0.053 0.028 0.382 0.077
Sixth-years 0.009 0.378 0.001 0.023 0.014 0.392 0.065
Notes: Spontaneous sign-up is defined as an average (across all semesters) 1-week increase in a relative proportion of a type
since the start of a recruitment event. Imitation sign-up is defined as an average (across all semesters) change in a relative
proportion of a type between the end of semester and a 7-day point after the start of recruitment. Drop-out is defined as
an average (across all semesters) decrease in a relative proportion of a type at the end of semester. Absolute values are
defined as a difference between the ending and starting values of a given measure. Relative values are defined as a difference
between the ending and starting values of a given measure divided by the starting value. Potential proportion is defined as
an average (across all semesters) relative proportion at the end of semester.
There are distinctive dynamic patterns of the relative proportions in Figure 5that are consistent
with the patterns predicted by the model. Immediately after recruitment, relative proportions
of both types exhibit a rapid growth, which, however, decays within about a week. The relative
proportions continue to grow in the months after recruitment and slowly converge to an upper limit.
Table 2shows that the spontaneous sign-up rates for males and females are almost identical.18 If
we look at the relative sign-up, females increase their participation in the subject pool by more
than 100% immediately after recruitment, while males increase their participation only by 84%.
Females and males are equally active in the imitation sign-up and increase their participation in
the subsequent months by around 19% and 20%, respectively. Table 2also shows that females are
relatively more likely to drop out of the subject pool.
Figure 6shows the dynamics of the pairwise bias between males and females, bmale,female
t. As
noted earlier, there is virtually no bias throughout the Fall 2014 semester. After that, a small but
persistent bias emerges. The immediate response of the bias to recruitment tends to fall within
one of the three categories predicted by the model. The response is either monotonically declining
18 The spontaneous and imitation sign-up measures used in Table 2do not directly map into the structural
parameters of the model. These parameters are estimated in Section 6.
25

Fall16
Spring17
Fall14
Spring15
Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug
Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep
0.02
0.04
0.06
0.02
0.04
0.06
Type Female Male
Figure 5: Relative Proportions of Males and Females
(as in Fall 2016), hill-shaped (as in Spring 2017), or dip-shaped (as in Fall 2014). The mid-term
dynamics of the bias are rather complex and exhibit both high- and low-frequency oscillations.
Over time the bias tends to converge to a fixed value. By the end of the observation period the
bias is just above 1.1.
Fall16
Spring17
Fall14
Spring15
Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug
Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep
1.0
1.1
1.2
1.0
1.1
1.2
Figure 6: Bias Between Males and Females
26

5.2.2 Cohort Types
On Figure 7, we present the results for the time-variant types. The potential proportions among
the cohorts range between 2.2% for the first-years and around 7.7% for students with 6+ years in
school. The participation gaps among different cohorts are large and persistent, though the ranking
of the cohorts by their participation varies across semesters. During the first two semesters younger
cohorts tend to participate more in the subject pool than older cohorts. By the Fall 2016 semester,
the ranking of the cohorts is completely reversed: each older cohort dominates the cohort a year
earlier. This pattern continues, except for the abnormally high participation among second-years,
throughout the Spring 2017 semester. Such a pattern is in line with the predictions Proposition 6
of the Decreasing case of the time-variant version of the model.
Fall16
Spring17
Fall14
Spring15
Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug
Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep
0.00
0.03
0.06
0.09
0.00
0.03
0.06
0.09
Type First-years
Second-years
Third-years
Fourth-years
Fifth-years
Sixth-years
Figure 7: Relative Proportions of Cohort Types
The dynamic response of the relative proportions to recruitment in the case of cohort types is
very similar to the case of gender types. There are large differences between cohorts, however, in
terms of their immediate and long-run response to recruitment. The second-years beat all other
cohorts in terms of the spontaneous sign-up, both absolute and relative, and increase their par-
27

ticipation by 180%, on average, immediately after recruitment. The closest competitors are the
first-years and the third-years with relative spontaneous sign-up rates of 85.1% and 80%, respec-
tively. The sixth-years have the lowest relative sign-up of 38%, while the first-years have the lowest
absolute sign-up of 0.002. On the other hand, the first-years have the strongest relative imitation
sign-up rate of 89.7%, while the second highest rate, belonging to the second-year students, is only
15.2%. The relative drop-out rates are very similar across the cohorts and range between 32.8%
for the first-years and 39.2% for the sixth-years.
Figure 8shows the dynamics of the pairwise biases between subsequent cohorts.19 Consistent
with the Decreasing case, we observe the prevalence of younger cohorts at the beginning of the
period of observations, which is reversed as time goes on. Note that by the end of the 2016-2017
academic year the biases between cohorts are limited, as compared to the previous periods. This
is consistent with the predictions of the Decreasing case.
Fall16
Spring17
Fall14
Spring15
Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug
Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep
0
1
2
0
1
2
Bias Third to Second Fourth to Third Fifth to Fourth Sixth to Fifth
Figure 8: Biases Between Cohort Types
19 We look only at the biases between an older cohort and a cohort a year younger to make the picture readable.
We note, however, that the ordering of cohorts by their relative proportions and resulting pairwise biases during the
last periods are consistent with the predictions of the last part of Proposition 6. The bias between the second-years
and first-years is omitted because of its huge size that trumps the sizes of all other biases.
28

Figure 8shows the possibility of decreasing, increasing, hill-shaped, and dip-shaped patterns
that are broadly consistent with the patterns generated by the model. Medium-run dynamics of
the biases often have fluctuations, but by the end of the semester, the biases tend to converge to
fixed values. The sizes of the long-run biases range between 0.6 and 2.5 with little stability between
semesters.
5.2.3 Summary
We summarize the patterns observed in the data in the following results.
Result 1 (Response of relative proportions to recruitment).The dynamic response of relative
proportions to recruitment is consistent with the model’s predictions and is characterized by three
features. First, there is rapid growth in relative proportions immediately after recruitment. Second,
this growth decays within about a week after recruitment. Third, relative proportions continue to
grow in the subsequent months until converging to upper limits.
Result 2 (Selection bias).The subject pool exhibits short-run and long-run selection biases between
types. There is a small long-run selection bias between males and females, while the long-run biases
between cohort types are much larger. For cohorts, by the end of the observation period each older
cohort dominates a cohort a year earlier, which is consistent with the Decreasing case of the time-
variant version of the model.
Result 3 (Response of selection bias to recruitment).The immediate dynamic response of biases
to recruitment is consistent with the model and tends to be either decreasing, increasing, dip-shaped,
or hill-shaped. The most common type of response is a decreasing response to recruitment. Over
time, biases converge to fixed values.
6 Structural Estimation
6.1 Estimation Procedure
Consider a discrete-time version of equation (2) with ∆tequal to one day (the data are daily time-
series in our case). For each type i∈ N , the growth in relative proportion follows the following
29

difference equation.
xi
t+1 −xi
t= (αi−xi
t)(pie−δit+qi1x1
t+. . . +qiN xN
t).(9)
Our goal is to estimate the structural parameters of the model, i.e., the immediate spontaneous
rate pi, the decay rate δi, the potential proportion αi, as well as the imitation rates qi1, . . . , qiN .
We first note that due to an almost perfect linear correlation between relative proportions
among gender types and cohort types, the imitation rates cannot be separately identified. We can
estimate, however, the aggregate imitation rate qifor each type. If xj
t=kijxi
t, then
qi1x1
t+. . . +qiN xN
t= (ki1qi1+. . . +kiN qiN )
| {z }
qi
xi
t.
We then assume a non-linear trend model of the form
xi
t=f(t, xi
0|pi, qi, δi, αi) + i
t,
where the non-linear trend f(·) is a function of the starting value xi
0and time tconditional on the
parameters of the model θi≡(pi, qi, δ i, αi), and i
tis mean-zero noise term. The non-linear trend
function f(·) is defined implicitly as a solution to the difference equation (9). The identification of
the parameters governing the two channels is ensured by our assumption on the functional form of
decay and the fact that the recruitment at GMU occurs over a short period of time at the beginning
of each semester.20 The potential proportion parameter is identified by the data on the relative
proportions at the end of a semester. We estimate the model by minimizing the sum of squared
deviations of the observed time-series xi
tfrom the predicted values ˆxi
t(θi) = f(t, xi
0|θi):
ˆ
θi= arg min
θi
T
X
t=0 xi
t−ˆxi
t(θi)2.
20 If recruitment procedures occurred throughout a semester that would complicate the separate identification of
the two channels.
30

6.2 Estimation Results
6.2.1 Gender Types
Table 3shows the estimation results for gender types. The parameter estimates vary across
semesters for both males and females, potentially due to differences in recruitment procedures
and students’ response to them. For females, the estimates of prange from 0.17 to 0.462, with p=
0.279 for the average semester. The estimate for the average semester implies that in a case when
there is no indirect channel, and the starting relative proportion is zero, the relative proportion
would grow to the 28% of the potential proportion in the first day after recruitment. Males, on
average, tend to be more responsive to recruitment than females. However, the results are mixed
if one looks at the estimates for individual semesters. For males, the estimates of prange from
0.089 to 0.456, with p= 0.303 for the average semester. The estimate for the average semester
implies that in a case when there is no indirect channel and the starting relative proportion is zero,
the relative proportion would grow to the 30% of the potential proportion in the first day after
recruitment.
Table 3: Estimation Results for Gender Types
Parameter Fall 14 Spring 15 Fall 16 Spring 17 Average
Panel A. Female
p0.389 (0.009) 0.462 (0.005) 0.170 (0.040) 0.375 (0.034) 0.279 (0.009)
q1.061 (0.036) 0.358 (0.009) 0.004 (0.001) 0.487 (0.026) 0.629 (0.023)
δ1.449 (0.149) 0.429 (0.010) 0.727 (0.031) 0.876 (0.332) 0.520 (0.035)
α0.039 (0.0001) 0.056 (0.00003) 0.046 (0.007) 0.042 (0.0001) 0.046 (0.0001)
Panel B. Male
p0.316 (0.022) 0.456 (0.037) 0.089 (0.062) 0.407 (0.060) 0.303 (0.005)
q1.240 (0.031) 0.202 (0.043) 0.010 (0.184) 0.327 (0.111) 0.629 (0.014)
δ1.239 (0.312) 0.443 (0.019) 0.520 (0.085) 0.518 (0.045) 0.522 (0.014)
α0.039 (0.0001) 0.061 (0.003) 0.063 (0.020) 0.047 (0.003) 0.048 (0.00005)
Observations 136 218 78 188 135
Notes: The table reports the estimates of the structural parameters of the model for each semester,
as well as for the average semester, broken down by gender. The standard errors are based on
500 bootstrap replication.
The estimates of the imitation rate qshow that the indirect channel is important for recruit-
ment. For females, the estimates of qrange from 0.004 to 1.061, with q= 0.629 for the average
31

semester. The strength of the indirect channel, on average, is virtually identical for males and
females despite some variation across individual semesters. The estimates of qfor males range from
0.01 to 1.24, with q= 0.629 for the average semester. Despite the high chances of being recruited
through interaction with peers, the strength of the indirect channel is limited due to low overall
participation.21
Another characteristic of recruitment effectiveness is the decay rate δ. For females, the estimates
of δrange from 0.429 to 1.449, with δ= 0.52 for the average semester. This estimate implies that
the spontaneous rate drops by half, on average, in 1.332 days. After a week, the spontaneous rate
drops by 97%. For males, the decay rate, on average, is not significantly different from the decay
rate for females. This result becomes mixed, however, if one looks at the individual semesters. The
estimates of δfor males range from 0.443 to 1.239, with δ= 0.522 for the average semester. This
estimate implies that the spontaneous rate drops by half, on average, in 1.328 days. After a week,
the spontaneous rate drops by 97%.
The low overall participation in the subject pool is evidenced by low estimates of the potential
proportions αfor both types. For females, the estimates of αrange from 0.039 to 0.056, with
α= 0.046 for the average semester, meaning that, on average, only 4.6% of the female student
population is in the subject pool in our sample. For males, the estimates of αrange from 0.039 to
0.063, with α= 0.048 for the average semester, meaning that, on average, 4.8% of the male student
population is in the subject pool, which is slightly higher than the corresponding value for females.
This difference implies a long-run pairwise bias between males and females of 1.054.
Overall, the proposed model does a remarkably good job at quantitatively matching the patterns
observed in the data. Figure 9illustrates this point by plotting the actual relative proportions for
the average semester against the values predicted by the model (left and middle panels), as well as
the actual pairwise bias between males and females for the average semester against the predicted
bias (right panel). The dashed horizontal lines on the graphs for relative proportions correspond
to the estimated values of α. The time is measured in weeks starting from the date of recruitment.
The predicted values track the actual data very closely. In particular, the estimated model is
capable of producing the three main features of the dynamics of relative proportions: a rapid growth
21In other words, conditional on interacting with a peer who is already in a subject pool, the chances of being
recruited are high, but since few peers are in a subject pool, the overall effect is small.
32

immediately after recruitment, a quick decay in this growth within a week after recruitment, and a
slow growth in subsequent months with a convergence to an upper limit by the end of a semester.
The predicted bias makes the trend in the data sharper and can be described by a hill-shaped
pattern. Immediately after recruitment, the bias spikes due to a higher spontaneous rate for males
than for females. The bias then slowly converges to its long-run value. The graph of the bias also
highlights the importance of the dynamic effects in evaluating the selection problem. Evaluating
the selection problem at the beginning of the recruitment cycle would yield different results as
compared to the end of the cycle.
Female
Male
Bias
0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18
1.03
1.04
1.05
1.06
1.07
0.030
0.035
0.040
0.045
0.025
0.030
0.035
0.040
0.045
Week
Actual Fitted Counterfactual
Figure 9: Time-series of Relative Proportions and Male-Female Bias for the Average Semester
To further illustrate the importance of the indirect channel, we conduct a counterfactual simu-
lation in which this channel is completely shut down. Figure 9presents the results for the simulated
relative proportions of females and males (left and center panel) and the implied male-female bias
(right panel). As is evident from the picture, shutting down the indirect channel would have a
dramatic effect on the relative proportions and bias. The relative proportions of both types would
never reach their potential proportions by the end of the semester. The relative proportion would
be virtually stuck at 0.036 for females and 0.039 for males. These numbers represent 53% and
64% of the total gain in participation (the difference between a potential proportion and a starting
value) for females and males, respectively. The indirect channel thus accounts for roughly one-half
33

to one-third of the total participation gain. The bias would be stuck at 1.064, which is 1% larger
than the ratio of the potential proportions.22
6.2.2 Cohort Types
Table 4presents the estimation results for cohort types. We restrict our attention to the average
semester and the first four cohorts. Individual semesters, for the most part, do not provide enough
variation in the data to allow for a meaningful estimation, which is also the case for the cohorts of
fifth- and sixth-years for the average semester.
Table 4: Estimation Results for Cohort Types
Parameter First-years Second-years Third-years Fourth-years
p−0.009 (0.015) 0.338 (0.012) 0.342 (0.007) 0.317 (0.021)
q1.865 (0.582) 0.284 (0.024) 0.427 (0.017) 0.166 (0.030)
δ0.048 (4.895) 0.382 (0.016) 0.449 (0.016) 0.403 (0.019)
α0.026 (0.0001) 0.086 (0.002) 0.068 (0.0002) 0.063 (0.001)
Observations 136 135 135 135
Notes: Reports the estimates of the structural parameters for each cohort for the
average semester. The standard errors are based on 500 bootstrap replications.
The table reveals dramatic differences among cohorts regarding their responsiveness to direct
and indirect recruitment, as well as potential proportions. The effectiveness of the direct channel is
lowest for the cohort of first-years. The estimate of the spontaneous rate for this cohort is virtually
zero,23 suggesting that the first-years do not respond, or respond weakly, to direct recruitment
events. The estimates of the spontaneous rates for the three older cohorts are significantly different
from zero. The third-years have the highest point estimate of p= 0.342. This number implies that
if there was no indirect channel and a starting relative proportion was zero, the relative proportion
would grow to the 34% of the potential proportion in the first day after recruitment. The differences
in spontaneous rates between the second-, third-, and fourth-years are small and not statistically
significant.
The effectiveness of the indirect channel is highest for the first-years with an estimate of q=
1.865. This estimate is in stark contrast to the virtually non-existent direct channel for the first-
22 The absence of the indirect channel does not necessarily imply a larger bias. The differences in the spontaneous
and decay rates will determine the size of the bias.
23While the point estimate is, in fact, negative, which is not allowed by the model, we interpret it as being virtually
zero based on its magnitude and standard error.
34

years, suggesting that the indirect channel plays a dominant role in the participation decisions of
this cohort. There is, however, a substantial amount of uncertainty around the estimate of qfor the
first-years. Compared to the first-years, the estimates of the imitation rates among the remaining
three cohorts are much smaller, with the fourth-years having the lowest estimate of q= 0.166.
Overall, the effectiveness of the indirect channel tends to decline with a cohort’s age.
There is substantial uncertainty in the estimate of δfor the first-years, which is not surprising
given the low estimate of the spontaneous rate. The estimates of the decay rate for the remaining
three cohorts range between 0.382 for the second-years and 0.449 for the third-years. These numbers
imply that, for instance, the effectiveness of the direct channel for the third-years would drop by
half in 1.544 days and that in a week following recruitment the spontaneous rate would drop by
96%, which is similar to what we found using gender types. Overall, the estimates of the decay
rates tend to increase with a cohort’s age.
The estimates of the potential proportions show low overall participation across the cohorts.
The second-years have the highest potential proportion of 0.086, while the first-years have the lowest
potential proportion of 0.026. These numbers imply that, on average, only 8.6% of the second-years
and 2.6% of the first-years participate in the subject pool. The third- and fourth-years lie in between
the other two cohorts in terms of their potential proportions.24
Just like in the case of gender types, the estimated model for cohorts does a remarkably good
job at matching the data. This is evident from Figure 10 in which each panel shows the time-series
of the actual relative proportion for a given cohort along with the model’s fit. The horizontal
dashed lines, as before, indicate the estimates of α, and the time is measured in weeks starting
from the date of recruitment. The three key features of the data—a rapid growth immediately after
recruitment, a quick decay in this growth within a week after recruitment, and a slow growth in
subsequent months with convergence to an upper limit by the end of a semester—are well-captured
by the model.
Figure 11 shows the time-series of pairwise biases between subsequent cohorts: the bias between
the second- and first-years, the bias between the third- and second-years, and the bias between the
fourth- and third-years. The fitted values of the bias track the actual values quite closely. The bias
24Note that since we are dealing with the average semester in cohort estimates, it is not possible to track how the
patterns of potential proportions across cohorts change by semester.
35

First-years
Second-years
Third-years
Fourth-years
0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18
0.025
0.050
0.075
Week
Actual Fitted Counterfactual
Figure 10: Time-series of the Relative Proportions of Cohorts for the Average Semester
between the second- and first-years has a hump-shaped pattern. This bias grows initially, reaches a
peak, and then slowly converges to the long-run bias of 3.302 defined by the ratio of the potential
proportions. The bias between the third- and second-years exhibits a decreasing pattern, however,
it is worth noting that since the bias value goes well below 1, it implies a stronger bias between
the two cohorts. The bias between these two cohorts converges over time to the long-run value of
0.781. The bias between the fourth- and third-years exhibits a similar decreasing pattern but is
more complex. After the decline in the first half of the semester, the bias starts to increase slightly
in the second half of the semester towards the long-run value of 0.931.
Second to First
Third to Second
Fourth to Third
0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18
0.90
0.93
0.96
0.99
0.8
0.9
1.0
3
4
5
6
7
Week
Actual Fitted Counterfactual
Figure 11: Time-series of Biases between Cohorts for the Average Semester
36

To highlight the importance of the indirect channel of recruitment, we conduct a counterfactual
analysis in which we completely shut down the indirect channel. The resulting time-series of
counterfactual relative proportions are shown in Figure 10. In the absence of the indirect channel,
the relative proportions would virtually stop growing after two weeks and would never reach the
potential proportions. The long-run values of the counterfactual relative proportions would only
attain around two-thirds of the total gain in participation (the difference between a potential
proportion and a starting value) for the three older cohorts. The relative proportion of the first-
years would remain virtually constant in the absence of the indirect channel. We can conclude that
the indirect channel accounts for roughly one-third of the total gain in participation.
Figure 11 shows the time-series of counterfactual biases between cohorts in the absence of the
indirect channel. The bias between the second- and first-years would be increasing and exceed the
ratio of the potential proportions by a large degree. The bias between the third- and second years
would be slightly lower (a weaker bias) than the ratio of potential proportions. The same would
also hold for the bias between the fourth- and third-years.
7 Conclusion
We study the dynamics of the selection problem in economic experiments. Our analysis shows
that while a static selection problem leads to biases in the estimates of types shares and treatment
effects, a time-varying selection problem further complicates potential treatment effect biases. The
introduction of dynamics also helps to explain some of the existing contradictions in the literature
(Slonim et al.,2013;Cleave et al.,2013). In order to understand the dynamic nature of the selection
problem, we develop a model of participation in a subject pool. The model assumes that agents’
participation evolves over time and is driven by the two channels: the direct channel of recruitment
and the indirect channel of agents’ interaction. Differences in potential proportions drive the long-
run selection biases, while differences in initial participation levels and participation rates drive
the short-run biases between agent types. The modification of the model in which types are time-
variant results in the possibility of long-run biases even when potential proportions are identical
across types.
37

In our empirical analysis of the recruitment data from ICES at GMU we find that the participa-
tion dynamics are consistent with the model’s predictions. We find evidence of short- and long-run
selection biases between males and females, as well as between cohorts. The counterfactual analysis
of the data using an estimated model shows that the indirect channel accounts for roughly one-third
of the total participation gain for both gender and cohort types. We use the model to show that
the selection bias would be higher or lower, depending on the type, in the absence of the indirect
channel.
Our findings imply that networks effects play a crucial role in shaping the dynamics of the
selection problem. The presence of the dynamic effects in the selection problem, in turn, leads
to several important policy implications for methodology, design of experiments, and recruitment.
The methodological implication is that future studies of the selection problem should address its
dynamic nature. For example, the measurements of selection biases could be done at various point
in time throughout the semester. This would help to avoid the seemingly contradictory results
about the presence of the selection problem in a subject pool. The implication for experimental
design is that studies using a between-subject design would benefit from randomizing treatments
within a session. This practice would minimize the potential treatment effect biases caused by
the variation in selection biases over time. Alternatively, if randomization to treatment within a
session is infeasible, sessions should be kept temporally close. The implication for recruitment is
that it would help to leave a short burn-in period after a main recruitment event. Since selection
biases tend to stabilize in the long-run, leaving a burn-in period would minimize the selection biases
caused by short-run fluctuations.
We propose that future research on the selection problem should focus more on its dynamics. A
major obstacle to studying the dynamics of the selection problem is the absence of high-frequency
subject pool data on unobservable subject characteristics, such as preferences and personality. The
lack of such data forced our analysis to focus on observable characteristics that are recorded at a time
of registration for a subject pool. While the dynamics of observable characteristics may provide
some insights into the dynamics of unobservable characteristics, a more direct measurement of
unobservables is desirable. Supplementing basic demographic data with preference and personality
data elicited at a time of registration would be highly beneficial.
38

References
Abeler J, Nosenzo D (2015). “Self-Selection Into Laboratory Experiments: Pro-Social Motives
Versus Monetary Incentives.” Experimental Economics,18(2), 195–214.
Averell L, Heathcote A (2011). “The Form of the Forgetting Curve and the Fate of Memories.”
Journal of Mathematical Psychology,55(1), 25 – 35. Special Issue on Hierarchical Bayesian
Models.
Bass FM (1969). “A New Product Growth for Model Consumer Durables.” Management science,
15(5), 215–227.
Biernacki P, Waldorf D (1981). “Snowball Sampling: Problems and Techniques of Chain Referral
Sampling.” Sociological Methods & Research,10(2), 141–163.
Cappelen AW, Nygaard K, Sørensen EØ, Tungodden B (2015). “Social Preferences in the Lab:
A Comparison of Students and a Representative Population.” The Scandinavian Journal of
Economics,117(4), 1306–1326.
Castillo M, Freer M (2018). “Revealed differences.” Journal of Economic Behavior & Organization,
145, 202–217.
Charness G, Gneezy U, Kuhn MA (2012). “Experimental methods: Between-subject and within-
subject design.” Journal of Economic Behavior & Organization,81(1), 1–8.
Cleave BL, Nikiforakis N, Slonim R (2013). “Is There Selection Bias in Laboratory Experiments?
The Case of Social and Risk Preferences.” Experimental Economics,16(3), 372–382.
Dohmen T, Falk A, Golsteyn BHH, Huffman D, Sunde U (2017). “Risk Attitudes Across The Life
Course.” The Economic Journal,127(605), F95–F116.
Ebbinghaus H (1885/1974). Memory: A Contribution to Experimental Psychology. New York:
Dover.
Exadaktylos F, Esp´ın AM, Bra˜nas-Garza P (2013). “Experimental Subjects Are Not Different.”
Scientific Reports,3, 1213 EP.
Falk A, Becker A, Dohmen T, Enke B, Huffman D, Sunde U (2018). “Global Evidence on Economic
Preferences.” The Quarterly Journal of Economics,133(4), 1645–1692.
Falk A, Heckman JJ (2009). “Lab Experiments Are a Major Source of Knowledge in the Social
Sciences.” Science,326(5952), 535–538.
Falk A, Meier S, Zehnder C (2013). “Do Lab Experiments Misrepresent Social Preferences? The
Case Of Self-Selected Student Samples.” Journal of the European Economic Association,11(4),
839–852.
G¨achter S (2010). “(Dis) Advantages of Student Subjects: What Is Your Research Question?”
Behavioral and Brain Sciences,33(2-3), 92–93.
Gillen B, Snowberg E, Yariv L (2019). “Experimenting with Measurement Error: Techniques with
Applications to the Caltech Cohort Study.” Journal of Political Economy,127(4), 1826–1863.
39

Harrison GW, Lau MI, Rutstr¨om EE (2009). “Risk Attitudes, Randomization to Treatment, and
Self-Selection into Experiments.” Journal of Economic Behavior & Organization,70(3), 498 –
507.
Henrich J, Heine SJ, Norenzayan A (2010). “The Weirdest People in the World?” Behavioral and
Brain Sciences,33(2-3), 61–83.
Jackson MO, L´opez-Pintado D (2013). “Diffusion and Contagion in Networks with Heterogeneous
Agents and Homophily.” Network Science,1(01), 49–67.
Krawczyk M (2011). “What Brings Your Subjects to the Lab? A Field Experiment.” Experimental
Economics,14(4), 482–489.
Levitt SD, List JA (2007). “What Do Laboratory Experiments Measuring Social Preferences Reveal
About The Real World?” Journal Of Economic Perspectives,21(2), 153–174.
l’Haridon O, Vieider FM, Aycinena D, Bandur A, Belianin A, Cingl L, Kothiyal A, Martinsson P
(2018). “Off the Charts: Massive Unexplained Heterogeneity in a Global Study of Ambiguity
Attitudes.” The Review of Economics and Statistics,100(4), 664–677.
List JA (2009). “Social Preferences: Some Thoughts From The Field.” Annual Review of Eco-
nomics,1(1), 563–579.
Murre JMJ, Dros J (2015). “Replication and Analysis of Ebbinghaus’ Forgetting Curve.” PLOS
ONE,10(7), 1–23.
Slonim R, Wang C, Garbarino E, Merrett D (2013). “Opting-In: Participation Bias in Economic
Experiments.” Journal of Economic Behavior & Organization,90, 43–70.
Smith VL (1989). “Theory, Experiment and Economics.” Journal of Economic Perspectives,3(1),
151–69.
40

Appendices
A Experimental Instructions
In this paper we are using the entire recruitment data from an experimental laboratory at the
Interdisciplinary Center for Economic Science (ICES) at George Mason University (GMU) rather
than data from a particular experiment. Therefore, instead of describing experimental instructions
for a particular experiment, we describe the recruitment procedures of the experimental laboratory.
The recruitment system at the ICES experimental laboratory in its current form (the recruiter,
henceforth) was launched in Fall 2014. The recruiter is used to manage the subject pool and, in
particular, to invite subjects to participate in experimental sessions. Two types of recruitment
events are conducted on campus to populate the subject pool. First, at the beginning of each
semester the recruiter sends a generic email to the entire student population at GMU. The email
encourages students to sign up for participation in economic experiments by registering in the
recruiter. A typical email is presented in Figure A.1. The email emphasizes the time commitment
that a typical experiment requires and that students can expect monetary compensation for their
time. A second type of recruitment events, which however did not occur within our observation
period, is class visits. This type of recruitment involves representatives of the ICES who come to
large undergraduate classes, deliver a short talk designed to encourage participation in economic
experiments, and distribute flyers with information on how to sign up. The content of the talk,
for the most part, mirrors the content of the recruitment email. Table A.1 shows the timeline of
recruitment events at the ICES during our observation period.
Upon following the link to the recruiter, a potential subject fills out demographic information,
such as gender, ethnicity, birth year, and major. He or she also gives consent to receiving invitations
from the recruiter to participate in experimental sessions. After a subject’s account is created, the
recruiter records information on when an account was created, when an account was last updated,
how many sessions a subject participated in, and when were the first and last times a subject
participated in a session.
The recruiter also keeps track of whether an account is active or not. At the beginning of
each semester, the recruiter de-activates all accounts and sends an email to all registered subjects
41

From:no-reply@ices-experiments.com
Subject: ICES Invitation
Date: November 7, 2018 at 3:31 PM
To:alekseev@chapman.edu
All Mason students have the opportunity to advance Mason’s research and
earn cash at the Interdisciplinary Center for Economic Science. You will
receive invitations for experiments after you sign up. All participants are
monetarily compensated for their time.
Join Us
The Interdisciplinary Center for Economic
Science is an international leader in economics
research.
Have questions? Ask Lab
Manager Matt McMahon. Please
email
mmcmaho8@masonlive.gmu.edu
or call 703-993-8583
Participating in experiments has
no effect on course grades. There
is no limit to the number of
experiments you may participate
in. Participants must be able to
respond to visual cues on a
computer. Sign-in to payment is
30-90 minutes.
...
example@email.com
555-555-5555
Figure A.1: An Example of a Recruitment Email
42

Table A.1: Timeline of Recruitment Events
Event ID Type Semester Start Date End Date
1 email Fall14 2014-09-05 2014-09-06
2 email Spring15 2015-01-23 2015-01-24
3 email Fall15 2015-09-13 2015-09-14
4 email Fall16 2016-10-27 2016-11-04
5 email Spring17 2017-01-26 2017-01-27
6 class Fall15 2015-09-28 2015-09-30
7 class Fall15 2015-10-01 2015-10-02
8 class Fall15 2015-10-06 2015-10-07
9 class Fall15 2015-12-08 2015-12-13
10 class Spring16 2016-02-01 2016-02-06
11 class Spring16 2016-02-08 2016-02-13
12 class Spring16 2016-04-11 2016-04-16
13 class Spring16 2016-04-25 2016-04-30
asking them to re-activate their account thus confirming their willingness to further participate in
economic experiments. Subjects who successfully re-activate their accounts become active, while
subjects who do not re-activate their accounts remain non-active. Those latter subjects effectively
drop out of the subject pool.
B Proofs
Proposition 1
Proof. If a subject pool does not have a selection problem, then there is a constant ¯α, such that
xt= (¯α, ¯α, . . . , ¯α). Clearly, any pairwise bias will be equal to 1 in this case. Similarly, if all the
pairwise biases are equal to 1, they can be written as bij
t= ¯α/¯α, which implies that xt= ¯αı.
Proposition 2
Proof. Consider the ratio of the shares
mI
˜mI=mI+mIIbII,I= 1 + mII bII,I−1.
If the ratio of the shares is one, then mII bII,I−1= 0, which implies that bII,I= 1 (apart from the
trivial case when there is just one type in the population). Similarly, if bII,I= 1 then mI/˜mI= 1.
43

Proposition 3
Proof. Consider the difference between an estimated treatment effect and a treatment effect in a
reference population:
∆Y−∆˜
Y=∆YI−∆YIImI−˜mI.
If the two treatment effects are identical, ∆Y−∆˜
Y= 0, then either ∆YI= ∆YII (homogeneous
treatment effects) must be true or mI= ˜mI(no selection problem) must be true, or both. Similarly,
if either condition is true (or both), then ∆Y= ∆ ˜
Y.
Proposition 6
Proof. Consider the asymptotic pairwise bias between two subsequent cohorts:
lim
t→∞
xi+1
t
xi
t
=1−(1 −p−qx)i+1
1−(1 −p−qx)i.
Since p+qx < 1, the fraction is greater than one. Differentiating the fraction w.r.t iyields
d
di 1−(1 −p−qx)i+1
1−(1 −p−qx)i!=ln(1 −p−qx)(1 −p−qx)i(p+qx)
(1 −(1 −p−qx)i)2.
This expression is negative since p+qx < 1. Moreover, consider cohorts iand i+j. The pairwise
bias is increasing in j, since
1−(1 −p−qx)i+j
1−(1 −p−qx)i>1−(1 −p−qx)i+j0
1−(1 −p−qx)iif j > j0.
44