Content uploaded by Aleksandr G. Alekseev

Author content

All content in this area was uploaded by Aleksandr G. Alekseev on Aug 12, 2019

Content may be subject to copyright.

Selection in the Lab: A Network Approach∗

Aleksandr Alekseev†Mikhail Freer‡

August 12, 2019

Abstract

We study the dynamics of the selection problem in economic experiments. We show that

adding dynamics signiﬁcantly complicates the eﬀect of the selection problem on external valid-

ity and can explain some contradictory results in the literature. We model the dynamics of the

selection problem using a network model of diﬀusion in which agents’ participation is driven by

the two channels: the direct channel of recruitment and the indirect channel of agent interaction.

Using rich recruitment data from a large public university, we ﬁnd that the patterns of partici-

pation and biases are consistent with the model. We ﬁnd evidence of both short- and long-run

selection biases between student types. Our empirical ﬁndings suggest that network eﬀects play

an important role in shaping the dynamics of the selection problem. We discuss the implications

of our results for experimental methodology, design of experiments, and recruitment procedures.

Keywords: selection problem, experiments, external validity, networks, diﬀusion, peer eﬀects

JEL codes: C32, C90, D85

∗We thank Daniel Houser for providing access to the data and Arthur Dolgopolov for helping with the extraction

and preparation of the data. This paper has greatly beneﬁted from suggestions by Glenn Harrison, Robert Slonim,

Nat Wilcox, and David Rojo-Arjona, as well as participants at the Economic Science Association World Meetings

(2017) and Southern Economic Association Meetings (2018).

†Economic Science Institute, Chapman University, One University Drive, Orange, CA, 92866, e-mail:

alekseev@chapman.edu, phone: +1 (714) 744-7083, ORCID: 0000-0001-6542-1920.

‡ECARES, Universit´e Libre de Bruxelles, Av. Franklin D. Roosevelt 50, CP 114/04, 1050 Bruxelles, Belgium.

1 Introduction

Experimental methods have become an indispensable tool in economic science (Smith,1989;Falk

and Heckman,2009). Lab and ﬁeld experiments provide a steady source of behavioral insights for

new models and serve as a convenient test bed for existing models. With increasing reliance on

experimental methods, however, comes the need to conﬁrm the robustness of their procedures. An

important issue that has gained considerable attention in recent years is external validity, or whether

the results obtained with experimental subjects generalize to a relevant reference population.1A

major threat to external validity is the selection problem, i.e., the fact that subjects represent a

non-random sample of a reference population.

A large and growing literature has emerged to address the selection problem. This literature

has produced contradictory results.2A major limitation of the existing literature, which could also

serve as a potential source of contradictory results, is a view that the selection problem is static

and individual in nature. In this paper we attempt to overcome this limitation by studying, both

theoretically and empirically, how the dynamic and network eﬀects shape the selection problem.

We begin our analysis by showing that the presence of dynamics signiﬁcantly complicates the

eﬀect of the selection problem on external validity. Conditions under which selection problem does

not pose a threat to external validity in the static case are no longer valid in the dynamic case. In

particular, even if treatment eﬀects are homogeneous across types (a suﬃcient condition for external

validity in the static case), the estimated treatment eﬀects may still deviate from the population

treatment eﬀect in the dynamic case. The presence of dynamics is likely to create a treatment

eﬀect bias even a subject pool does not have a selection problem initially.

Having established the importance of dynamic eﬀects in the selection problem, we proceed

by developing a dynamic network model of participation in a subject pool. Our model is based

on a classic Bass (1969) network model of diﬀusion. The distinctive feature of our model, as

1For example, Levitt and List (2007) and List (2009) argue that social preferences, which remain one of the most

actively studied topic in behavioral and experimental economics, are grossly over-estimated in laboratory experiments.

Henrich et al. (2010) argue that the behavior and characteristics of student subjects in Western countries are somewhat

atypical when compared with other societies across the world. G¨achter (2010), on the other hand, argues that student

subjects are a great sample for the research questions economists usually ask.

2Some papers (Cleave et al.,2013;Exadaktylos et al.,2013;Falk et al.,2013) report that the behavior of students

who sign up for economic experiments is not signiﬁcantly diﬀerent from the behavior of student or general population

in social and risky settings. Other papers (Slonim et al.,2013;Cappelen et al.,2015) do ﬁnd signiﬁcant diﬀerences

between student participants and reference populations.

1

opposed to the models typically used in the literature (e.g., Abeler and Nosenzo (2015), Slonim

et al. (2013)), is that it recognizes the dynamic and social (via networking) nature of the decisions

to participate in a subject pool. The central assumption of the model is that agents can sign

up for participation via two channels. The ﬁrst channel is a direct channel represented by an

experimenter’s recruitment eﬀorts, such as e-mail invitations or class visits. The second channel

is an indirect channel represented by the interactions between agents within their network.3This

channel highlights the idea that an agent may be convinced to participate by her peers. In each

period, the agents who are not in a subject pool may arrive into the pool via either direct or indirect

channels at given type-speciﬁc rates.4

In our ﬁrst speciﬁcation of the model, an agent’s type is a time-invariant characteristic. The

model generates three key features of the participation dynamics: rapid growth immediately after

recruitment, a slowdown in growth later on, and steady growth towards an upper limit in subsequent

periods. A natural implication of this model is that over time the participation of each type

asymptotically approaches a value that we refer to as a potential proportion. Diﬀerences in potential

proportions, which may result from diﬀerences in preferences, such as risk preferences (Harrison

et al.,2009), is what drives the long-run selection biases between types in this version of the model.

Additionally, if types have diﬀerent initial participation levels or have diﬀerent participation growth

rates, a subject pool will also exhibit short-run selection biases. Simulations show that a typical

dynamic pattern is a high initial bias that declines steadily in subsequent periods. Such a pattern

is consistent with the results in two recent studies (Slonim et al.,2013;Cleave et al.,2013) that

have almost identical designs but diﬀer in the points in time at which biases were evaluated.

In our second speciﬁcation of the model we allow types to be time-variant. We consider a case

when types are cohorts of agents. An important insight from this version of the model is that

long-run biases are possible even when potential proportions are the same across types that is not

the case with time-invariant types. The model predicts oversampling of older cohorts in a subject

pool relative to younger cohorts. The participation levels among types will be asymptotically lower

than their respective potential proportions, unlike in the model with time-invariant types, due to

3Recruitment through an indirect channel is related to a recruitment technique in sociology known as snowball

sampling (Biernacki and Waldorf,1981).

4Our decision to introduce two separate channels is motivated by two considerations. First, it is a more accurate

description of an actual recruiting process. Second, having two channels leads to diﬀerent dynamics of the selection

problem, and hence to diﬀerent policy implications, as compared to having only one channel.

2

the constant outﬂow of agents. The dynamic path and magnitude of the selection problem between

cohorts will depend on the network structure of a reference population.

We use recruitment data from the Interdisciplinary Center for Economic Science (ICES) at

George Mason University (GMU) and GMU’s registrar enrollment data to construct daily time-

series of student participation in the subject pool. Our observation period is from the Fall semester

of 2014 until the Spring semester of 2017. We have data on when each subject entered and left the

subject pool, as well as subjects’ characteristics. We compare the number of students of each type

who are in the subject pool to the number of students of that type in the student population to

identify selection biases.

While our theoretical model allows for any reference population, in the empirical analysis we

use the student population as our reference population. Our choice of the reference population is

motivated by convenience and the fact that the GMU registrar provides high-quality data on the

student population. Using alternative reference populations would have changed the scales of our

time-series but not the dynamic trends, which are the focus of the present study.

Similarly, even though our theoretical model allows for types to be unobservable, as well as ob-

servable, we choose to use observables in our empirical analysis. This choice is dictated by necessity:

the analysis of the dynamics of selection requires high-frequency data on student types. We are

aware of no recruitment system that currently features such high-frequency data on unobservable

characteristics. The analysis of the dynamics of observable types, however, does provide insights

into the dynamics of the unobservable types. If a decision to participate in a subject pool reﬂects

a subject’s preferences, then the trends in observable types should reﬂect these preferences. Put

diﬀerently, if the dynamics of unobservable types are ﬂat and there is a mapping between observable

and unobservable characteristics, the dynamics of observable types must also be ﬂat.

Our empirical analysis yields three main ﬁndings. First, we show that the dynamics of the

subject pool is consistent with the dynamics predicted by the model. In particular, we conﬁrm the

existence of potential proportions for each type. These patterns are observed regardless of whether

we deﬁne gender (time-invariant) or cohort (time-variant) as a type. Second, we ﬁnd evidence of

both short- and long-run biases between types. Males tend to be consistently oversampled relative

to females both in the short- and long-run, though the size of the bias is small. Cohorts data reveal

that younger cohorts are initially oversampled in the subject pool relative to older cohorts but over

3

time a reversal of this ranking occurs. Finally, we conduct the structural estimation of the model

and document the heterogeneity between types in terms of the strengths of the direct and indirect

channels. The counterfactual analysis shows that the indirect channel accounts for one-third of the

total participation gain and plays a substantial role is shaping the selection bias.

Three policy recommendations emerge from our analysis of the dynamics of the selection prob-

lem. First, future studies of the selection problem should recognize its dynamic nature. The mea-

surements of the selection problem, for example, could be done at various points in time throughout

a recruitment cycle. Ignoring the dynamic nature of the selection problem could lead to seemingly

contradictory results about the presence of the selection problem in a subject pool. Second, stud-

ies using a between-subject design would beneﬁt from randomizing treatments on a subject-level

within a session. That would minimize the potential treatment eﬀect bias caused by the dynamic

component the selection problem. If randomization to treatment within a session is infeasible,

keeping sessions temporally close (or even running them simultaneously at diﬀerent places) would

be the second-best option. Third, it would make sense to leave a short (one to two weeks) burn-in

period after a main recruitment event. Selection biases tend to stabilize in the long-run. Therefore,

leaving a burn-in period would minimize the selection biases caused by short-run ﬂuctuations.

Our paper contributes to the growing literature on the selection problem in economic exper-

iments by providing a theoretical framework that captures the dynamic and network nature of

participation in a subject pool, as well as empirical evidence on the dynamics of student participa-

tion and the selection problem. Other papers have proposed theoretical models of participation in a

subject pool. Abeler and Nosenzo (2015) assume that the aggregate utility of participation consists

of three parts: monetary utility, pro-social utility, and a ﬁxed cost. Monetary and pro-social parts

have diﬀerent weights, and these weights determine which motive is stronger. The model assumes

that recruitment changes potential participants’ priors about the expected monetary reward and

the need for social approval. Slonim et al. (2013) introduce a more general utility-of-participation

function that has four components: monetary reward, leisure time, intellectual curiosity, and so-

cial preferences. Their model predicts that students with lower income, more leisure time, higher

curiosity, and higher pro-social preferences are more likely to participate in a subject pool.

Our theoretical analysis highlights the mechanisms that drive short- and long-run selection bi-

ases and helps to explain some contradictory results in the existing literature. Falk et al. (2013)

4

look at the entire student population of the University of Zurich from 1998 to 2004 and compare

social preferences of students who signed up for participation in economic experiments with social

preferences of the general student population. As a measure of social preferences, they use informa-

tion from the school about students’ donation choices. The main ﬁnding is that pro-social behavior

does not diﬀer between the two groups. Cleave et al. (2013) study the student population at the

University of Melbourne by conducting experiments in classes and also inviting students to partici-

pate in economic experiments. Five months later, they compare the choices of students in the class

experiments to the choices of those students who agreed to participate in experiments, registered

in the database and showed up for a laboratory experiment. Their analysis reveals that the choices

are very similar across the two groups, as are their demographic characteristics. Slonim et al.

(2013), using a design similar to that of Cleave et al. (2013) but a shorter waiting period of several

weeks, do ﬁnd signiﬁcant diﬀerences in some characteristics and preferences of participants and

non-participants in lab experiments. Taken at face value, the results from the two former papers

contradict each other, however, viewing these results from the perspective of our dynamic model

alleviates the contradiction. It is likely that the results in Slonim et al. (2013) reﬂect the short-run

biases in the data, while the results in Cleave et al. (2013) reﬂect the long-run biases, which are

typically smaller. None of the existing models of participation can account for this pattern.

The remainder of the paper is organized as follows. Section 2deﬁnes the selection problem

and illustrates how the selection problem aﬀects external validity. Section 3introduces the model

with time-invariant types and presents theoretical results for the dynamics of participation and

selection bias. Section 4extends the model by allowing for time-variant types. Section 5describes

the data and presents the reduced-form results for participation and selection. Section 6contains

the procedure and results of the structural analysis of the data. Section 7concludes and discusses

directions for further research.

2 Selection Problem

Let N={1, . . . , N }be the set of types of agents in a population based upon a certain characteristic.

We call the characteristics that do not change over time time-invariant characteristics. Characteris-

5

tics that change over time are called time-variant characteristics. Characteristics can be observable

(such as demographic characteristics) or unobservable (such as preferences and personality traits).

Let ˜

xt≡(˜x1

t,...,˜xN

t)0be a vector of proportions of the number of agents of each type in

a subject pool relative to the total number of agents in a reference population at time t. A

reference population can be the population of people in a school that hosts a subject pool, the

population of a local area in which the school is located, the general population of a country, or

any other population deemed relevant by a researcher. For example, if we use risk aversion as a

characteristic and assign Type I to risk averse/neutral people and Type II to risk loving people,

then ˜

xt=10 = (0.1,0.2)0means that at the 10th period of observation a subject pool contained

10% of people in the reference population who are risk averse/neutral and 20% of people in the

reference population who are risk loving people. If we denote Stto be the total number of people

in a reference population at time t, then a subject pool can be deﬁned by the number of agents

of each type in it: St˜

xt. For example, if there are S10 = 10,000 people in a reference population

at time 10, a subject pool would be composed of 1,000 risk averse/neutral subjects and 2,000 risk

loving subjects.

While there are more risk loving subjects than risk averse/neutral subjects in this subject pool,

this does not necessarily cause a selection problem, since the shares of each type in a reference

population may also be diﬀerent. For instance, if there are twice as many risk loving people in

a reference population than risk averse/neutral people, then the composition of the subject pool

perfectly matches the composition of the reference population, and there is no selection problem.

Let m≡(m1, . . . , mN)0,ı0m= 1,(ıis a sum vector of order N), be a vector of shares of each type

in a reference population. In the previous example, if there are twice as many risk loving people as

risk averse/neutral people, the reference population is 33% risk averse/neutral and 66% risk loving,

and so m= (1/3,2/3)0. Let M≡diag(m) be a diagonal matrix with the population proportions

of each type on the diagonal. Then xt≡M−1˜

xt= (˜x1

t/m1,...,˜xN

t/mN)0is a vector of proportions

of the number of agents of each type in a subject pool relative to the shares of agents of these

types in a reference population. This quantity will be called a vector of relative proportions and its

dynamics is the main object of study in this paper.

6

What does it mean, then, to have a selection problem? A selection problem occurs when the

composition of a subject pool does not match the composition of a reference population. We can

formally deﬁne the selection problem as follows.

Deﬁnition 1 (Selection Problem).A subject pool is said to have a selection problem at time t

(asymptotically) if there is no constant ¯α∈[0,1], such that the vector of relative proportions can

be written as xt= ¯αı(limt→∞ xt= ¯αı).

Suppose that for some subject pool xt6= ¯αıfor any ¯αso that there is a selection problem at

time t. How bad is this problem? To answer this question, we can look, for instance, at pairwise

biases between types and deﬁne bij

t≡xi

t/xj

t, i, j ∈ N . This number determines the ratio of a

relative proportion of agents of type ito a relative proportion of type jat time t. If bij

t>1, there

are more agents of type iin a subject pool than agents of type jrelative to their population shares,

and the bigger is bij

t, the stronger is the selection bias between these two types. On the ﬂip side, bji

t

in this example will be less than one. The strength of the bias is therefore determined by how far

a pairwise bias is from 1. If all the pairwise biases are equal to one, there is no selection problem.

On the other hand, if there is at least one pairwise bias that is not equal to one, there is a selection

problem, and a subject pool is biased.

Proposition 1. A subject pool does not have a selection problem at time t(asymptotically) iﬀ for

any two types i, j ∈ N the pairwise bias bij

t= 1 limt→∞ bij

t= 1.

The existence of a selection problem makes it harder to extrapolate the results obtained from a

subject pool to a reference population. Two inference problems arise because of the selection prob-

lem. First, the estimated shares of each type will not be representative of the reference population.

Consider the example with two types of risk preferences. The share of Type I (risk averse/neutral

people) in the subject pool is given by5

˜mI≡˜xI

˜xI+ ˜xII =mI

mI+mIIbII,I.

5To simplify the exposition, below we focus on the case with only two types. However, the logic carries over to

a more general case.

7

The existence of a selection bias between the two types (bII,I6= 1) will drive the wedge between the

shares of each type in a subject pool and the actual shares in a reference population. The following

proposition follows.

Proposition 2. The shares of each type in a subject pool, ˜mi, will be identical to a reference

population shares, mi, iﬀ a subject pool does not have a selection problem.

If a research question is to measure the prevalence of a certain type in the population (e.g.,

based on risk aversion or altruism), then the presence of a selection problem will bias the results.

A selection problem will also lead to a second inference problem: biased estimates of treatment

eﬀects. To see this, consider again the case of two types. The average treatment eﬀect in a reference

population, ∆Y, is given by

∆Y=mI∆YI+mII∆YII = ∆YII +mI∆YI−∆YII ,

where ∆YIand ∆YII are the treatment eﬀects for the two types. In a subject pool, the average

treatment eﬀect, ∆ ˜

Y, is given by

∆˜

Y= ˜mI∆YI+ ˜mII∆YII = ∆YII + ˜mI∆YI−∆YII .

The following proposition follows.

Proposition 3. A treatment eﬀect in a subject pool will coincide with a treatment eﬀect in a

reference population iﬀ either (or both) condition holds: a) treatments eﬀects are homogeneous

across all types; b) a subject pool does not have a selection problem.

This implies that if neither condition holds, i.e., treatment eﬀects are heterogeneous and there is

a selection problem, a treatment eﬀect in a subject pool will be diﬀerent from a treatment eﬀect in

a reference population. The existence of heterogeneous treatment eﬀects across types is a common

scenario in many experimental settings (l’Haridon et al.,2018;Castillo and Freer,2018). Hence,

selection problem will be suﬃcient to cause biases in estimated treatment eﬀects.

If a selection problem is dynamic, then the inference problems become more complicated. Con-

sider the case with two types when the relative proportions of types vary in time and treatments

8

are assigned in diﬀerent moments in time. It is a common practice for experimental economists to

assign treatments on a session level in a between-subject design and then run diﬀerent sessions in

sequence, which creates a temporal separation of treatments (Charness et al.,2012). Assume that

at time tboth types receive the baseline condition that produces response Yi

0in Type iand that at

time t+ 1 both types receive the treatment condition that produces response Yi

1in Type i. Then

an average treatment eﬀect in a subject pool is given by

∆˜

Y= ∆YII + ˜mI

t∆YI−∆YII+ ∆ ˜mIYI

1−YII

1,

where ˜mI

tis share of Type I in a subject pool at time t, ∆ ˜mI≡˜mI

t+1 −˜mI

tis the change in a subject

pool share of Type I from time tto t+ 1. The diﬀerence between the treatment eﬀects in a subject

pool and a reference population, a treatment eﬀect bias, is then

∆Y−∆˜

Y=∆YI−∆YIImI−˜mI

t

|{z }

Static component

−∆ ˜mIYI

1−YII

1

| {z }

Dynamic component

,

where the ﬁrst term represents the static component of the bias in an average treatment eﬀect, and

the second component represents the dynamic component of the bias. The following proposition

then holds.

Proposition 4. In the presence of a dynamic selection problem, an estimated treatment eﬀect in a

subject pool will coincide with a treatment eﬀect in a reference population iﬀ the static component

equals the dynamic component.

In contrast to the previous case with no time-varying selection problem, it is possible to have

a bias in an estimated average treatment eﬀect even if treatment eﬀects are homogeneous across

types. On the other hand, even if there is a selection problem at time tand treatment eﬀects are

heterogeneous across types, it does not necessarily imply that there will be a bias in an average

treatment eﬀect. It is possible that the change in the selection bias will be just right to compensate

for the diﬀerences between treatment responses.6Such a fortunate scenario, however, is unlikely.

If we further assume that types diﬀer in their treatment eﬀects and treatment responses, which is

6The precise condition is YI

0−YII

0

YI

1

−YII

1

=mI

−˜mI

t+1

mI−˜mI

t

, assuming that the denominators are not zero.

9

reasonable to expect, the presence of the dynamic component will create a treatment eﬀect bias

even a subject pool does not have a selection problem initially. If a subject pool has a selection

problem initially, it is unclear a priori whether the dynamic component will make a treatment

eﬀect bias smaller or larger. It is clear, however, that the dynamic component complicates the

interaction between a selection problem and related inference problems and thus deserves a closer

consideration.

3 Model For Time-Invariant Types

In order to guide our empirical analysis in the next section and make testable predictions, we build

a simple model of the participation in a subject pool and potential bias among types with time-

invariant characteristics. We use Bass (1969) model of diﬀusion as a starting point and adapt it

to the environment of a subject pool.7Suppose that at the start of the observation period, t= 0,

there are S˜

x0agents in a subject pool. We allow the relative proportions xtto have upper limits

given by a vector of potential proportions α= (α1, . . . , αN)0, αj∈[0,1], j ∈ N . These potential

proportions will allow for the existence of asymptotic biases.

New subjects can arrive into a subject pool via two channels. The ﬁrst channel is a direct

channel of recruitment eﬀorts, such as emails, posters, and class visits. The growth rate at which

agents who are not a pool will join it immediately after a recruitment event is given by a vector of

spontaneous rates p≡(p1, . . . , pN)0, pj>0, j ∈ N . These rates can be diﬀerent across types. We

allow for a decay in the spontaneous rates, since recruitment events are typically infrequent and

short-lived. Let δ≡(δ1, . . . , δN)0be a vector of decay rates and let dt≡exp(−δt) be a vector of

decay factors.8Then at each time t, the vector of spontaneous rates is deﬁned as pt≡diag(dt)p.

The second channel is an indirect channel of agent interactions: agents who are already in a

subject pool can induce their peers to join the pool. For example, a person who earned a decent

amount of money in an experiment or just enjoyed being a part of it may share this experience with

her friends. This channel is captured by a matrix of imitation rates Q≡ {qij }, i, j ∈ N . Each

7We are certainly aware of more advanced network models with types, e.g., Jackson and L´opez-Pintado (2013).

However, we chose in favor of a simple model of diﬀusion to make the exposition more intuitive. Despite its simplicity,

the model is ﬂexible enough to incorporate the relevant features of the student pool.

8Research in psychology suggests that exponential decay is a reasonably close approximation of the so-called

“forgetting curve” (Murre and Dros,2015;Averell and Heathcote,2011;Ebbinghaus,1885/1974).

10

element qij >0 represents the growth rate in the participation of type ibecause of the interaction

with agents of type j. The diagonal elements of Qare own-imitation rates (imitation of the own

type), and the oﬀ-diagonal elements are cross-imitation rates (imitation of other types). Matrix

Qcan be thought of as a network structure of a reference population. The own-imitations rates

are allowed to diﬀer between types. It is natural to assume that the own-imitation rates are higher

than the cross-imitation rates, but this is not required. The strength of the indirect channel will

depend on the relative proportions of agents of each type in a pool: there has to be someone to

imitate, and the more agents of a given type are in the pool, the more likely they will inﬂuence

the decision of their peers. We assume that imitation rates are constant, but this can be relaxed

by assuming that they depend on the activity of an experimental laboratory. In this case, running

more experiments would result in a higher likelihood that participants will share their experience

with their friends.

Our decision to introduce two separate channels, as opposed to the direct channel only, is

motivated by two considerations. First, having two channels is a realistic description of an actual

recruiting process. For example, in some schools the majority of the agent inﬂow into a subject pool

occurs via word-of-mouth (indirect channel). Ignoring the indirect channel thus would considerably

limit the model’s descriptive ability. The second, and more pragmatic, consideration is that the

model with two distinct channels produces very diﬀerent dynamics of the selection problem as

compared to the model with the direct channel alone. Since diﬀerent dynamic patterns of the

selection problem will have diﬀerent policy implications, entertaining the possibility of the two

distinct channels and comparing their relative strength is relevant for policy recommendations.

In the practice of laboratory experiments, agents arrive into a subject pool from a student

population. However, a student population does not have to be a reference group. If a reference

group is chosen to be the general population, the rates in the model will simply reﬂect the joint

rate of transitioning from the general population into a student population and transitioning from

a student population into a subject pool. The choice of a reference group does not aﬀect the intra-

semester or intra-year dynamics of a subject pool (measured at a daily frequency in our application),

since the transition into a student population occurs at a much lower frequency (typically, yearly).

The choice of a reference group will, however, aﬀect the levels of time series.

11

Let the spontaneous and imitation rates be deﬁned over a small time period ∆t>0. The dy-

namics of the subject pool participation can then be described by the following system of equations:

˜xi

t+∆t= ˜xi

t+pi

t∆t(αimi−˜xi

t) + qi1˜x1

t

m1+. . . +qiN ˜xN

t

mN!∆t(αimi−˜xi

t), i ∈ N .(1)

This equation attributes the growth of a subject pool participation to the two channels described

above. The second term on the right-hand side represents the inﬂow of subjects due to recruitment

eﬀorts. The spontaneous rate is applied to the proportion of agents who are not currently in the

subject pool. Note that we multiply the share mby the potential proportion αto allow for non-full

participation, as typically only a fraction of a reference population participates in a subject pool.9

The third term reﬂects the inﬂow due to the imitation of the behavior of own types and other

types. The own- and cross-imitation rates are scaled by the relative proportions of agents: the

more agents participate in the subject pool, the stronger is the indirect channel.

Dividing both parts of equation (1) by the share mi, rearranging the terms, and taking the limit

∆t→0, we obtain the following system of diﬀerential equations:

˙xi= (αi−xi)(pi

t+q0

i·xt), i ∈ N ,(2)

where q0

i·is an i-th row of Q, or in matrix form,

˙

x= diag(α−xt)(pt+Qxt).(3)

Since the second terms on the right-hand side are positive, the steady state of the system is

simply the vector of potential proportions α. Starting from any initial value x06α, the relative

proportions will converge to the potential proportions: limt→∞ xt=α. The data on the relative

proportions at the end of a semester can then be used to infer the potential proportions. If the

potential proportions are diﬀerent across types, long-run biases will occur. The diﬀerences in the

potential proportions can be caused by diﬀerences in preferences. For example, since women are

9In the case when the reference population is diﬀerent from a student population, the potential proportion reﬂects

the joint long-term rate of transitioning from the reference population into a student population and from a student

population into a subject pool.

12

typically found to be more risk-averse than men (Falk et al.,2018), women may have lower potential

proportions than men. We summarize this result in the following proposition.

Proposition 5. A subject pool has a selection problem asymptotically iﬀ there is no constant ¯α,

such that the vector of potential proportions can be written as α= ¯αı.

The short-run dynamics of the relative proportions and biases are determined by the starting

values x0, the potential proportions and the rate parameters of the model. The time path of relative

proportions can be found analytically in some special cases. The ﬁrst case arises when there are

only own-imitation and no cross-imitation between types, and there are no subjects in the pool at

the beginning of the observation period. The formula below also assumes that the direct channel is

always present, i.e., there is no decay in the spontaneous rates. This case would apply in a situation

when, for example, recruitment posters are posted on a school’s announcement board and remain

there until the end of a semester.

Example 1. Assume that there is no cross-imitation between types, i.e., all the oﬀ-diagonal ele-

ments of the imitation matrix Qare zero, x0=0, and there is no decay, δ=0. Then the relative

proportions of each type evolve according to

xi

t=αi1−e−(pi+αiqii)t

1 + αiqii

pie−(pi+αiqii)t, i ∈ N .

The second case arises when there is no direct channel, and the indirect channel works only

through own-imitation. For example, some time might have passed after recruitment and sponta-

neous rates decayed to almost zero. In order for the subject pool to grow, some initial amount of

subjects has to be there. A subject pool will exhibit logistic growth in this case.

Example 2. Assume that there is no cross-imitation between types, (all the oﬀ-diagonal elements of

the imitation matrix Qare zero), that own-imitation rate is non-zero, that there is no direct channel

(pt=0), and that the subject pool starts with Sx0subjects in it. Then the relative proportions of

each type evolve according to

xi

t=αi

1 + αi−xi

0

x0e−αiqiit, i ∈ N .

13

The model, in general, generates a time path of relative proportions that is characterized by a

quick growth immediately after a recruitment event followed by a slow growth towards potential

proportions, as illustrated on Figure 1(panel A). Three features of the model produce this pattern.

First, a quick growth immediately after a recruitment event is caused by spontaneous rates. In the

model, spontaneous rates enter the growth equation directly, while imitation rates are scaled by the

relative proportions of imitated types. Therefore, imitation rates cannot account for initial rapid

growth. Figure 1(panel B) illustrates this point. It simulates the dynamics of the relative propor-

tions of two types when the direct channel is shut down (spontaneous rates are zero). The only

source of growth in the simulation is the indirect channel, which works because of non-zero starting

values. The relative proportions show steady but slow growth, reaching potential proportions only

by the end of the simulation period.

A sharp decrease in the growth rate of relative proportions is produced by the presence of

decay rates that dampen the eﬀect of recruitment and reduce spontaneous rates over time. In the

absence of decay, the relative proportions would very quickly converge to the potential proportions.

Figure 1(panel C) shows simulation results for the case of zero decay rates and when only the direct

channel is present. Relative proportions show a steep growth immediately after the beginning of

the simulation period and quickly reach potential proportions.

Slow growth of relative proportions in the subsequent periods is produced by imitation rates.

In the absence of imitations rates, the growth rate would plummet to almost zero, and relative

proportions would never reach potential proportions. Figure 1(panel D) shows the simulation

results featuring the direct channel with decaying rates and completely no indirect channel. After

initial rapid growth, relative proportions ﬂatten out and converge to steady states that are well

below potential proportions.

The model can produce several distinct dynamic patterns of biases. On Figure 2we focus on

the case with two types, in which there is a high initial bias and no asymptotic bias. Panel A

shows the simulation that features both direct and indirect channels, as well as decay, and all

the rates are identical across types. The only diﬀerence between types is in the starting values

of potential proportions. The initially high bias produced by diﬀerent starting values quickly

converges, following a recruitment event, to the bias implied by the ratio of potential proportions.

14

A. All Features Present

B. Zero Spontaneous Rates

C. Zero Decay

D. Zero Imitation Rates

0 50 100 150 0 50 100 150 0 50 100 150 0 50 100 150

0.02

0.03

0.04

0.05

Time

Figure 1: Simulated Relative Proportions Under Diﬀerent Parameter Values

Diﬀerences in spontaneous rates between types can alter the picture in two distinct ways.10

In particular, if the initial spontaneous rate of a type that starts low is higher than the initial

spontaneous rate of a type that starts high, a dip in the bias is observed, as Panel B illustrates.

A diﬀerent pattern occurs if the type that starts high also has a higher initial spontaneous rate,

as illustrated on Panel C. After an initially high level, the bias increases even further following a

recruitment event, after which the bias drops and converges to the level implied by the ratio of

potential proportions.

A. Drop

B. Dip

C. Hill

0 50 100 150 0 50 100 150 0 50 100 150

1.0

1.2

1.4

1.6

Time

Figure 2: Simulated Biases Under Diﬀerent Parameter Values

4 Model For Time-Variant Types

In the previous section we assumed that the population is static in that it consists of only time-

invariant types. The natural result of this assumption is that all the agents who can potentially

10 We focus on the eﬀect of spontaneous rates on the dynamics of the bias since these rates have the most impact.

Adding diﬀerences in imitation rates can produce even more complex patterns.

15

be in a subject pool eventually end up there. In order to model low overall participation and

asymptotic biases between types, we had to set potential proportions exogenously and make them

diﬀerent across types. In this section we are relaxing the assumption about a static population by

introducing cohorts of agents as time-variant types and look at how this changes our results. One

important consequence of this change is that non-full participation and biases between types will

arise endogenously, even when potential proportions are the same across types.

We assume that a refernce population consist of cohorts indexed by elements of a set N=

{0, . . . , N −1}, N >2. For simplicity, we assume that the shares of each cohort in a population are

identical: m≡(m0, . . . , mN−1)0= (1/N, . . . , 1/N)0. Agents who belong to cohort iin one period

move to cohort i+ 1 in the next period. Agents of cohort N−1 in the next period automatically

drop out of a subject pool (if they were there). At the same time, a new cohort, n= 0, arrives

every period. Each cohort represents a diﬀerent type of agents. For example, in the case of college

students, as students advance through their programs, their type (e.g., risk or time preferences)

changes due to new experiences and acquired knowledge.11 To make the point about endogenous

biases and non-full participation stronger, we assume that a ﬁxed proportion α∈[0,1] of each

cohort can potentially be recruited for a subject pool.12

As before, we assume that there are two channels through which agents may be induced to

participate in a subject pool. The direct channel is given by a vector of spontaneous rates p≡

(p0, . . . , pN−1)0, pi∈[0,1], i ∈ N . Importantly, the spontaneous rates are deﬁned over the same

period as cohorts, and thus we do not consider decay in these rates within a period. In other words,

the rates in this version of the model are deﬁned over academic years (or semesters) rather than

days.

The indirect channel is given by a matrix of imitation rates Q≡ {qij}, i, j ∈ N . It is natural

to assume that the probability to imitate own type is at least as big as that of other types, qii >

qij,∀i, j ∈ N . For instance, imitation rates can decline with the distance between cohorts, qij =

qii/(1 + d(i, j )), where d:N ×N 7→ R+is some distance function, such as |i−j|. This speciﬁcation

11 For example, Dohmen et al. (2017) show, using representative panel data from Germany and Netherlands, that

people become more risk averse as they age. In particular, the willingness to take risks declines almost linearly with

age.

12 As we will show shortly, the equality of the potential proportions does not lead to the asymptotic equality of

relative proportions in the case of time-variant types. Diﬀerences in the potential proportions, therefore, will only

make this result stronger.

16

captures the idea that, for example, sophomores are more likely to interact with freshmen and

juniors than with seniors, and are even more likely to interact among themselves.

Let ˜

xt≡(˜x0

t,...,˜xN−1

t)0be a vector of proportions of the total number of agents in each cohort,

who are in a subject pool at time t, to the total number of agents, S. Given some initial number

of people in a subject pool, S˜

x0, the dynamics of these proportions are described by the following

system of equations:13

˜xi+1

t+1 = ˜xi

t+piαmi−˜xi

t+X

j∈N

qij ˜xj

t

mjαmi−˜xi

t, i ∈ N /{N−1}.(4)

These equations state that the number of agents of cohort i+1 who are in the subject pool at time

t+ 1 consists of agents who belonged to a previous cohort one period ago plus those who arrived

at the subject pool via direct and indirect channels. Equations (4) can be viewed as a natural

extension of the system (1), with the diﬀerence being that types change over time. We assume that

the newly arrived cohort does not provide any agents for a subject pool, since recruitment eﬀorts

and agents’ interactions happen after cohorts are formed, so that ˜x0

t+1 = ˜x0

t= 0,∀t>0.

The proportion of all agents who are in a subject pool at time trelative to the total number of

agents is

˜xt=X

i∈N

˜xi

t.

Using this equation and equation (4), we ﬁnd that the dynamics of the proportion of all agents

relative to a total number of agents in a population are given by

˜xt+1 = ˜xt−˜xN−1

t+

N−2

X

i=0

piα

N−˜xi

t+

N−2

X

i=0

N−1

X

j=0

qij ˜xj

t

mjα

N−˜xi

t.(5)

Note that we have to subtract the agents who were in cohort N−1 at time tsince they leave the

subject pool in the next period.

Denote xt≡(˜x0

t/(αm0),...,˜xN−1

t/(αmN−1)) to be a vector of the proportions of agents of

each cohort who are in a subject pool at time trelative to their potential shares in a reference

population. We will call xta vector of relative proportions. Note that it is deﬁned diﬀerently than

in the previous section. We scale ˜

xtnot only by the shares m, but also by the potential proportions

13 In this version of the model we are focusing on the discrete time, so ∆t= 1.

17

α. In the context of time-invariant types, this deﬁnition would imply the absence of selection bias

in the long-run. In the context of time-variant types, however, this will not be the case. Similarly,

deﬁne xt≡˜xt/α to be the proportion of all agents in a subject pool relative to their potential

share. We will call this quantity a total relative proportion.

We now show that the selection problem occurs in the model with time-variant types even if the

types are a priori identical. We consider a symmetric case with no heterogeneity in spontaneous

or imitation rates across cohorts, i.e., pi=p, qij =q, ∀i, j ∈ N , and (without loss of generality) the

potential share α= 1. We bound the rate parameters by the condition p+qxt<1,∀tto ensure

that cohorts do not immediately reach their potential proportions. Under these assumptions, the

dynamics of the relative proportions can be rewritten as follows.

xi+1

t+1 =xi

t+p(1 −xi

t) + qxt(1 −xi

t), i ∈ N /{N−1}.(6)

Similarly, the dynamics of the total relative proportion are given by

xt+1 =xt−xN−1

t

N+p

N−1

N− xt−xN−1

t

N!

+qxt

N−1

N− xt−xN−1

t

N!

.

First, consider the asymptotic behavior of the subject pool. We are interested in the steady

state of the system when xi

t+1 =xi

t=xiand xt+1 =xt=x. Plugging these values into (6) yields

xi+1 =xi1−(p+qx)+p+qx.

This is a ﬁrst-order linear diﬀerence equation, whose solution is xi= 1 −1−(p+qx)i. Using

this solution, we can ﬁnd the stead-state equation for the total relative proportion:

x−1 + 1−1−(p+qx)N

N(p+qx)= 0.(7)

While a closed-form solution for xcannot be obtained, note that it will depend on the rate param-

eters of the model, as well as on the number of cohorts. Therefore, not everyone ends up in the

subject pool among those who can potentially be there, even if we wait long enough. Moreover,

18

this result has implications for the nature of the selection problem that would arise in a subject

pool.

Proposition 6. A subject pool has a selection problem asymptotically in the presence of cohorts.

The agents from older cohorts are oversampled in a subject pool, while agents from younger cohorts

are undersampled. The pairwise bias between two subsequent cohorts decreases in the cohort number.

The further cohorts are apart from each other the larger is the bias between them.

The intuition for this result is straightforward: agents in older cohorts have more chances

to be recruited over time. In the example with college students, if recruiting occurs every year,

juniors experience two recruitment periods (at the beginning of a school year), while sophomores

experienced only one. The problem will also exist for any time t. Thus we have shown that even

in the simplest symmetric case and potential proportion equal to 1, non-full participation and

selection problem arises endogenously.

We now consider possible asymmetric cases and compare them with the baseline results. We

consider four cases:

1. Baseline. Spontaneous, own- and cross-imitation rates are the same (pi=p, qij =q, ∀i, j ∈

N). This case is one from Proposition 6.

2. Constant. Spontaneous and own-imitation rates are constant for every cohort (pi=p, qii =

q, ∀i∈ N ), but the cross-imitation rates decline with the distance between the cohorts ac-

cording to

qij =qii

1 + |i−j|(8)

This represents the case when the cohorts are identical in terms of their interaction with their

own members and their sensitivity to recruitment, but their interaction with other cohorts

declines with the distance between the cohorts.

3. Increasing. Spontaneous and own-imitation rates increase exponentially with the cohort num-

ber (pi=p1/(i+1), qii =q1/(i+1),∀i∈ N ), and the cross-imitation rates decline with the dis-

tance between the cohorts according to (8). This pattern of rates would occur when not only

the interaction with other cohorts is diﬀerent from the interaction within a cohort, but also

the interaction within a cohort becomes more pronounced, and the propensity to be recruited

19

increases. In the example with college students, students might get to know each other bet-

ter and build stronger bonds among themselves over time. Also, students can become less

risk-averse and more willing to engage in new activities, as they overcome the initial stress of

ﬁrst college years and feel more comfortable in school.

4. Decreasing. Spontaneous and own-imitation rates decrease exponentially with the cohort

number (pi=p(i+1), qii =q(i+1) ,∀i∈ N ), and the cross-imitation rates decline with the

distance between the cohorts according to (8). In the example with college students, this case

reﬂects the possibility that as students progress through their programs, they become more

focused on their major, which leads to less interaction with their peers. Students also become

less willing to engage in new activities such as participation in experiments, because a larger

chunk of their time is now dedicated to completing thesis papers and ﬁnding jobs.

A. Baseline

B. Constant

C. Increasing

D. Decreasing

0 2 4 6 8 10 0 2 4 6 8 10 0 2 4 6 8 10 0 2 4 6 8 10

0.00

0.25

0.50

0.75

1.00

Time

Cohort First Second Third

Figure 3: Simulated Relative Proportions for Cohorts

Figure 3shows the simulated time-series of relative proportions xi

tfor each cohort, as well as the

total relative proportion xt(thick dashed line), for the four cases described above. The ﬁgures do

not show the results for cohort i= 0 since it never participates in the subject pool by assumption.

Comparing the Constant (panel B) and the Baseline (panel A) cases, we see that the reduced

interaction between the cohorts leads to lower long-run participation in the subject pool. We can

20

also see that the long-run participation increases with the cohort number since older cohorts have

more chances to end up in a subject pool, in line with the theoretical prediction.

Panel C shows that the increasing interaction within cohorts can more than oﬀset the reduc-

tion caused by lower interaction between cohorts. The oldest cohort almost reaches its potential

maximum of participation in the long-run. In the Decreasing case (panel D), the participation is

even lower than in the Constant case, and thus decreasing rates is another important factor that

could drive low overall participation in a subject pool. Interestingly, the participation is no longer

greater for older cohorts at each time t: initially, younger cohorts are more represented in a subject

pool than older cohorts, but over time the older cohorts catch up and start to dominate the subject

pool.

A. Baseline

B. Constant

C. Increasing

D. Decreasing

0 2 4 6 8 10 0 2 4 6 8 10 0 2 4 6 8 10 0 2 4 6 8 10

0

1

2

3

Time

Bias 3rd and 2nd 3rd and 1st 2nd and 1st

Figure 4: Simulated Biases Between Cohorts

This point is worth developing further by looking at the graphs of pairwise biases over time on

Figure 4. In all the cases, except for the Decreasing one, we observe the growth of the bias over

time. Older cohorts are oversampled in a subject pool relative to younger cohorts in the long-run.

The bias increases in the distance between cohorts, e.g., the bias between cohorts 3 and 1 is higher

than the bias between cohorts 3 and 2. As we depart from the Baseline, the selection problem

becomes worse. It is particularly strong in the Increasing case. The Decreasing case is a notable

exception since the pattern of the spontaneous and imitation rates is oﬀsetting the oversampling of

21

older cohorts. Even though the asymptotic bias is still present in the Decreasing case, its magnitude

is much lower than in other cases.

5 Empirical Results

5.1 Data Description

The data we use come from an online recruitment system (the recruiter, henceforth) used at the

ICES at GMU. Our observation period is from the beginning of the Fall semester of 2014 until

the end of the Spring semester of 2017.14 Two types of recruitment events are conducted at the

ICES: email invitations sent to the entire student population at GMU and selected class visits

(typically large undergraduate classes). Both types of events encourage students to sign up for

participation in economic experiments. In order to sign up, a student has to create an account and

ﬁll out personal information, such as gender, ethnicity, age, starting year at college, and major.

Moreover, the recruiter records when an account was created, as well as the ﬁrst and the last dates

of participation in a session for each subject.

We use an account creation date as the date when a subject enters the subject pool. Upon en-

tering the pool, a subject’s account becomes active. Only subjects with active accounts are allowed

to participate in experiments. All accounts are deactivated at the beginning of each semester, and

a corresponding email notiﬁcation is sent to the entire subject pool. The email notiﬁes subjects

that they will not receive any invitations for experiments until they reactivate their accounts. In

order to reactivate an account, a subject only needs to login to the recruiter. Subjects who do not

reactivate their accounts become non-active and are assumed to have dropped out of the subject

pool. We use the last day of a semester in which a non-active subject last participated in a session

as the date when they exited the subject pool. By the end of Spring 2017 the ICES subject pool

consisted of around 1,000 active accounts with complete demographic information.

We concentrate our analysis on two subject characteristics: gender (time-invariant type) and

year in college, or cohort, (time-variant type). Gender is self-reported by a subject upon registering

14 The ICES recruiter has been launched in Fall 2014. Unlike the previously used ORSEE system, the ICES

recruiter allows us to collect the data necessary for our analysis. We exclude Fall 15/Spring 16 academic year from

the analysis, however, due to a temporary shift in the recruitment procedures used. We also include summer months

in the Spring semester, since there are some changes in the subject pool during summer, but these changes are not

enough to have a separate Summer semester.

22

in the recruiter. Year in school is imputed from the self-reported starting year at college (the year

in which a subject started their program at GMU). In our analysis we use only undergraduate

student population since most economic experiments are conducted with undergraduate students.

We use year in college instead of a “class year” (freshman, sophomore, junior, senior), because the

information about the class year is not regularly updated by the subjects.

While our model allows the types to be observable or unobservable, we opt for using observables.

The primary reason for our choice is data availability. The analysis of the dynamics of selection

requires high-frequency data on students’ types, which are simply not available for the unobservable

characteristics.15 And even if data on unobservables were available, it would have likely been much

noisier than the data on observables (Gillen et al.,2019), which in turn would have made uncovering

the dynamic trends harder. We do believe, however, that the dynamics of the observable types are

still informative of the dynamics of the unobservable types. If a decision to participate in a subject

pool reﬂects a subjects’ preferences, as the literature suggests (Abeler and Nosenzo,2015;Slonim

et al.,2013;Krawczyk,2011), then the trends in the observable types, which we uncover in our

data, should reﬂect some of these unobservable preferences.

We use the data from the Oﬃce of Institutional Research and Eﬀectiveness (OIRE) at GMU to

construct the characteristics of the reference population, which in our case is the student popula-

tion. Each semester OIRE publishes an overview of the entire student population, which includes

gender and ethnic composition, the total number of students enrolled, the number of undergraduate

students enrolled,16 as well as how many new students were admitted and what are the retention

rates for every class starting from 2007. Retention rate is a share of students who are still in college

after a given number of years. Students are considered to be in college only if they are still pursuing

a bachelor degree. For instance, for the cohort who started their program in 2007, 83.5% of the

students remain in college after one year, while only 5% of them remain in college after six years.

15 It is conceivable, and even advisable, that a recruitment system supplements basic demographic questions

with preference elicitation instruments. It will be impractical, of course, to deploy fully incentivized risk elicitation

tasks at the recruitment stage. However, survey questions, such as the ones proposed by Falk et al. (2018), can be

easily incorporated into a recruitment system and will not add too much burden on potential student participants

during registration. While arguably being crude, these instruments will shed at least some light on the unobservable

characteristics of entire subject pools.

16 The OIRE demographic composition data combines both undergraduate and graduate students. We use the

gender ratio for the total student population to impute the undergraduate gender composition.

23

Using these data in combination with the initial number of students in every cohort allows us to

impute the number of students who spend between 1 and 6+ years in college.17

While our model allows for any reference population, in our empirical analysis we opt for using

the student population at GMU as a reference. This choice is motivated by convenience and the

fact that the university’s registrar provides high-quality data on the student population. There is

little loss of generality from using the student population at GMU as a reference, as opposed to the

general population of the city of Fairfax, the state of Virginia, or the United States. Using these

alternative reference populations would change the scales of our time-series but not the dynamic

trends, the focus of this study. At the same time, the distributions of types in these alternative

reference populations are arguably measured with more noise than in the GMU registrar’s data.

Admitted Freshmen Total Undergraduate % Female

2014-2015 3,113 21,672 54%

Fall 2015 3,226 22,304 53%

Fall 2016 3,254 23,174 53%

Table 1: Description of the Population by Semester

Table 1provides a brief description of the student population at GMU by semester. Overall we

observe around 22,000 students every academic year of whom there are slightly more females than

males. About 3,000 new undergraduate students are admitted every Fall.

5.2 Results

5.2.1 Gender Types

We begin the analysis by looking at gender as a time-invariant type and present the dynamics and

summary statistics of the relative proportions of male and female participants in the subject pool

in Figure 5and in Table 2(see table notes for variables deﬁnitions). The shaded vertical bars in

Figure 5indicate the start and end of recruitment. The long-run participation in the subject pool

is very limited among both types, rarely exceeding 5%. Apart from the Fall 2014 semester, there

is a constant gap between the relative proportions of males and females, with males participating

at higher rates than females. The average potential proportion of males is slightly higher than the

17 We lump together the students who spend 6 and more years in college to increase the sample size in this small

group of students.

24

average potential proportion of females indicating the existence of a small but persistent long-run

bias.

Table 2: Summary Statistics

Spontaneous sign-up Imitation sign-up Drop-out Potential

proportion

Absolute Relative Absolute Relative Absolute Relative

Gender types

Female 0.014 1.018 0.006 0.190 0.009 0.213 0.043

Male 0.015 0.839 0.006 0.201 0.009 0.191 0.046

Cohort types

First-years 0.002 0.851 0.009 0.897 0.011 0.328 0.022

Second-years 0.037 1.800 0.010 0.150 0.034 0.359 0.075

Third-years 0.020 0.800 0.006 0.108 0.026 0.355 0.066

Fourth-years 0.013 0.419 0.004 0.083 0.024 0.356 0.067

Fifth-years 0.018 0.517 0.004 0.053 0.028 0.382 0.077

Sixth-years 0.009 0.378 0.001 0.023 0.014 0.392 0.065

Notes: Spontaneous sign-up is deﬁned as an average (across all semesters) 1-week increase in a relative proportion of a type

since the start of a recruitment event. Imitation sign-up is deﬁned as an average (across all semesters) change in a relative

proportion of a type between the end of semester and a 7-day point after the start of recruitment. Drop-out is deﬁned as

an average (across all semesters) decrease in a relative proportion of a type at the end of semester. Absolute values are

deﬁned as a diﬀerence between the ending and starting values of a given measure. Relative values are deﬁned as a diﬀerence

between the ending and starting values of a given measure divided by the starting value. Potential proportion is deﬁned as

an average (across all semesters) relative proportion at the end of semester.

There are distinctive dynamic patterns of the relative proportions in Figure 5that are consistent

with the patterns predicted by the model. Immediately after recruitment, relative proportions

of both types exhibit a rapid growth, which, however, decays within about a week. The relative

proportions continue to grow in the months after recruitment and slowly converge to an upper limit.

Table 2shows that the spontaneous sign-up rates for males and females are almost identical.18 If

we look at the relative sign-up, females increase their participation in the subject pool by more

than 100% immediately after recruitment, while males increase their participation only by 84%.

Females and males are equally active in the imitation sign-up and increase their participation in

the subsequent months by around 19% and 20%, respectively. Table 2also shows that females are

relatively more likely to drop out of the subject pool.

Figure 6shows the dynamics of the pairwise bias between males and females, bmale,female

t. As

noted earlier, there is virtually no bias throughout the Fall 2014 semester. After that, a small but

persistent bias emerges. The immediate response of the bias to recruitment tends to fall within

one of the three categories predicted by the model. The response is either monotonically declining

18 The spontaneous and imitation sign-up measures used in Table 2do not directly map into the structural

parameters of the model. These parameters are estimated in Section 6.

25

Fall16

Spring17

Fall14

Spring15

Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug

Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep

0.02

0.04

0.06

0.02

0.04

0.06

Type Female Male

Figure 5: Relative Proportions of Males and Females

(as in Fall 2016), hill-shaped (as in Spring 2017), or dip-shaped (as in Fall 2014). The mid-term

dynamics of the bias are rather complex and exhibit both high- and low-frequency oscillations.

Over time the bias tends to converge to a ﬁxed value. By the end of the observation period the

bias is just above 1.1.

Fall16

Spring17

Fall14

Spring15

Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug

Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep

1.0

1.1

1.2

1.0

1.1

1.2

Figure 6: Bias Between Males and Females

26

5.2.2 Cohort Types

On Figure 7, we present the results for the time-variant types. The potential proportions among

the cohorts range between 2.2% for the ﬁrst-years and around 7.7% for students with 6+ years in

school. The participation gaps among diﬀerent cohorts are large and persistent, though the ranking

of the cohorts by their participation varies across semesters. During the ﬁrst two semesters younger

cohorts tend to participate more in the subject pool than older cohorts. By the Fall 2016 semester,

the ranking of the cohorts is completely reversed: each older cohort dominates the cohort a year

earlier. This pattern continues, except for the abnormally high participation among second-years,

throughout the Spring 2017 semester. Such a pattern is in line with the predictions Proposition 6

of the Decreasing case of the time-variant version of the model.

Fall16

Spring17

Fall14

Spring15

Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug

Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep

0.00

0.03

0.06

0.09

0.00

0.03

0.06

0.09

Type First-years

Second-years

Third-years

Fourth-years

Fifth-years

Sixth-years

Figure 7: Relative Proportions of Cohort Types

The dynamic response of the relative proportions to recruitment in the case of cohort types is

very similar to the case of gender types. There are large diﬀerences between cohorts, however, in

terms of their immediate and long-run response to recruitment. The second-years beat all other

cohorts in terms of the spontaneous sign-up, both absolute and relative, and increase their par-

27

ticipation by 180%, on average, immediately after recruitment. The closest competitors are the

ﬁrst-years and the third-years with relative spontaneous sign-up rates of 85.1% and 80%, respec-

tively. The sixth-years have the lowest relative sign-up of 38%, while the ﬁrst-years have the lowest

absolute sign-up of 0.002. On the other hand, the ﬁrst-years have the strongest relative imitation

sign-up rate of 89.7%, while the second highest rate, belonging to the second-year students, is only

15.2%. The relative drop-out rates are very similar across the cohorts and range between 32.8%

for the ﬁrst-years and 39.2% for the sixth-years.

Figure 8shows the dynamics of the pairwise biases between subsequent cohorts.19 Consistent

with the Decreasing case, we observe the prevalence of younger cohorts at the beginning of the

period of observations, which is reversed as time goes on. Note that by the end of the 2016-2017

academic year the biases between cohorts are limited, as compared to the previous periods. This

is consistent with the predictions of the Decreasing case.

Fall16

Spring17

Fall14

Spring15

Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug

Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep

0

1

2

0

1

2

Bias Third to Second Fourth to Third Fifth to Fourth Sixth to Fifth

Figure 8: Biases Between Cohort Types

19 We look only at the biases between an older cohort and a cohort a year younger to make the picture readable.

We note, however, that the ordering of cohorts by their relative proportions and resulting pairwise biases during the

last periods are consistent with the predictions of the last part of Proposition 6. The bias between the second-years

and ﬁrst-years is omitted because of its huge size that trumps the sizes of all other biases.

28

Figure 8shows the possibility of decreasing, increasing, hill-shaped, and dip-shaped patterns

that are broadly consistent with the patterns generated by the model. Medium-run dynamics of

the biases often have ﬂuctuations, but by the end of the semester, the biases tend to converge to

ﬁxed values. The sizes of the long-run biases range between 0.6 and 2.5 with little stability between

semesters.

5.2.3 Summary

We summarize the patterns observed in the data in the following results.

Result 1 (Response of relative proportions to recruitment).The dynamic response of relative

proportions to recruitment is consistent with the model’s predictions and is characterized by three

features. First, there is rapid growth in relative proportions immediately after recruitment. Second,

this growth decays within about a week after recruitment. Third, relative proportions continue to

grow in the subsequent months until converging to upper limits.

Result 2 (Selection bias).The subject pool exhibits short-run and long-run selection biases between

types. There is a small long-run selection bias between males and females, while the long-run biases

between cohort types are much larger. For cohorts, by the end of the observation period each older

cohort dominates a cohort a year earlier, which is consistent with the Decreasing case of the time-

variant version of the model.

Result 3 (Response of selection bias to recruitment).The immediate dynamic response of biases

to recruitment is consistent with the model and tends to be either decreasing, increasing, dip-shaped,

or hill-shaped. The most common type of response is a decreasing response to recruitment. Over

time, biases converge to ﬁxed values.

6 Structural Estimation

6.1 Estimation Procedure

Consider a discrete-time version of equation (2) with ∆tequal to one day (the data are daily time-

series in our case). For each type i∈ N , the growth in relative proportion follows the following

29

diﬀerence equation.

xi

t+1 −xi

t= (αi−xi

t)(pie−δit+qi1x1

t+. . . +qiN xN

t).(9)

Our goal is to estimate the structural parameters of the model, i.e., the immediate spontaneous

rate pi, the decay rate δi, the potential proportion αi, as well as the imitation rates qi1, . . . , qiN .

We ﬁrst note that due to an almost perfect linear correlation between relative proportions

among gender types and cohort types, the imitation rates cannot be separately identiﬁed. We can

estimate, however, the aggregate imitation rate qifor each type. If xj

t=kijxi

t, then

qi1x1

t+. . . +qiN xN

t= (ki1qi1+. . . +kiN qiN )

| {z }

qi

xi

t.

We then assume a non-linear trend model of the form

xi

t=f(t, xi

0|pi, qi, δi, αi) + i

t,

where the non-linear trend f(·) is a function of the starting value xi

0and time tconditional on the

parameters of the model θi≡(pi, qi, δ i, αi), and i

tis mean-zero noise term. The non-linear trend

function f(·) is deﬁned implicitly as a solution to the diﬀerence equation (9). The identiﬁcation of

the parameters governing the two channels is ensured by our assumption on the functional form of

decay and the fact that the recruitment at GMU occurs over a short period of time at the beginning

of each semester.20 The potential proportion parameter is identiﬁed by the data on the relative

proportions at the end of a semester. We estimate the model by minimizing the sum of squared

deviations of the observed time-series xi

tfrom the predicted values ˆxi

t(θi) = f(t, xi

0|θi):

ˆ

θi= arg min

θi

T

X

t=0 xi

t−ˆxi

t(θi)2.

20 If recruitment procedures occurred throughout a semester that would complicate the separate identiﬁcation of

the two channels.

30

6.2 Estimation Results

6.2.1 Gender Types

Table 3shows the estimation results for gender types. The parameter estimates vary across

semesters for both males and females, potentially due to diﬀerences in recruitment procedures

and students’ response to them. For females, the estimates of prange from 0.17 to 0.462, with p=

0.279 for the average semester. The estimate for the average semester implies that in a case when

there is no indirect channel, and the starting relative proportion is zero, the relative proportion

would grow to the 28% of the potential proportion in the ﬁrst day after recruitment. Males, on

average, tend to be more responsive to recruitment than females. However, the results are mixed

if one looks at the estimates for individual semesters. For males, the estimates of prange from

0.089 to 0.456, with p= 0.303 for the average semester. The estimate for the average semester

implies that in a case when there is no indirect channel and the starting relative proportion is zero,

the relative proportion would grow to the 30% of the potential proportion in the ﬁrst day after

recruitment.

Table 3: Estimation Results for Gender Types

Parameter Fall 14 Spring 15 Fall 16 Spring 17 Average

Panel A. Female

p0.389 (0.009) 0.462 (0.005) 0.170 (0.040) 0.375 (0.034) 0.279 (0.009)

q1.061 (0.036) 0.358 (0.009) 0.004 (0.001) 0.487 (0.026) 0.629 (0.023)

δ1.449 (0.149) 0.429 (0.010) 0.727 (0.031) 0.876 (0.332) 0.520 (0.035)

α0.039 (0.0001) 0.056 (0.00003) 0.046 (0.007) 0.042 (0.0001) 0.046 (0.0001)

Panel B. Male

p0.316 (0.022) 0.456 (0.037) 0.089 (0.062) 0.407 (0.060) 0.303 (0.005)

q1.240 (0.031) 0.202 (0.043) 0.010 (0.184) 0.327 (0.111) 0.629 (0.014)

δ1.239 (0.312) 0.443 (0.019) 0.520 (0.085) 0.518 (0.045) 0.522 (0.014)

α0.039 (0.0001) 0.061 (0.003) 0.063 (0.020) 0.047 (0.003) 0.048 (0.00005)

Observations 136 218 78 188 135

Notes: The table reports the estimates of the structural parameters of the model for each semester,

as well as for the average semester, broken down by gender. The standard errors are based on

500 bootstrap replication.

The estimates of the imitation rate qshow that the indirect channel is important for recruit-

ment. For females, the estimates of qrange from 0.004 to 1.061, with q= 0.629 for the average

31

semester. The strength of the indirect channel, on average, is virtually identical for males and

females despite some variation across individual semesters. The estimates of qfor males range from

0.01 to 1.24, with q= 0.629 for the average semester. Despite the high chances of being recruited

through interaction with peers, the strength of the indirect channel is limited due to low overall

participation.21

Another characteristic of recruitment eﬀectiveness is the decay rate δ. For females, the estimates

of δrange from 0.429 to 1.449, with δ= 0.52 for the average semester. This estimate implies that

the spontaneous rate drops by half, on average, in 1.332 days. After a week, the spontaneous rate

drops by 97%. For males, the decay rate, on average, is not signiﬁcantly diﬀerent from the decay

rate for females. This result becomes mixed, however, if one looks at the individual semesters. The

estimates of δfor males range from 0.443 to 1.239, with δ= 0.522 for the average semester. This

estimate implies that the spontaneous rate drops by half, on average, in 1.328 days. After a week,

the spontaneous rate drops by 97%.

The low overall participation in the subject pool is evidenced by low estimates of the potential

proportions αfor both types. For females, the estimates of αrange from 0.039 to 0.056, with

α= 0.046 for the average semester, meaning that, on average, only 4.6% of the female student

population is in the subject pool in our sample. For males, the estimates of αrange from 0.039 to

0.063, with α= 0.048 for the average semester, meaning that, on average, 4.8% of the male student

population is in the subject pool, which is slightly higher than the corresponding value for females.

This diﬀerence implies a long-run pairwise bias between males and females of 1.054.

Overall, the proposed model does a remarkably good job at quantitatively matching the patterns

observed in the data. Figure 9illustrates this point by plotting the actual relative proportions for

the average semester against the values predicted by the model (left and middle panels), as well as

the actual pairwise bias between males and females for the average semester against the predicted

bias (right panel). The dashed horizontal lines on the graphs for relative proportions correspond

to the estimated values of α. The time is measured in weeks starting from the date of recruitment.

The predicted values track the actual data very closely. In particular, the estimated model is

capable of producing the three main features of the dynamics of relative proportions: a rapid growth

21In other words, conditional on interacting with a peer who is already in a subject pool, the chances of being

recruited are high, but since few peers are in a subject pool, the overall eﬀect is small.

32

immediately after recruitment, a quick decay in this growth within a week after recruitment, and a

slow growth in subsequent months with a convergence to an upper limit by the end of a semester.

The predicted bias makes the trend in the data sharper and can be described by a hill-shaped

pattern. Immediately after recruitment, the bias spikes due to a higher spontaneous rate for males

than for females. The bias then slowly converges to its long-run value. The graph of the bias also

highlights the importance of the dynamic eﬀects in evaluating the selection problem. Evaluating

the selection problem at the beginning of the recruitment cycle would yield diﬀerent results as

compared to the end of the cycle.

Female

Male

Bias

0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18

1.03

1.04

1.05

1.06

1.07

0.030

0.035

0.040

0.045

0.025

0.030

0.035

0.040

0.045

Week

Actual Fitted Counterfactual

Figure 9: Time-series of Relative Proportions and Male-Female Bias for the Average Semester

To further illustrate the importance of the indirect channel, we conduct a counterfactual simu-

lation in which this channel is completely shut down. Figure 9presents the results for the simulated

relative proportions of females and males (left and center panel) and the implied male-female bias

(right panel). As is evident from the picture, shutting down the indirect channel would have a

dramatic eﬀect on the relative proportions and bias. The relative proportions of both types would

never reach their potential proportions by the end of the semester. The relative proportion would

be virtually stuck at 0.036 for females and 0.039 for males. These numbers represent 53% and

64% of the total gain in participation (the diﬀerence between a potential proportion and a starting

value) for females and males, respectively. The indirect channel thus accounts for roughly one-half

33

to one-third of the total participation gain. The bias would be stuck at 1.064, which is 1% larger

than the ratio of the potential proportions.22

6.2.2 Cohort Types

Table 4presents the estimation results for cohort types. We restrict our attention to the average

semester and the ﬁrst four cohorts. Individual semesters, for the most part, do not provide enough

variation in the data to allow for a meaningful estimation, which is also the case for the cohorts of

ﬁfth- and sixth-years for the average semester.

Table 4: Estimation Results for Cohort Types

Parameter First-years Second-years Third-years Fourth-years

p−0.009 (0.015) 0.338 (0.012) 0.342 (0.007) 0.317 (0.021)

q1.865 (0.582) 0.284 (0.024) 0.427 (0.017) 0.166 (0.030)

δ0.048 (4.895) 0.382 (0.016) 0.449 (0.016) 0.403 (0.019)

α0.026 (0.0001) 0.086 (0.002) 0.068 (0.0002) 0.063 (0.001)

Observations 136 135 135 135

Notes: Reports the estimates of the structural parameters for each cohort for the

average semester. The standard errors are based on 500 bootstrap replications.

The table reveals dramatic diﬀerences among cohorts regarding their responsiveness to direct

and indirect recruitment, as well as potential proportions. The eﬀectiveness of the direct channel is

lowest for the cohort of ﬁrst-years. The estimate of the spontaneous rate for this cohort is virtually

zero,23 suggesting that the ﬁrst-years do not respond, or respond weakly, to direct recruitment

events. The estimates of the spontaneous rates for the three older cohorts are signiﬁcantly diﬀerent

from zero. The third-years have the highest point estimate of p= 0.342. This number implies that

if there was no indirect channel and a starting relative proportion was zero, the relative proportion

would grow to the 34% of the potential proportion in the ﬁrst day after recruitment. The diﬀerences

in spontaneous rates between the second-, third-, and fourth-years are small and not statistically

signiﬁcant.

The eﬀectiveness of the indirect channel is highest for the ﬁrst-years with an estimate of q=

1.865. This estimate is in stark contrast to the virtually non-existent direct channel for the ﬁrst-

22 The absence of the indirect channel does not necessarily imply a larger bias. The diﬀerences in the spontaneous

and decay rates will determine the size of the bias.

23While the point estimate is, in fact, negative, which is not allowed by the model, we interpret it as being virtually

zero based on its magnitude and standard error.

34

years, suggesting that the indirect channel plays a dominant role in the participation decisions of

this cohort. There is, however, a substantial amount of uncertainty around the estimate of qfor the

ﬁrst-years. Compared to the ﬁrst-years, the estimates of the imitation rates among the remaining

three cohorts are much smaller, with the fourth-years having the lowest estimate of q= 0.166.

Overall, the eﬀectiveness of the indirect channel tends to decline with a cohort’s age.

There is substantial uncertainty in the estimate of δfor the ﬁrst-years, which is not surprising

given the low estimate of the spontaneous rate. The estimates of the decay rate for the remaining

three cohorts range between 0.382 for the second-years and 0.449 for the third-years. These numbers

imply that, for instance, the eﬀectiveness of the direct channel for the third-years would drop by

half in 1.544 days and that in a week following recruitment the spontaneous rate would drop by

96%, which is similar to what we found using gender types. Overall, the estimates of the decay

rates tend to increase with a cohort’s age.

The estimates of the potential proportions show low overall participation across the cohorts.

The second-years have the highest potential proportion of 0.086, while the ﬁrst-years have the lowest

potential proportion of 0.026. These numbers imply that, on average, only 8.6% of the second-years

and 2.6% of the ﬁrst-years participate in the subject pool. The third- and fourth-years lie in between

the other two cohorts in terms of their potential proportions.24

Just like in the case of gender types, the estimated model for cohorts does a remarkably good

job at matching the data. This is evident from Figure 10 in which each panel shows the time-series

of the actual relative proportion for a given cohort along with the model’s ﬁt. The horizontal

dashed lines, as before, indicate the estimates of α, and the time is measured in weeks starting

from the date of recruitment. The three key features of the data—a rapid growth immediately after

recruitment, a quick decay in this growth within a week after recruitment, and a slow growth in

subsequent months with convergence to an upper limit by the end of a semester—are well-captured

by the model.

Figure 11 shows the time-series of pairwise biases between subsequent cohorts: the bias between

the second- and ﬁrst-years, the bias between the third- and second-years, and the bias between the

fourth- and third-years. The ﬁtted values of the bias track the actual values quite closely. The bias

24Note that since we are dealing with the average semester in cohort estimates, it is not possible to track how the

patterns of potential proportions across cohorts change by semester.

35

First-years

Second-years

Third-years

Fourth-years

0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18

0.025

0.050

0.075

Week

Actual Fitted Counterfactual

Figure 10: Time-series of the Relative Proportions of Cohorts for the Average Semester

between the second- and ﬁrst-years has a hump-shaped pattern. This bias grows initially, reaches a

peak, and then slowly converges to the long-run bias of 3.302 deﬁned by the ratio of the potential

proportions. The bias between the third- and second-years exhibits a decreasing pattern, however,

it is worth noting that since the bias value goes well below 1, it implies a stronger bias between

the two cohorts. The bias between these two cohorts converges over time to the long-run value of

0.781. The bias between the fourth- and third-years exhibits a similar decreasing pattern but is

more complex. After the decline in the ﬁrst half of the semester, the bias starts to increase slightly

in the second half of the semester towards the long-run value of 0.931.

Second to First

Third to Second

Fourth to Third

0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18

0.90

0.93

0.96

0.99

0.8

0.9

1.0

3

4

5

6

7

Week

Actual Fitted Counterfactual

Figure 11: Time-series of Biases between Cohorts for the Average Semester

36

To highlight the importance of the indirect channel of recruitment, we conduct a counterfactual

analysis in which we completely shut down the indirect channel. The resulting time-series of

counterfactual relative proportions are shown in Figure 10. In the absence of the indirect channel,

the relative proportions would virtually stop growing after two weeks and would never reach the

potential proportions. The long-run values of the counterfactual relative proportions would only

attain around two-thirds of the total gain in participation (the diﬀerence between a potential

proportion and a starting value) for the three older cohorts. The relative proportion of the ﬁrst-

years would remain virtually constant in the absence of the indirect channel. We can conclude that

the indirect channel accounts for roughly one-third of the total gain in participation.

Figure 11 shows the time-series of counterfactual biases between cohorts in the absence of the

indirect channel. The bias between the second- and ﬁrst-years would be increasing and exceed the

ratio of the potential proportions by a large degree. The bias between the third- and second years

would be slightly lower (a weaker bias) than the ratio of potential proportions. The same would

also hold for the bias between the fourth- and third-years.

7 Conclusion

We study the dynamics of the selection problem in economic experiments. Our analysis shows

that while a static selection problem leads to biases in the estimates of types shares and treatment

eﬀects, a time-varying selection problem further complicates potential treatment eﬀect biases. The

introduction of dynamics also helps to explain some of the existing contradictions in the literature

(Slonim et al.,2013;Cleave et al.,2013). In order to understand the dynamic nature of the selection

problem, we develop a model of participation in a subject pool. The model assumes that agents’

participation evolves over time and is driven by the two channels: the direct channel of recruitment

and the indirect channel of agents’ interaction. Diﬀerences in potential proportions drive the long-

run selection biases, while diﬀerences in initial participation levels and participation rates drive

the short-run biases between agent types. The modiﬁcation of the model in which types are time-

variant results in the possibility of long-run biases even when potential proportions are identical

across types.

37

In our empirical analysis of the recruitment data from ICES at GMU we ﬁnd that the participa-

tion dynamics are consistent with the model’s predictions. We ﬁnd evidence of short- and long-run

selection biases between males and females, as well as between cohorts. The counterfactual analysis

of the data using an estimated model shows that the indirect channel accounts for roughly one-third

of the total participation gain for both gender and cohort types. We use the model to show that

the selection bias would be higher or lower, depending on the type, in the absence of the indirect

channel.

Our ﬁndings imply that networks eﬀects play a crucial role in shaping the dynamics of the

selection problem. The presence of the dynamic eﬀects in the selection problem, in turn, leads

to several important policy implications for methodology, design of experiments, and recruitment.

The methodological implication is that future studies of the selection problem should address its

dynamic nature. For example, the measurements of selection biases could be done at various point

in time throughout the semester. This would help to avoid the seemingly contradictory results

about the presence of the selection problem in a subject pool. The implication for experimental

design is that studies using a between-subject design would beneﬁt from randomizing treatments

within a session. This practice would minimize the potential treatment eﬀect biases caused by

the variation in selection biases over time. Alternatively, if randomization to treatment within a

session is infeasible, sessions should be kept temporally close. The implication for recruitment is

that it would help to leave a short burn-in period after a main recruitment event. Since selection

biases tend to stabilize in the long-run, leaving a burn-in period would minimize the selection biases

caused by short-run ﬂuctuations.

We propose that future research on the selection problem should focus more on its dynamics. A

major obstacle to studying the dynamics of the selection problem is the absence of high-frequency

subject pool data on unobservable subject characteristics, such as preferences and personality. The

lack of such data forced our analysis to focus on observable characteristics that are recorded at a time

of registration for a subject pool. While the dynamics of observable characteristics may provide

some insights into the dynamics of unobservable characteristics, a more direct measurement of

unobservables is desirable. Supplementing basic demographic data with preference and personality

data elicited at a time of registration would be highly beneﬁcial.

38

References

Abeler J, Nosenzo D (2015). “Self-Selection Into Laboratory Experiments: Pro-Social Motives

Versus Monetary Incentives.” Experimental Economics,18(2), 195–214.

Averell L, Heathcote A (2011). “The Form of the Forgetting Curve and the Fate of Memories.”

Journal of Mathematical Psychology,55(1), 25 – 35. Special Issue on Hierarchical Bayesian

Models.

Bass FM (1969). “A New Product Growth for Model Consumer Durables.” Management science,

15(5), 215–227.

Biernacki P, Waldorf D (1981). “Snowball Sampling: Problems and Techniques of Chain Referral

Sampling.” Sociological Methods & Research,10(2), 141–163.

Cappelen AW, Nygaard K, Sørensen EØ, Tungodden B (2015). “Social Preferences in the Lab:

A Comparison of Students and a Representative Population.” The Scandinavian Journal of

Economics,117(4), 1306–1326.

Castillo M, Freer M (2018). “Revealed diﬀerences.” Journal of Economic Behavior & Organization,

145, 202–217.

Charness G, Gneezy U, Kuhn MA (2012). “Experimental methods: Between-subject and within-

subject design.” Journal of Economic Behavior & Organization,81(1), 1–8.

Cleave BL, Nikiforakis N, Slonim R (2013). “Is There Selection Bias in Laboratory Experiments?

The Case of Social and Risk Preferences.” Experimental Economics,16(3), 372–382.

Dohmen T, Falk A, Golsteyn BHH, Huﬀman D, Sunde U (2017). “Risk Attitudes Across The Life

Course.” The Economic Journal,127(605), F95–F116.

Ebbinghaus H (1885/1974). Memory: A Contribution to Experimental Psychology. New York:

Dover.

Exadaktylos F, Esp´ın AM, Bra˜nas-Garza P (2013). “Experimental Subjects Are Not Diﬀerent.”

Scientiﬁc Reports,3, 1213 EP.

Falk A, Becker A, Dohmen T, Enke B, Huﬀman D, Sunde U (2018). “Global Evidence on Economic

Preferences.” The Quarterly Journal of Economics,133(4), 1645–1692.

Falk A, Heckman JJ (2009). “Lab Experiments Are a Major Source of Knowledge in the Social

Sciences.” Science,326(5952), 535–538.

Falk A, Meier S, Zehnder C (2013). “Do Lab Experiments Misrepresent Social Preferences? The

Case Of Self-Selected Student Samples.” Journal of the European Economic Association,11(4),

839–852.

G¨achter S (2010). “(Dis) Advantages of Student Subjects: What Is Your Research Question?”

Behavioral and Brain Sciences,33(2-3), 92–93.

Gillen B, Snowberg E, Yariv L (2019). “Experimenting with Measurement Error: Techniques with

Applications to the Caltech Cohort Study.” Journal of Political Economy,127(4), 1826–1863.

39

Harrison GW, Lau MI, Rutstr¨om EE (2009). “Risk Attitudes, Randomization to Treatment, and

Self-Selection into Experiments.” Journal of Economic Behavior & Organization,70(3), 498 –

507.

Henrich J, Heine SJ, Norenzayan A (2010). “The Weirdest People in the World?” Behavioral and

Brain Sciences,33(2-3), 61–83.

Jackson MO, L´opez-Pintado D (2013). “Diﬀusion and Contagion in Networks with Heterogeneous

Agents and Homophily.” Network Science,1(01), 49–67.

Krawczyk M (2011). “What Brings Your Subjects to the Lab? A Field Experiment.” Experimental

Economics,14(4), 482–489.

Levitt SD, List JA (2007). “What Do Laboratory Experiments Measuring Social Preferences Reveal

About The Real World?” Journal Of Economic Perspectives,21(2), 153–174.

l’Haridon O, Vieider FM, Aycinena D, Bandur A, Belianin A, Cingl L, Kothiyal A, Martinsson P

(2018). “Oﬀ the Charts: Massive Unexplained Heterogeneity in a Global Study of Ambiguity

Attitudes.” The Review of Economics and Statistics,100(4), 664–677.

List JA (2009). “Social Preferences: Some Thoughts From The Field.” Annual Review of Eco-

nomics,1(1), 563–579.

Murre JMJ, Dros J (2015). “Replication and Analysis of Ebbinghaus’ Forgetting Curve.” PLOS

ONE,10(7), 1–23.

Slonim R, Wang C, Garbarino E, Merrett D (2013). “Opting-In: Participation Bias in Economic

Experiments.” Journal of Economic Behavior & Organization,90, 43–70.

Smith VL (1989). “Theory, Experiment and Economics.” Journal of Economic Perspectives,3(1),

151–69.

40

Appendices

A Experimental Instructions

In this paper we are using the entire recruitment data from an experimental laboratory at the

Interdisciplinary Center for Economic Science (ICES) at George Mason University (GMU) rather

than data from a particular experiment. Therefore, instead of describing experimental instructions

for a particular experiment, we describe the recruitment procedures of the experimental laboratory.

The recruitment system at the ICES experimental laboratory in its current form (the recruiter,

henceforth) was launched in Fall 2014. The recruiter is used to manage the subject pool and, in

particular, to invite subjects to participate in experimental sessions. Two types of recruitment

events are conducted on campus to populate the subject pool. First, at the beginning of each

semester the recruiter sends a generic email to the entire student population at GMU. The email

encourages students to sign up for participation in economic experiments by registering in the

recruiter. A typical email is presented in Figure A.1. The email emphasizes the time commitment

that a typical experiment requires and that students can expect monetary compensation for their

time. A second type of recruitment events, which however did not occur within our observation

period, is class visits. This type of recruitment involves representatives of the ICES who come to

large undergraduate classes, deliver a short talk designed to encourage participation in economic

experiments, and distribute ﬂyers with information on how to sign up. The content of the talk,

for the most part, mirrors the content of the recruitment email. Table A.1 shows the timeline of

recruitment events at the ICES during our observation period.

Upon following the link to the recruiter, a potential subject ﬁlls out demographic information,

such as gender, ethnicity, birth year, and major. He or she also gives consent to receiving invitations

from the recruiter to participate in experimental sessions. After a subject’s account is created, the

recruiter records information on when an account was created, when an account was last updated,

how many sessions a subject participated in, and when were the ﬁrst and last times a subject

participated in a session.

The recruiter also keeps track of whether an account is active or not. At the beginning of

each semester, the recruiter de-activates all accounts and sends an email to all registered subjects

41

From:no-reply@ices-experiments.com

Subject: ICES Invitation

Date: November 7, 2018 at 3:31 PM

To:alekseev@chapman.edu

All Mason students have the opportunity to advance Mason’s research and

earn cash at the Interdisciplinary Center for Economic Science. You will

receive invitations for experiments after you sign up. All participants are

monetarily compensated for their time.

Join Us

The Interdisciplinary Center for Economic

Science is an international leader in economics

research.

Have questions? Ask Lab

Manager Matt McMahon. Please

email

mmcmaho8@masonlive.gmu.edu

or call 703-993-8583

Participating in experiments has

no effect on course grades. There

is no limit to the number of

experiments you may participate

in. Participants must be able to

respond to visual cues on a

computer. Sign-in to payment is

30-90 minutes.

...

example@email.com

555-555-5555

Figure A.1: An Example of a Recruitment Email

42

Table A.1: Timeline of Recruitment Events

Event ID Type Semester Start Date End Date

1 email Fall14 2014-09-05 2014-09-06

2 email Spring15 2015-01-23 2015-01-24

3 email Fall15 2015-09-13 2015-09-14

4 email Fall16 2016-10-27 2016-11-04

5 email Spring17 2017-01-26 2017-01-27

6 class Fall15 2015-09-28 2015-09-30

7 class Fall15 2015-10-01 2015-10-02

8 class Fall15 2015-10-06 2015-10-07

9 class Fall15 2015-12-08 2015-12-13

10 class Spring16 2016-02-01 2016-02-06

11 class Spring16 2016-02-08 2016-02-13

12 class Spring16 2016-04-11 2016-04-16

13 class Spring16 2016-04-25 2016-04-30

asking them to re-activate their account thus conﬁrming their willingness to further participate in

economic experiments. Subjects who successfully re-activate their accounts become active, while

subjects who do not re-activate their accounts remain non-active. Those latter subjects eﬀectively

drop out of the subject pool.

B Proofs

Proposition 1

Proof. If a subject pool does not have a selection problem, then there is a constant ¯α, such that

xt= (¯α, ¯α, . . . , ¯α). Clearly, any pairwise bias will be equal to 1 in this case. Similarly, if all the

pairwise biases are equal to 1, they can be written as bij

t= ¯α/¯α, which implies that xt= ¯αı.

Proposition 2

Proof. Consider the ratio of the shares

mI

˜mI=mI+mIIbII,I= 1 + mII bII,I−1.

If the ratio of the shares is one, then mII bII,I−1= 0, which implies that bII,I= 1 (apart from the

trivial case when there is just one type in the population). Similarly, if bII,I= 1 then mI/˜mI= 1.

43

Proposition 3

Proof. Consider the diﬀerence between an estimated treatment eﬀect and a treatment eﬀect in a

reference population:

∆Y−∆˜

Y=∆YI−∆YIImI−˜mI.

If the two treatment eﬀects are identical, ∆Y−∆˜

Y= 0, then either ∆YI= ∆YII (homogeneous

treatment eﬀects) must be true or mI= ˜mI(no selection problem) must be true, or both. Similarly,

if either condition is true (or both), then ∆Y= ∆ ˜

Y.

Proposition 6

Proof. Consider the asymptotic pairwise bias between two subsequent cohorts:

lim

t→∞

xi+1

t

xi

t

=1−(1 −p−qx)i+1

1−(1 −p−qx)i.

Since p+qx < 1, the fraction is greater than one. Diﬀerentiating the fraction w.r.t iyields

d

di 1−(1 −p−qx)i+1

1−(1 −p−qx)i!=ln(1 −p−qx)(1 −p−qx)i(p+qx)

(1 −(1 −p−qx)i)2.

This expression is negative since p+qx < 1. Moreover, consider cohorts iand i+j. The pairwise

bias is increasing in j, since

1−(1 −p−qx)i+j

1−(1 −p−qx)i>1−(1 −p−qx)i+j0

1−(1 −p−qx)iif j > j0.

44