Comparative Effectiveness of Matching Methods for Causal Inference

Abstract

Matching methods for causal inference selectively prune observations from the data in order to reduce model dependence. They are successful when simultaneously maximizing balance (between the treated and control groups on the pre-treatment covariates) and the number of observations remaining in the data set. However, ex-isting matching methods either fix the matched sample size ex ante and attempt to reduce imbalance as a result of the procedure (e.g., propensity score and Mahalanobis distance matching) or fix imbalance ex ante and attempt to lose as few observations as possible ex post (e.g., coarsened exact matching and calpier-based approaches). As an alternative, we offer a simple graphical approach that addresses both criteria simultaneously and lets the user choose a matching solution from the imbalance-sample size frontier. In the process of applying our approach, we also discover that propensity score matching (PSM) often approximates random matching, both in real applications and in data simulated by the processes that fit PSM theory. Moreover, contrary to conventional wisdom, random matching is not benign: it (and thus often PSM) can degrade inferences relative to not matching at all. Other methods we study do not have these or other problems we describe. However, with our easy-to-use graphical approach, users can focus on choosing a matching solution for a particular application rather than whatever method happened to be used to generate it.

Full text

Page 1

Comparative Effectiveness of Matching Methods
for Causal Inference∗
Gary King†
Richard Nielsen‡
Carter Coberley§
James E. Pope¶
Aaron Wells?
December 9, 2011
Abstract
Matching methods for causal inference selectively prune observations from the data
in order to reduce model dependence. They are successful when simultaneously
maximizing balance (between the treated and control groups on the pre-treatment
covariates) and the number of observations remaining in the data set. However, ex-
isting matching methods either fix the matched sample size ex ante and attempt to
reduceimbalanceasaresultoftheprocedure(e.g., propensityscoreandMahalanobis
distance matching) or fix imbalance ex ante and attempt to lose as few observations
as possible ex post (e.g., coarsened exact matching and calpier-based approaches).
As an alternative, we offer a simple graphical approach that addresses both criteria
simultaneously and lets the user choose a matching solution from the imbalance-
sample size frontier. In the process of applying our approach, we also discover that
propensity score matching (PSM) often approximates random matching, both in real
applications and in data simulated by the processes that fit PSM theory. Moreover,
contrary to conventional wisdom, random matching is not benign: it (and thus often
PSM) can degrade inferences relative to not matching at all. Other methods we study
do not have these or other problems we describe. However, with our easy-to-use
graphical approach, users can focus on choosing a matching solution for a particular
application rather than whatever method happened to be used to generate it.
∗Our thanks to Stefano Iacus and Giuseppe Porro for always helpful insights, suggestions, and collab-
oration on related research; this paper could not have been written without them. Thanks also to Alberto
Abadie, Seth Hill, Kosuke Imai, John Londregan, Adam Meirowitz, Brandon Stewart, Liz Stuart and par-
ticipants in the Applied Statistics workshop at Harvard for helpful comments; Healthways and the Institute
for Quantitative Social Science at Harvard for research support; and the National Science Foundation for a
Graduate Research Fellowship to Richard Nielsen.
†Albert J. Weatherhead III University Professor, Institute for Quantitative Social Science, 1737 Cam-
bridge Street, Harvard University, Cambridge MA 02138; http://GKing.harvard.edu, king@harvard.edu,
(617) 500-7570.
‡Ph.D. Candidate, Institute for Quantitative Social Science, 1737 Cambridge Street, Harvard University,
Cambridge MA 02138; http://people.fas.harvard.edu/∼rnielsen, rnielsen@fas.harvard.edu, (857) 998-8039.
§Director of Health Research and Outcomes, Healthways, Inc. 701 Cool Springs Blvd, Franklin TN
37067
¶Chief Science Officer, Healthways, Inc., 701 Cool Springs Blvd, Franklin TN 37067
?Principal Investigator of Health Outcomes Research, Healthways, Inc., 701 Cool Springs Blvd,
Franklin TN 37067

Page 2

1 Introduction
As is widely recognized, matching methods for causal inference are best applied with
an extensive, iterative, and typically manual search across different matching solutions,
simultaneously seeking to maximize covariate balance between the treated and control
groups and the matched sample size. A measureof the difficulty ofthis process may bethe
fact that most applied publications report running only a single matching solution. This
is especially problematic because no existing matching method simultaneously optimizes
both goals: For example, Mahalanobis Distance Matching (MDM) and Propensity Score
Matching(PSM)chooseafixednumberofobservationsexante(typicallyamultipleofthe
number of treated units) and hope for imbalance reduction as a result of the procdedure. In
contrast, Coarsened Exact Matching (CEM) and caliper-based approaches choose a fixed
level of imbalance ex ante and hope that the number of observations left as a result of the
procedure is sufficiently large.
We attempt to close some of this gap between methodological best practices and the
practical use of these methods. To do this, we describe a simple graphical tool to identify
the frontier (and evaluate the trade off) of matching solutions that define maximal levels
of balance for given matched sample sizes; users can then easily choose a solution that
best suits their needs in real applications. We also use this approach to elucidate differ-
ences among individual matching methods. We study PSM and MDM (with and without
calipers) and CEM. PSM and CEM each represent the most common member of one of
the two known classes of matching methods (Rubin, 1976; Iacus, King and Porro, 2011).
Other matching methods can easily be incorporated in our framework too.
In the process of developing this graphical tool for applied researchers, we are also led
to several new methodological findings. For example, we discover a serious problem with
PSM (and especially PSM with calipers) that causes it to approximate random matching in
common situations. Moreover, and contrary to conventional wisdom, random matching is
not benign; on average, it increases imbalance (and variance) compared to not matching.
Asweshow, suchresultscallintoquestiontheautomaticuseofPSMformatchingwithout
comparison with other methods; the common practice of calipering PSM matches worse
1

Page 3

than 1/4 of standard deviation; using PSM to adjust experimental results; including all
available covariates in PSM models; and some other commonly recommended practices.
To understand this problem, we analyze data simulated to fit propensity score theoretical
requirements, and we find the problem again. With these simulations, and a simple 12-
observation example we develop, we illuminate precisely why the problem occurs and
what to do about it. Even with these findings, PSM can be a useful procedure, but it needs
to be used comparatively; our graphical procedure can help.
The comparative effectiveness approach to matching methodology discussed here of-
fers an easy way to implement best practices, enabling applied researchers to choose
matching solutions that improve inferences in their data rather than having to decide on a
theoretical basis which matching methods may or may not work best in general.
2 Matching Methods
Here we summarize three matching methods, as they are commonly used in practice. We
begin with notation, the quantities of interest and estimation, and then discuss how to
choose among matching methods.
Notation
For unit i (i = 1,...,n), let Tidenote a treatment variable coded 1 for units
from the treated group and 0 for units from the control group. For example, Ti = 1
may indicate that person i is given a particular medicine and Ti= 0 means that person
i is given a different medicine (or a placebo). Let Yi(t) (for t = 0,1) be the value the
outcome variable would take if Ti= t, a so-called “potential outcome”. By definition,
for each i, either Yi(1) or Yi(0) is observed, but never both. This means we observe
Yi = TiYi(1) + (1 − Ti)Yi(0). Finally, we denote a vector of available pre-treatment
control variables Xi.
Quantities of Interest and Estimation
Denote the treatment effect of T on Y for unit i
asTEi= Yi(1)−Yi(0). Nowconsideranobservationfromthetreatedgroup, soYi(1) = Yi
is observed, and so Yi(0) is unobserved. The simplest version of matching is to estimate
TEiby replacing the unobserved Yi(0) with an observed unit (or units) j from the control
group such that it is matched to observation i either as Xi= Xj(which is known as “exact
matching”) or, as is usually necessary, Xi≈ Xj. (Matching methods differ primarily by
2

Page 4

how they define approximate matching.) Unmatched observations are pruned from the
data set before further analyses.
The immediate goal then is to maximize both balance, the similarity between the mul-
tivariate distributions of the treated and control units, and the size of the matched data set.
Any remaining imbalance must be dealt with by statistical modeling assumptions. The
primary advantage of matching is that it greatly reduces the dependence of our conclu-
sions on these assumptions (Ho et al., 2007).
Inmoststudies, wedonotseektoestimateTEiforeachi; weinsteadestimateaverages
of it over relevant subsets of units. One such quantity of interest is the sample average
treatment effect on the treated, SATT =
1
nT
?
i∈{T=1}TEi, which is the treatment effect
averaged over all (nT) treated units. However, since most real data sets contain some
observationswithoutgoodmatches, perhapsbecausecommonsupportbetweenthetreated
and control groups do not fully coincide, best practice has been to compute a causal effect
among only those treated observations for which good matches exist. We designate this
as the feasible sample average treatment effect on the treated or FSATT.
This decision is somewhat unusual in that FSATT is defined as part of the estimation
process, but in fact it follows the usual practice in observational data analysis of collecting
data and making inferences only where it is possible to learn something. The difference is
that the methodology makes a contribution to ascertaining which quantities of interest can
be estimated (e.g., Crump et al., 2009; Iacus, King and Porro, 2011); one must only be
careful to characterize the units being used to define the new estimand. As Rubin (2010,
p.1993) puts it, “In many cases, this search for balance will reveal that there are members
of each treatment arm who are so unlike any member of the other treatment arm that they
cannot serve as points of comparison for the two treatments. This is often the rule rather
than the exception, and then such units must be discarded.... Discarding such units is the
correct choice: A general answer whose estimated precision is high, but whose validity
rests on unwarranted and unstated assumptions, is worse than a less precise but plausible
answer to a more restricted question.” In the analyses below, we use FSATT, and so let
the quantity of interest change, but everything described below can be restricted to a fixed
3

Page 5

quantity of interest, decided ex ante, which will of course will sometimes be of interest.
Matching Procedures
Mahalanobis distance matching (MDM) and propensity score
matching (PSM) are built on specific notions of distance between observations of pre-
treatment covariates. MDM measures the distance between the two observations Xiand
Xjwith the Mahalanobis distance, M(Xi,Xj) =
?(Xi− Xj)?S−1(Xi− Xj), where S
is the sample covariance matrix of X. In PSM, we first collapse the vectors to a scalar
“propensity score,” which is the probability that an observation receives treatment given
the covariates, usually estimated by a logistic regression, πi≡ Pr(Ti= 1|X) = 1/(1 +
eXiβ); then, the distance between observations with vectors Xiand Xjis the simple scalar
difference between the two estimates ˆ πi− ˆ πj(or sometimes Xiβ − Xjβ).
The most common implementation of each approach is to apply one-to-one nearest
neighbor greedy matching without replacement (Austin, 2009, p.173). This procedure
matches each treated unit in some arbitrary sequence to the nearest control unit, using that
method’s chosen distance metric. Some procedure is then applied to remove treated units
that are unreasonably distant from (or outside the common support of) the control units
to which they were matched. The most common such procedure is calipers, which are
chosen cutoffs for the maximum distance allowed (Stuart and Rubin, 2007; Rosenbaum
and Rubin, 1985).
Coarsened Exact Matching (CEM) works somewhat differently. First, each variable is
temporarily coarsened as much as is reasonable. For example, years of education might be
coarsened for some purposes into grade school, high school, college, and post-graduate.
Second, units with the same values for all the coarsened variables are placed in a single
stratum. And finally, for further analyses, control units within each stratum are weighted
to equal the number of treated units in that stratum. Strata without at least one treated and
one control unit are thereby weighted at zero, and thus pruned from the data set. (The
weights for each treated unit is 1; the weights for each control unit equals the number of
treated units in its stratum divided by the number of control units in the same stratum,
normalized so that the sum of the weights equals the total matched sample size.) The
unpruned units with the original uncoarsened values of their variables are passed on to
4