How is causal analysis different from regression analysis?

Usually in Regression Analysis we consider as known the cause (x) and the effect (y) while we are regressing y ~ x.

In Causal Analysis the question is to find if x → y or if y → x.

1) Multiple regression, even given a perfect fit along some line in Rn, doesn't imply direction. It's similar to why correlation isn't causation: simplistically, correlation between A and B implies that either A causes B, B causes A, or C causes both. Multiple regression shows similar relationships (i.e., directionless and based upon the spread of data points from the regression "line").

2) Causal analyses are somewhat controversial in whether they do, in fact, show causal relationships. For one thing, there is no agreed upon definition of causality (even were we talking only about efficient causality). For example, a common model is counterfactual (granted that x and y occurred, x causes y iff had x not occurred, y wouldn't have occurred). This seems to fail in quantum physics. Another is very much mathematical in that a cause y is treated as a function of x (this runs into problems with circular causality, Rosen's [M,R] systems, etc.). In statistics, inferential models and methods can, theoretically, allow one to infer direction. Bayesian inference in particular is a fairly straightforward generalization of the epistemological basis of counterfactual causality to a quantitative framework.

3) There are a variety of causal analyses in the social sciences and not all of them correspond to or subscribe to the same model of causality. Stable association, for example, is a common basis for causal analyses but can be implemented via numerous statistical methods.

4) Some statistical methods are more or less designed from the ground up for this purpose (structural equation modeling is fairly prominent here). SEM and similar methods rely much on path analysis and graph-theoretic models. Simplistically, instead of showing relationships among variables as in multiple regression, causal models frequently rely on proposed/possible factors hypothesized to be causes (confirmatory analyses) or they use cluster, factor, path, etc., analyses to "uncover" hidden/latent relationships among variables by mathematically classifying the data points and projecting them onto a new lower-dimensional space. Whether exploratory or confirmatory, the main idea is to explore not just the relationship among variables but underlying relationships that explain the data.

5) Finally, when we're really lucky, someone uses predictive models. These are much easier to evaluate in terms of their validity: if your model says x is supposed to happen and it doesn't, your models is wrong (of course, "all models are wrong but some are useful"; however, this really means that there models will always be off at least a little not that they are "wrong").

Another tool for causality analysis is Granger causality:

Unfortunately the relevant test many times gives the same answer for x → y with y → x !

It is a highly open and interesting area...

There is one basic problem concerning the 'Granger causality'. In the literature, the so-called 'Granger causality' is an econometric relationship which tests whether additional information from variable x helps explain y. But the variables x and y should be stochastic variables.

If I may continue my previous answer, in the 'Granger causality' the variables x and y should be stochastic (not deterministic) variables. In the regression analysis (e.g. OLS), this assumption is not necessary, i.e. the variables could be deterministic as well. 

Toda Yamamoto gives a better test of causality than the so called Granger causality because of its estimation bias  and other other shortcomings of the causality analysis

Regression analysis is a statistical procedure to obtain estimates. Causal analysis isn't a specific statistical procedure, it can be regression analysis, path analysis, or variance analysis. For example, if the research designs allows causal conclusions, then a regression analyses on that data will be a causal analysis.  

“There are two main uses of multiple regression: prediction and causal analysis. In a prediction study, the goal is to develop a formula for making predictions about the dependent variable, based on the observed values of the independent variables….In a causal analysis, the independent variables are regarded as causes of the dependent variable. The aim of the study is to determine whether a particular independent variable really affects the dependent variable, and to estimate the magnitude of that effect, if any.”

As in most regression textbooks, I then proceeded to devote the bulk of the book to issues related to causal inference—because that’s how most academic researchers use regression most of the time.

Outside of academia, however, regression (in all its forms) is primarily used for prediction. And with the rise of Big Data, predictive regression modeling has undergone explosive growth in the last decade. It’s important, then, to ask whether our current ways of teaching regression methods really meet the needs of those who primarily use those methods for developing predictive models.

Despite the fact that regression can be used for both causal inference and prediction, it turns out that there are some important differences in how the methodology is used, or should be used, in the two kinds of application. I’ve been thinking about these differences lately, and I’d like to share a few that strike me as being particularly salient. I invite readers of this post to suggest others as well.

1. Omitted variables. For causal inference, a major goal is to get unbiased estimates of the regression coefficients. And for non-experimental data, the most important threat to that goal is omitted variable bias. In particular, we need to worry about variables that both affect the dependent variable and are correlated with the variables that are currently in the model. Omission of such variables can totally invalidate our conclusions.

With predictive modeling, however, omitted variable bias is much less of an issue. The goal is to get optimal predictions based on a linear combination of whatever variables are available. There is simply no sense in which we are trying to get optimal estimates of “true” coefficients. Omitted variables are a concern only insofar as we might be able to improve predictions by including variables that are not currently available. But that has nothing to do with bias of the coefficients.

2. R2. Everyone would rather have a big R2 than a small R2, but that criterion is more important in a predictive study. Even with a low R2, you can do a good job of testing hypotheses about the effects of the variables of interest. That’s because, for parameter estimation and hypothesis testing, a low R2 can be counterbalanced by a large sample size.

For predictive modeling, on the other hand, maximization of R2 is crucial. Technically, the more important criterion is the standard error of prediction, which depends both on the R2 and the variance of y in the population. In any case, large sample sizes cannot compensate for models that are lacking in predictive power.

3. Multicollinearity. In causal inference, multicollinearity is often a major concern. The problem is that when two or more variables are highly correlated, it can be very difficult to get reliable estimates of the coefficients for each one of them, controlling for the others. And since the goal is accurate coefficient estimates, this can be devastating.

In predictive studies, because we don’t care about the individual coefficients, we can tolerate a good deal more multicollinearity. Even if two variables are highly correlated, it can be worth including both of them if each one contributes significantly to the predictive power of the model.

4. Missing data. Over the last 30 years, there have been major developments in our ability to handle missing data, including methods such as multiple imputation, maximum likelihood, and inverse probability weighting. But all these advances have focused on parameter estimation and hypothesis testing. They have not addressed the special needs of those who do predictive modeling.

There are two main issues in predictive applications. First, the fact that a data value is missing may itself provide useful information for prediction. And second, it’s often the case that data are missing not only for the “training” sample, but also for new cases for which predictions are needed. It does no good to have optimal estimates of coefficients when you don’t have the corresponding x values by which to multiply them.

Both of these problems are addressed by the well-known “dummy variable adjustment” method, described in my book Missing Data, even though that method is known to produce biased parameter estimates. There may well be better methods, but the only article I’ve seen that seriously addresses these issues is a 1998 unpublished paper by Warren Sarle.

5. Measurement error. It’s well known that measurement error in predictors leads to bias in estimates of regression coefficients. Is this a problem for a predictive analysis? Well, it’s certainly true that poor measurement of predictors is likely to degrade their predictive power. So efforts to improve measurement could have a payoff. Most predictive modelers don’t have that luxury, however. They have to work with what they’ve got. And after-the-fact corrections for measurement error (e.g., via errors-in-variables models or structural equation models) will probably not help at all.

I’m sure this list of differences is not exhaustive. If you think of others, please add a comment. One could argue that, in the long run, a correct causal model is likely to be a better basis for prediction than one based on a linear combination of whatever variables happen to be available. It’s plausible that correct causal models would be more stable over time and across different populations, compared with ad hoc predictive models. But those who do predictive modeling can’t wait for the long run. They need predictions here and now, and they must do the best with what they have.

a very interesting topic by PAUL ALLISON published at : http://www.statisticalhorizons.com/prediction-vs-causation-in-regression-analysis

Dear Hisham, interesting analysis.

However we have to make an agreement for the quality of our available tools:

I think that we have given too much attention at 1. above and this has led to discussion about monstrous regressions, especially in Macroeconomics, where all countries is supposed to follow certain relations (Philips curves and other).

Traditional analyses of causality specify causal relations in terms of the usual logical conditions of

i) necessity: C is a cause of E iff (if and only if) both are real and C is necessary for E (that is, E cannot occur without C), and

ii) sufficiency: C is a cause of E iff both are real and C is sufficient for E (that is, whenever C occurs, E too does).

However, the presence of multiple causes (over-determination) renders the necessary condition in the above specification ineffective. In fact, virtually all the rational schools of Indian philosophy recognized that effects might require a conjunction of causes to occur. Thus the Buddhist scholars emphasized that cause and effect need not be linear in relation, but that desired effect requires a conjunctive set of right conditions for their fruition (pratītya samutpāda): thus, for a plant to grow successfully, it would need not only the right seed, but also the right type of soil, fertilization, sunlight and water. It is partly in recognition of this fact that the INUS condition of causality (Mackie 1965) and the related probabilistic causality (for example, Suppes 1970) emerged to gain some universal acceptance. The INUS condition for some effect is “an insufficient but necessary part of a condition which is itself unnecessary but sufficient for the result” (Mackie 1965). Suppose, for example, a short circuit causes a fire in a certain house; but the short circuit is not a necessary condition for the fire, it could happen in a number of other cases, for example, by “the overturning of a lighted oil stove”. And the short circuit, by itself, is not sufficient also; the fire would not have broken out had there been no inflammable material nearby, had there been “an efficient automatic sprinkler at just the right spot”, and so on. The short circuit is thus a part, “an indispensable part”, of some constellation of conditions jointly sufficient for the fire. It is “an indispensable part” because given that it is this set of conditions that has occurred, rather than some other set sufficient for fire, the short circuit is necessary: fire does not occur in such circumstances without short circuit. Thus the short circuit is an insufficiently necessary but unnecessarily sufficient (INUS) condition for the fire. In economics, and in social sciences in general, causality seems to be defined in this sense; the INUS condition corresponds to the ceteris paribus condition (also see Hicks 1979: 45).

It is in a similar vein that the probabilistic school defines causality: cause makes its effect more likely. The central idea of probabilistic causality is that cause raises the probability of its effect and is formally expressed using the conditional probability apparatus. If St – 1= i and St = j represent events that potentially stand in causal relations, then the event i is said to be a prima facie cause of the event j if and only if

Pr{St = j | St – 1 = i} > Pr{St = j}, or simply

P(j | i) > P(j),

where P(j) > 0, keeping the assumption of the temporal order of the events a la Hume (Suppes 1970: 12). Note that i is only a prima facie cause, not a cause simpliciter, since there may be clear circumstances of no causality between i and j, even though the prima facie conditions are satisfied. To cite the classic example, a falling barometer is a prima facie cause of a storm, but we do not take it as the genuine cause, since we know that it is a fall in atmospheric pressure that causes both the effects of falling barometer and storm. Such problems of spurious correlation, where both A and B are caused by a third factor C, and A prima facie causes B, that is, P(B | A) > P(B | not-A), are addressed by requiring that cause raises the probability of its effect ceteris paribus. It should be emphasized here that “‘measures of association’ is the term commonly used in the statistical literature for measures of causal relationship” required by this definition (Suppes 1970: 13). In this sense, the above definition holds i and j as positively associated; if the inequality is reversed, they are negatively correlated. If equality holds, then the two are probabilistically independent. Hence it is argued by some that ‘greater than’ be replaced with ‘does not equal’ in the above definition for causality (such that i causes j if P(j | i) is not equal to P(j)), for example, Granger (1980:330). Another argument requires that a causal relationship make the associated event probable, such that i causes j if P(j | i) > 0.5 (Papineau 1985: 57ff). (For references, see my article "Causality and Error Correction In Markov Chain: Inflation In India Revisited".)

About the so-called Granger causality test: though popularly known as Granger (non-) causality test (Granger 1969), it was first suggested by Wiener (Wiener 1956), and is often referred to more properly as Wiener-Granger causality test. This model has prompted a great deal of debate among economists (for example, Zellner 1979) and even philosophers (for example, Holland 1986). who have questioned the very term ‘causality’; how can one mean ‘cause-effect’ relationship, when there is only temporal lead-lag relationship? The proper term should not have been ‘causality’, but ‘precedence’ as suggested by Edward Leamer.

Unfortunately there have been several studies to infer ‘cause-effect’ relationship using this methodology. Thus wrote  Adrian Pagan (1989) on Granger causality: “There was a lot of high powered analysis of this topic, but I came away from a reading of it with the feeling that it was one of the most unfortunate turnings for econometrics in the last two decades, and it has probably generated more nonsense results than anything else during that time.” Pagan, A.R. (1989), '20 Years After: Econometrics 1966-1986,' in B. Cornet and H. Tulkens (eds)., Contributions to Operations Research and Econometrics, The XXth Anniversary of CORE, (Cambridge, Ma., MIT Press)

Let us be clear about the methodology of regression that it has nothing to do with 'causality'. Similar to correlation, it too provides some information about the relationship between variables. Let us forget the old text book misinterpretation that  correlation shows association and regression means 'causality'. Remember that correlation between X and Y is given by Cov(X,Y) divided by the product of the SDs of X and Y, where the denominator is only to standardize the measure; the association being determined by the Cov(X,Y), whereas the regression coefficient is given by Cov(X,Y) divided by Var(X), again the association being determined by the Cov(X,Y)! Let us not impute anything esoteric onto what is obvious!

Causal analysis = regression analysis (or any analysis) + theory (and hypothesis)

From a causal point of view regression estimates the direct causal impacts, but neglects indirect effects. In addition, most causal analysis methods allow to include latent variables not just manifests. SEM models can under some assumptions handle confounder that may lead to uncausal findings in regression models. Latest Causal Analysis methods build on Machine Learning techniques and can explore unexpected properties of causal relations such as unexpected nonlinearities and unexpected interactions.

Regression is a statistical analysis technique, the purpose is to predict a target variable y according to some other variables x1, x2, ..., namely, if you passively "see" that X=x, what the value of Y will be?

Regression cares about correlation relationships. You can always get a regression formula between Y and X, even when they by no means entail any causal relationships. 

However, correlation does not necessary mean causation. 

An example is that we can build a regression model between "chocolate consumption quantity of a person" and "the probability the person to win the Nobel price" (cf.: http://www.businessinsider.com.au/chocolate-consumption-vs-nobel-prizes-2014-4?r=US&IR=T ) , but there shouldn't exist actual causal relationship between them.

Causal analysis tries to estimate the effect of intervention. that is, if you actively "make" X=x, what will Y be?

For better understanding, I really recommend Judea's very classic Causality book "Causality: Models, Reasoning and Inference".

Hello Everyone,

Recently in a paper where I used panel data and applied PMG framework and noted that few of my variables are insignificant and later when I applied Heterogeneous panel causality test I noted all my studied variable are showing significant bidirectional causality. 

Now I am in a paradoxical situation. PMG model is suggesting for no significant relation while Dumitrescu and Hurlin (2012) test showing bidirectional s causal significant relation.

Hello Everybody. Good to see you here. This question has actually been answered, in the linear limit, by Liang (2014) in the following paper:

X.S. Liang 2014: Unraveling the cause-effect relation between time series. Phys. Rev. E, 90, 052150.

which is freely available at the Physical Review E site as well as ResearchGate . A news report in PhysicsToday can be seen at

http://physicstoday.scitation.org/do/10.1063/PT.5.7124/full/

For linear systems, a regression cofficient has to be multiplied by another term in order to be made for causal inference. Other more theoretical papers, such as

Liang 2016: Information flow and causality as rigorous notions ab initio. Phys. Rev. E, 94, 052201.

are also available. Thanks for the attention.

In regression analysis we only adjust for measured confounders whereas in causal modelling we adjust for measured as well as unmeasured confounders using different approaches.

Causal model involve regression or correlation analysis as well as a strong theoretical logic linking the two or more variables. Regression is changes between a dependent and one or more independent variables, the changes observed in one variable due to some unit changes other variable(s). It does not indicate causality in phenomena. Structural Equation Modelling (SEM), path analysis, Confirmatory Factor Analysis are some of the few causality oriented statistics.

Causal Model utilizes path analysis which is an extension of multiple regression useful in determining the relationship between the dependent variable and many independent variables. It enables assessment of the adequacy of fit of a hypothesized model to the data as indicated by the degree to which the specified model led to an exact production on population variance matrix of the manifested variables. This is model with with a chi-square/degrees of freedom value between 0 and 2 with p-value greater than or equal to 0.05. The Root Mean Square Error Approximation value must be less than 0.05. The other indices such as Normal Fit Index, Tucker Lewis Index, and Goodness of Fit Index must be all greater than 0.95.

I'm sorry you got so some wrong answers Robin, albeit some good answers.

Regression, propensity-score, Bayes, etc, are statistical concepts. Causal analysis is based on causal assumptions, which have to be assessed. You may have particular cases where regressions coefficients have a causal meaning, beyond their statistical trends perspective but they are just not the rule.

The key issue you should keep in mind is to differentiate the statistical concepts from the causal one. Historically, there are a lot of misconception in Econometrics about the subject. Go to amazon and type causal inference and read all those books. Good luck :9

Regression (also called linear regression) is used when there is a linear relationship between dependent and independent variables or covariates. The Covariates have no correlation, i.e., no multicolinearity. However, if the covariates have mutual correlations (positive or negative), the relationships between dependent variable and its covariates become casual that can be determined through path analyses.