Correct tests to run when Homogeneity of variance is violated in ANOVA?
I have been running some data in SPSS and the homogeneity of variance test has been violated. All three groups in the test have the same sample size. Am I correct in thinking the best tests to run are the Welch and Brown-Forsythe tests and then a post hoc test of Games-Howell.
I am new to stats so thank you for the help in advance.
Before searching a procedure that will work under weird assumptions you should first try to find out what is the reason that your data behaves so considerably different than expected under your assumptions. This way you have the chance to better understand the whole data and to better interpret the results. Some of these questions are:
Why does the variance of the residuals depend on the predictors?
How does it depend on the predictors?
Can you identify a more reasonable model for the data?
If the symmetry of the residuals is not a concern but the uncertainty of the estimates, then you should consider a weighted regression. Using weights inversely proportional to the variance often works well.
Josh, you appear to be concluding that you have problematic heterogeneity because some test of the variances was significant. This raises several questions (Q) & comments (C).
Q1. What are the SDs?
Q2. What homogeneity of variance test did you use?
C1. When the sample sizes are all the same (as in your case), or nearly the same, ANOVA is quite robust to heterogeneity of variance. As Box (1953) said, "To make the preliminary test on variances is rather like putting to sea in a rowing boat to find out whether conditions are sufficiently calm for an ocean liner to leave port!"
C2. You could try some simulations yourself to try to determine how much impact your observed level of variance heterogeneity is likely to have. The link below shows how to do this using Stata, for example.
C3. If you are talking about one-way ANOVA, the standard methods for dealing with heterogeneity of variance are the Welch or Brown-Forsythe F-tests. (These are options for the ONEWAY procedure in SPSS, for example.) Alternatively, if you estimate your model with a procedure intended for multilevel models (e.g., MIXED procedure in SPSS, proc MIXED in SAS, etc), you can allow the variances to be heterogeneous. This approach can be used for factorial ANOVA models too. Here are some examples using SPSS MIXED.
* One-way ANOVA via MIXED procedure.
* COVTYPE(DIAG) on the REPEATED sub-command allows
* the variances to be heterogeneous (by group).
MIXED Y BY Group
/FIXED=Group | SSTYPE(3)
/REPEATED=Group | SUBJECT(UniqueID) COVTYPE(DIAG)
/EMMEANS = tables(Group).
* Two-way ANOVA via MIXED, allowing for heterogeneity of variance.
thanks for the help guys it is much appreciated. I'm new to learning about stats so only have a basic knowledge so far.
The data which i have violated the levene's test of homogeneity of varinace with a result of .048. I have also ran a Shapiro-Wilko test for distribution as well on the data before I ran the one-way ANOVA
At first, reading your question and various answers here, I thought you were looking at heteroscedasticity in a regression, which should be treated more as a degree, not yes there is heteroscedasticity, or no there isn't. But then it became apparent that you are looking at different samples, not regression, and just comparing variance estimates for those samples, right? Well, if so, then the estimates of variance would differ from sample to sample, somewhat, even if drawn from the same population. And even if they came from different populations with different variances, as Jochen indicated, you have to think about how this may or may not impact your study.
If you are looking at differences in means, you might just want to get an estimate of each mean, an estimate of each variance, and any covariance if appilcable, and just estimate confidence intervals about differences in means, one pair at a time. This could be much more meaningfully interpreted than ANOVA for all your samples.
Dear Josh, I am experiencing the same problem as you did. Which approch did ypu take finlally?... I have a 3 factorial design of 2X2X3 levels which give me 12 treatments. I ran the test of homogeneity of variance and it is significant so I can not apply the factorial ANOVA. I was told to apply a Kruskal Wallis test but that is a one way ANOVA. I am also new to statistics. If you have any input to my case I would appreciate it very much.
Is the amount of heteroscedasticity relevant (to your problem)?
Is there a more sensible error-model than the normal distribution for your data (e.g. are you analyzing concentrations, rates, waiting-times, counts, proportions that ought to be handeled by gamma-, Poisson-, negative-binomial, beta-, binomial or some other error-model)?
I also have a problem with homogeneity of variance. I run Mixed model ANOVA with 2 factors (one within and one between), each factor has 2 levels, each level consist of 12 participants When I run SPSS, the result in Levene ‘s test is violated in one of treatment. Can I still run F test? Or should I make my data homogeneous first?
Does the assumption of normal distributed residuals make sense in the first place? If so, you can use a weights to account for the diffence in variance. If not, chose a more reasonable distributional model (you may then still check things like over- or under-dispersion, as the models typically assumes some relationosship between variance and expectation).
I am new to stat test. I have two factorial data(4 levels under each factor) pooled from two years study. In my case, data are conforming the normality test but failed for homogeneity variance test. now what i should do?. if i go for non parametric test, do i possible to get information regarding the interaction between factors?
No test! Just check it. Plot the data. Look. Maybe calculate the variances and simply compare them. Is one at least ten times larger than another?
I don't know your sample size(s). I assume this is large enough to get some reliable impression of the variance. If not, there is anyway not much to do.
Another thing: do the variances increase with the means? If so, it may be that the normality assumption is not ok in the first place. You should then check the distribution of the logarithms (usually, the variable is log-normal distributed, so the logs will be normal distributed, and they shuld have homogeneous variances).
Based on the question and answers I think its pertinent to make another question here. I want to use Games-Howell post hoc test in Stata but i haven't found to the moment found a way to run this test. Does anybody has a solution for this?
Hello Carlos Vicente Rey. Type <search pwmc> in Stata's command window. From the help for -pwmc-:
pwmc performs pairwise comparisons of means. It computes pairwise differences of the means of varname over the levels of over(varname). Confidence intervals are derived using procedures allowing for unequal variances across groups.
pwmc implements Dunnett's C, GH and T2 procedures, as proposed by Dunnett (1980), Games and Howell (1976) and Tamhane (1979).
Just to clarify, I have used robust estimates, as my Levene's test was highly significant (p<.001), and I have quite unequal groups sizes for one of the factors. If any statistical amateurs like me are reading this with the same problem, and want a paper that tells you why that might be a problem, I found Field and Wilcox, 2017 pretty non-technical on this. Check it out here: https://osf.io/v3nz4/).
My sample size is 255.
So SPSS generates my output, and all is well and good.
As you might expect, SPSS dutifully outputs: the normal "Test of Between Subjects Effects" which contains my Type III Sum of Squares, F statistics, p value and partial eta sqaured.
It also produces a table of "Paramter Estimates with Robust Standard Errors", which contains the b coefficient, robust standard error, the t-statistic, the p value, and confidence intervals (which I assume are confidence intervals for the b coefficients). It produces such information for one level of each of the terms in the model (i.e. it gives estimates for level 1 of factor 1, and no estimate for level 0 of the same factor--as SPSS says "This parameter is set at 0 because it is redundant"--presumably because the b co-efficients are an estimate of the effect of one level of the factor relative to the other 'base' level of the factor, when also controlling for all other terms in the model).
The rub is that I now don't know what to do with my output in terms of how to report it in papers (I will publish in psychology).
Here's where I could do with your help:
If this were a non-robust three-way ANOVA, I would say something like the following in my paper in this case: "a three-way ANOVA revealed a main effect of factor 1 (F(1, 247)=42.34, p<.001, partial eta sqaured=.14)" and I might report means of the two groups too along with their standard deviations.
But as I understand it, I can't do that in this case, because the statistics provided by the "tests of between subjects effects" table are not calculated using robust standard error estimates. I assume this because the statistics provided are the same as if I had told SPSS to do a normal three way ANOVA without selecting the "Parameter estimates with robust standard errors". I also *believe* that it would defeat the point to report these statistics, and moreover would be misleading to do so, as the F statistic and p value can be affected by heteroskedasticity. That was the whole reason for wanting to use robust standard errors!
So what have I done instead? At the moment, I've written: "A three-way ANOVA was conducted, using heteroskedasticity-consistent standard errors as a Levene’s test indicated that homogeneity of variance in the DV could not be assumed and the number of participants in each group were not equal. There was a main effect of variable 1 (b=1.04,t(247)=2.26, p<.05, ηp2=.02): with participants in the... (imagine descriptions of differences in means between groups here)."
My general question is:
(1) Is that daft? Is a reviewer going to scoff and reveal me for the fraud that I feel like I am when it comes to statistics?
Notice, I am not provided with a degree of freedom for the t statistic in "Parameter Estimates with Robust Standard Errors". As I understand it though, I can just read the "df" for this test statistic off the "Error" line from the "Tests of Between Subjects Effects" table (i.e. the one that contains output for a normal, i.e. non-standard-error-corrected, ANOVA), since the t statistic is for the b coefficient, and the b coefficient gives me an estimate of the effect of a one-unit change in the factor concerned on my outcome variable (which in this case, is the same as talking about the effect of an experimental condition relative to a control condition), *when controlling for all the other terms in the model*. Where it's the lattermost part of that sentence that tells me that I have the same amount of degrees of freedom as a normal ANOVA.
My specific questions are:
(2) Is it fine to take my df for the t statistic from the traditional ANOVA output in SPSS?
(3) Is it fine to call what I've done a three-way ANOVA, since there is no F statistic reported here, but rather, horror of horrors, a t statistic?
(4) Does anyone know of any examples of papers in psychology that illustrate how three-way ANOVAs with robust standard errors tend to be reported (I can't find any)?
If you could reply in idiot-proof terms that would be so appreciated. To give you an idea of who you're dealing with here, I tried to learn R last year: after three days of watching tutorials online, I still couldn't even open my dataset in R, and nearly threw my computer out the window.
(Finally, it you're an amateur like me reading this and faced with a similar problem, you might want to check out these other threads for some great tips--I wouldn't have been able to get to this point without reading these first:
When analyzing experimental chemical data, it is often necessary to incorporate the structure of the study design into the chemometric/statistical models to effectively address the research questions of interest. ANOVA-Simultaneous Component Analysis (ASCA) is one of the most prominent methods to include such information in the quantitative analysi...
Non-functional requirements (NFRs) demonstrate how the software system works, though functional requirements represent the tasks of the system (software system). It does not indicate that the latter is more significant, but a majority of requirement collecting approaches emphasis on functional requirements. Therefore, due to the subjective nature a...
When one wants to follow up a significant test, it runs into the problem of multiple comparisons. We are interested in drawing inference on various component hypothesis when the overall hypothesis is rejected. The problem of simultaneous estimation of the entire class of k * =k(k-1)/2 pairwise comparisons θ i -θ j of the means θ i , 1≤i≤k, for imba...