# How can I apply a power analysis on an interaction in order to show that even if I test more participants, I'll never get the effect ?

The interaction is not significant but I need to show that it is not a matter of a lack of power. What is the simplest way to do so?

## Popular Answers

Gabor Borgulya· Freelance biostatistics consultant and locum doctorA remark on "never getting the effect": if there is a statistical interaction or not depends on the measurement scales of the (two or more) predictors the interaction of is studied. For example if two continuous variables are NOT in interaction using their original scales, then their log transformed variants will be in interaction. So in my opinion showing no interaction also means finding the appropriate measurement scales.

If you have the appropriate measurement scales you can declare that there is no interaction if you powered your study for equivalence and the study results confirmed it; alternatively you can declare that "you will never get the effect" if your power analysis demonstrates that showing the interaction with a test for difference would require a sample size that is practically infeasible.

Power analyses are performed before data collection. The wording "is not significant" suggests that you may already have the data. In this case look at the confidence interval after appropriate transformations and compare it to the relevance thresholds.

Jamie I D Campbell· University of SaskatchewanYou could also use MorePower 6.0 (Campbell & Thompson, 2012). MorePower 6.0 computes sample size, effect size and power statistics for a specified ANOVA effect. It also calculates confidence intervals for the effect based on formulas from Jarmasz and Hollands (2009), as well as Bayesian posterior probabilities for the null and alternative hypotheses using the Bayesian Information Criterion (Masson, 2011; Wagenmakers, 2007). The program affords a straightforward comparison of these alternative approaches to interpretation of ANOVA. MorePower 6.0 is freely available at https://wiki.usask.ca/pages/viewpageattachments.action?pageId=420413544

The MorePower calculator is very easy to use, but let me know if you have any questions.

Cheers

Jamie

Campbell, J. I. D., & Thompson, V. A. (2012). MorePower 6.0 for ANOVA with relational confidence intervals and Bayesian analysis. Behavior Research Methods, 44, 1255-1265. doi: 10.3758/s13428-012-0186-0

## All Answers (41)

Gabor Borgulya· Freelance biostatistics consultant and locum doctorA remark on "never getting the effect": if there is a statistical interaction or not depends on the measurement scales of the (two or more) predictors the interaction of is studied. For example if two continuous variables are NOT in interaction using their original scales, then their log transformed variants will be in interaction. So in my opinion showing no interaction also means finding the appropriate measurement scales.

If you have the appropriate measurement scales you can declare that there is no interaction if you powered your study for equivalence and the study results confirmed it; alternatively you can declare that "you will never get the effect" if your power analysis demonstrates that showing the interaction with a test for difference would require a sample size that is practically infeasible.

Power analyses are performed before data collection. The wording "is not significant" suggests that you may already have the data. In this case look at the confidence interval after appropriate transformations and compare it to the relevance thresholds.

Martin Schmettow· Universiteit TwenteWhat you actually have to do is define for yourself what a practically or theoretically relevant effect size would be. Ziliak and McColskey call this the oomph [2]. Then, perhaps you are already there with your current data, when the confidence limits of your current estimate fall inside the "irrelevant" range.

[1] White, John Myles. Criticism 5 of NHST: p-Values Measure Effort, Not Truth. http://www.johnmyleswhite.com/notebook/2012/07/17/criticism-5-of-nhst-p-values-measure-effort-not-truth/

[2] Ziliak, S., & McCloskey, D. N. (2009). The cult of statistical significance. How the Standard Error Costs Us Jobs, Justice, and Lives.

Luis A. Apiolaza· University of CanterburyYou may have to think what would be the minimum size of the interaction effect that you'd consider significant. Based on that you could simulate data with your sample size and see if you can pick it up.

Jacques P. Beaugrand· Université du Québec à MontréalSome furmulae allow you to increase the N, keeping other parameters constant.

If the power does not change, then it will be a demonstration that continuing won't change anything in principle.

I would also examine the possibility of *bootstrapping* that is performing a statistical simulation sampling from the data you already have obtained (you can also add noise) to see what happens with the parameters of the two populations compared.

Warren L May· University of Mississippi Medical CenterCatherine Thevenot· University of LausanneLuisiana Cundin· Die Wand : leben heißt kampfDeletedGeorge Ostrouchov· Oak Ridge National LaboratoryDavid H Abbott· U.S. Department of Veterans AffairsTobias Katus· Birkbeck, University of LondonRather, you might want to look at effect sizes (η²). For instance, a minimum-effect null-hypothesis would state that an effect is practically negligible if η² < 0,01 (less than 1% of explained variance).

If you are able to understand german, have a look at this:

http://books.google.co.uk/books?id=13GbPUYAUHsC&pg=PA635&lpg=PA635&dq=bortz+minimum+effekt&source=bl&ots=1y6X91U8PG&sig=RdJsvwEy4E7XDcgQtpjQtgN5IBU&hl=de&sa=X&ei=1g3DUaL1AcGWPeqwgOgG&ved=0CCsQ6AEwAA#v=onepage&q=bortz%20minimum%20effekt&f=false

Otherwise, this publication might be helpful:

Murphy, K. R., & Myors, B. (2004). Statistical power analysis: A simple and general model for traditional and modern hypothesis tests (2nd ed.).Mahwah, NJ: Lawrence Erlbaum Associates.

cheers

Tobi

Ellen Frances Zakreski· McGill UniversityAfter all, science aims to falsify (reject the null) rather than seeking confirmatory evidence.

The answer to your question however depends on the nature of the interaction. Briefly describing the experimental design etc. could help others assist you.

Good luck Catherine!

Daniel Gallant· Université du Québec à Rimouski UQARFor calculating statistical power one could simply simulate data according to deisred parameters and conduct the test a large number of time and see the proportion of the trial where the test correctly rejects the null hypothesis. The way of simulation makes it more easy to apply statistical power analysis on unconventional tests and maybe on things like interraction terms

Tom CloydNoelle L Brown· United States Naval Research LaboratoryGabor Borgulya· Freelance biostatistics consultant and locum doctorTobias Katus· Birkbeck, University of LondonRoughly, it works like this:

1) you have to define a "minimal effect size" ... if your empirical effect is smaller, you can consider it as practically negligible, even if it is statistically significant.

2) you have to test a lot of participants. How large your sample needs to be depends on your design (degrees of freedoms) and of course, your predefined "minimal effect size".

You can lookup the optimal sample size in a table here:

Bortz J, Döring N. 2006. Forschungsmethoden und Evaluation für Human- und Sozialwissenschaftler. 4th ed. Berlin, Heidelberg, New York: Springer.

The power analysis approach is quite useful for applied research, because there it makes sense to consider whether an effect size is worth the effort of an intervention. For instance, if you want to sell an expensive drug against a deadly disease, few people will buy it if they survive 3mins longer relative to baseline (although 3mins may be a highly significant increase in survival time if you had tested a million people).

I don't know whether power analysis is accepted in fundamental research, though. Maybe it is worth to give it a try.

cheers

Tobi

John Christie· Dalhousie UniversityIverson, G. J., Lee, M. D., & Wagenmakers, E.-J. (2009). Prep misestimates the probability of replication. Psychonomic Bulletin & Review, 16(2), 424-429.

Lecoutre, B., & Killeen, P. R. (2010). Replication is not coincidence: Reply to iverson, lee, and wagenmakers (2009). Psychonomic Bulletin & Review, 17(2), 263-269. doi: 10.3758/PBR.17.2.263

Iverson, G. J., Lee, M. D., & Wagenmakers, E.-J. (2010). The random effects prep continues to mispredict the probability of replication. Psychonomic Bulletin & Review, 17(2), 270-272. doi: 10.3758/PBR.17.2.270

Noelle L Brown· United States Naval Research LaboratoryBruce E Oddson· Laurentian UniversityThe best explication of the counter null I have found is in Rubin & Rosenthal's book on Contrast Analysis. There are also some easily accessible articles. e.g. doi:10.1111/j.1467-9280.1994.tb00281.x

Good luck.

Andy D Mealor· University of SussexBest of luck!

Dennis Cox· Rice UniversityI don't condone prejudice, simply put - pre-judgement. Anyway, the answer is fairly straightforward. A "power-analysis" is generally employed in the design of a study. Apparently, the study is done, so one needs only generate a valid confidence interval based on the data and if it includes the point of "no-effect" or overlaps with the region of "no-effect," then you are done. If you are still accruing patients or whatever and want to show futility, then that is another discussion.

Gerd Bohner· Bielefeld UniversityAs other commentators have pointed out, it may be fruitless to try and prove that you'll "never get the effect". However, it is of course possible to test if the lack of a significant effect may be due to a lack of power.

To do so, you may compute a post-hoc power analysis. That is, you calculate power based on an assumed population effect size, the actual N of your study, and the alpha level of your significance test. Regarding the population effect size, Jacob Cohen has introduced conventional values for "small", "medium", and "large" effect sizes in his book "Statistical power analysis for the behavioral sciences".

A relatively simple way of doing the calculations you need is to use G*Power, a software that is free for non-commercial use (download here: http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/download-and-register ).

Once you have started the program, choose the kind of statistical test you are applying and in the field "type of power analysis" choose "post-hoc". Then enter the input parameters, including the assumed population effect size. When you drag the mouse pointer over the effect size field, the program will show you Cohen's conventions for small, medium, and large effects. Of course you may wish to test all three possibilities.

Your final report may then look like this: "To examine if the absence of a significant interaction effect may be due to low power, I conducted a post-hoc power analysis using G*Power (ref.). This analysis showed that, for detecting a medium-sized population effect (f = .25; see Cohen, 1988) at an alpha-level of .05 (two-tailed), effective power was .xx."

Of course you wish ".xx" to be as large as possible. :o)

I hope this helps.

Gerd

Catherine Thevenot· University of LausannePatrice S. Rasmussen· University of South FloridaBest,

Patrice

Khan academy will answer your question with a video discussion.This will clarify this issue for you.

Patrice S. Rasmussen· University of South FloridaBlessings,

Patrice

Jamie I D Campbell· University of SaskatchewanYou could also use MorePower 6.0 (Campbell & Thompson, 2012). MorePower 6.0 computes sample size, effect size and power statistics for a specified ANOVA effect. It also calculates confidence intervals for the effect based on formulas from Jarmasz and Hollands (2009), as well as Bayesian posterior probabilities for the null and alternative hypotheses using the Bayesian Information Criterion (Masson, 2011; Wagenmakers, 2007). The program affords a straightforward comparison of these alternative approaches to interpretation of ANOVA. MorePower 6.0 is freely available at https://wiki.usask.ca/pages/viewpageattachments.action?pageId=420413544

The MorePower calculator is very easy to use, but let me know if you have any questions.

Cheers

Jamie

Campbell, J. I. D., & Thompson, V. A. (2012). MorePower 6.0 for ANOVA with relational confidence intervals and Bayesian analysis. Behavior Research Methods, 44, 1255-1265. doi: 10.3758/s13428-012-0186-0

Catherine Thevenot· University of LausannePatrice S. Rasmussen· University of South FloridaDr.Kromrey my Professor is a specialist or,obsessed with power analysis. However, if your results are not significant you need to look at the research design and look at more than that your power was good. It does not really matter to the world if you have great power...the point of having more power is to get a significant result. I think it sounds like you need to star with your sample size and how your experiment is designed. When the design of the study is appropriate you most probably will get better results and a good study should always augment the power as a given.

Just a thought.

Patrice S. Rasmussen· University of South FloridaCatherine Thevenot· University of LausannePatrice S. Rasmussen· University of South FloridaPatrice :)

Patrice S. Rasmussen· University of South FloridaI enjoy the pivot tables for playing with my data!

Best,

Patrice :)

Roshini Sooriyarachchi· University of ColomboJoachim Vandekerckhove· University of California, IrvineRoshini Sooriyarachchi· University of ColomboRobert Edmunds· Cardiff Metropolitan Universityhttp://www.gpower.hhu.de/en.html

Can you help by adding an answer?