Laurentian University
Question
Asked 20 October 2015
Aggregation with multilevel data: always a bad idea or not necessarily?
Hi,
I have daily level data nested within individuals from a diary study. Let's say I only have hypotheses on between level effects: in this case, is it then always wrong to aggregate the data over the days and use standard OLS type analyses (e.g. correlations)?
I know there are some concerns with aggregation, e.g. it assumes that within variance is zero, gives people with fewer observations relatively more weight).The latter is less of a problem in my data since people reported about equal numbers of observations and quite a lot of them.
But given that I only have expectations on between person effects, is aggregation such a bad thing?
Most recent answer
Dear Dirk,
When you say that it should be a theoretical decision I have to say that I am a little more on the fence about this. It should be a theoretical decision if your theory (see Kelvyn's resources) is precise enough. In your case you have already run the MLM and seen the result. So your basis to run a second OLS model is based on the evidence at hand and your desire to report r squared and some other easier to interpret statistics. In practice I see nothing wrong with reporting the finding from the MLM model and then following up in the way that you think makes the most sense.
In theory, of course, those results are already in the MLM and you have the opportunity to address other questions as they arrive.
I tend to be pragmatic about what I report. In this case I think you should report the initial MLM model and then follow it with what you believe to be the best description of your data. If you believe that readers are more likely to make correct judgements based on an OLS table then put it in.
In the original question you posed it was universal - is it always wrong. I think that many would answer MLM is in principle better. When you can show an MLM table that assures me that the interesting effects can be captured in an OLS table then I really can't see a problem with it! I have more trouble understanding the people who don't check in the first place!
Take care
All Answers (4)
Laurentian University
Dear Dirk,
Models are supposed to serve your interests, not the other way around. So don't feel compelled to use a multi-level model. Psychologists, especially, often take repeated measures and derive an average. They then analyse differences between averages. It is particularly suitable if you are actually interested in average performance and no curiosity about the actual distributions involved.
Two things to reflect on, though. One is that your between person effects can often be modelled more precisely and accurately using a multi-level model then can be done using an aggregated approach. The second is that the size of the difference between the two approaches can be easily verified if you use a MLM or HLM first... so assuming that the aggregation effects are irrelevant seems a little silly once you become accustomed to MLM/HLMs. The literature on the aggregation bias and ecological fallacy give reasonable pointers on what sorts of mistakes in interpretation are likely to be made by ignoring these points - however, as I wrote, in some fields aggregation is still the standard approach.
WRT to what you wrote I am not so sure that these points should dominate your decision. Aggregation doesn't assume that within level/person variance is zero. It is done precisely because people are deciding that the variance is large enough to matter and that they think it is irrelevant to what they wish to test. There is no compelling reason to worry about the weights given to participants as a function of the number of measurements if the pattern of missingness is not informative (MCAR) and you are disinterested in that level of variation. But participant averages can be weighted by their number of constituent observations - and Bayesians would likely insist on this - without much effort on your part.
So, to summarize, it is not always wrong and not always a problem; in your field aggregation may even make more sense ( I can picture how sources of variation like temperature, feed humidity, sunlight might be less interesting than the average milk output). The things to watch for are biases in interpretation and a rather unnecessary loss of information.
Hope this helps
1 Recommendation
Vrije Universiteit Amsterdam
Dear Bruce, thank you for your elaborate answer. You are right, of course, that you would actually decide for a multilevel model if you expect that there is variance at both levels and you want to take them apart.
I am asking because I ran a series of multilevel models, only including between effects. Effectively, a between effect is just a mean level difference (over the days). When I compare the coefficients from these models with my results when I aggregate my data and use OLS techniques, they are almost exactly the same.
So in regards to your comment "The second is that the size of the difference between the two approaches can be easily verified if you use a MLM or HLM first...": in my case there appear to be little if any difference at all.
The second thing is that I am comparing the relative influence of different between person variables: R2 in multilevel models can be tricky, and are more straightforward in OLS techniques.
Still, based on the possible pitfalls of aggregation, like the ones you name, it makes me inclined to in fact use multilevel models: the fact that aggregation does not appear to be a problem in my data should not influence the decision per se, it should be a theoretical decision. Would you agree?
Again thanks, Dirk
University of Bristol
Aitken and Longford (1986) - " the means on means analysis is meaningless".
Seriously though - it really depends on what you want to find out; and for what I do the within person variation can be important.
Theory is overwhelmingly concerned with averages - we have not usually thought about heterogeneity and a full random coefficient approach allows this in quite subtle ways. Thus men and women may have the same average but differ in their variability - to me this would be interesting. In a diary study the mean response at the weekend is the same as the weekday but the variability is less; again this is interesting. The ability to structure the variance by predictors (categorical and continuous) at any level is a powerful aspect of the random coefficient approach - for me multilevel modelling allows you to model explicitly the mean and the variance simultaneously and when I teach I stress this variance function approach:
THis paper looks at the aggregation and atomistic fallacy using some classic data
Laurentian University
Dear Dirk,
When you say that it should be a theoretical decision I have to say that I am a little more on the fence about this. It should be a theoretical decision if your theory (see Kelvyn's resources) is precise enough. In your case you have already run the MLM and seen the result. So your basis to run a second OLS model is based on the evidence at hand and your desire to report r squared and some other easier to interpret statistics. In practice I see nothing wrong with reporting the finding from the MLM model and then following up in the way that you think makes the most sense.
In theory, of course, those results are already in the MLM and you have the opportunity to address other questions as they arrive.
I tend to be pragmatic about what I report. In this case I think you should report the initial MLM model and then follow it with what you believe to be the best description of your data. If you believe that readers are more likely to make correct judgements based on an OLS table then put it in.
In the original question you posed it was universal - is it always wrong. I think that many would answer MLM is in principle better. When you can show an MLM table that assures me that the interesting effects can be captured in an OLS table then I really can't see a problem with it! I have more trouble understanding the people who don't check in the first place!
Take care
Similar questions and discussions
What does a CFA Factorloadings greater than 1.00 with standardised loadings in RStudio mean?
Hi,
I am running a CFA in RStudio and would like to get standardised factor loadings. However, I am still receiving factor loadings greater than 1.00 (see bold). Is this possible?
My thoughts:
- I thought I had used the right formula by typing standardized=True as last sentence.
- I thought the factor loadings can be identified by looking under Latent Variables - Estimates.
- I am doing a CFA and would like to get standardised factor loadings. However, I am still receiving factor loadings greater than 1.00. Is this possible?
Am I typing the wrong formula, looking at the wrong place, or is my assumption that standardised factor loadings can not be greater than 1.00 wrong? (or something else)?
Thanks!
INPUT
#cfa part 2, 18 items, 6 factors, ordinal data, unit variance identification
model <- "f1 =~ fb1 +fb2 + fb3 +fb14
f2 =~ fb9 + fb4
f3 =~ fb6 + fb7
f4 =~ fb12 + fb13
f5 =~ fb15 + fb16 +fb21
f6 =~ fb18 + fb19 +fb20 +fb22 + fb23"
fit<-cfa(model,std.lv=T,data=p2,ordered=items)
#standardised factor loadings
summary(fit,standardized=TRUE)
OUTPUT
vaan 0.6-11 ended normally after 44 iterations
Estimator DWLS
Optimization method NLMINB
Number of model parameters 69
Number of observations 328
Model Test User Model:
Standard Robust
Test Statistic 174.908 245.633
Degrees of freedom 120 120
P-value (Chi-square) 0.001 0.000
Scaling correction factor 0.872
Shift parameter 45.025
simple second-order correction
Parameter Estimates:
Standard errors Robust.sem
Information Expected
Information saturated (h1) model Unstructured
Latent Variables:
Estimate Std.Err z-value P(>|z|) Std.lv Std.all
f1 =~
fb1 0.557 0.075 7.412 0.000 0.557 0.477
fb2 0.446 0.077 5.775 0.000 0.446 0.453
fb3 0.788 0.076 10.437 0.000 0.788 0.669
fb14 0.514 0.080 6.459 0.000 0.514 0.527
f2 =~
fb9 0.886 0.160 5.543 0.000 0.886 0.836
fb4 0.646 0.120 5.390 0.000 0.646 0.618
f3 =~
fb6 0.905 0.058 15.548 0.000 0.905 0.790
fb7 1.013 0.054 18.597 0.000 1.013 0.827
f4 =~
fb12 0.588 0.092 6.430 0.000 0.588 0.523
fb13 1.146 0.117 9.792 0.000 1.146 1.038
f5 =~
fb15 0.879 0.059 14.886 0.000 0.879 0.743
fb16 0.866 0.057 15.177 0.000 0.866 0.767
fb21 0.798 0.064 12.463 0.000 0.798 0.678
f6 =~
fb18 0.831 0.051 16.323 0.000 0.831 0.694
fb19 0.761 0.055 13.729 0.000 0.761 0.694
fb20 0.785 0.056 13.960 0.000 0.785 0.676
fb22 0.681 0.058 11.655 0.000 0.681 0.624
fb23 0.586 0.063 9.292 0.000 0.586 0.518
Covariances:
Estimate Std.Err z-value P(>|z|) Std.lv Std.all
f1 ~~
f2 -0.004 0.086 -0.043 0.966 -0.004 -0.004
f3 0.053 0.086 0.619 0.536 0.053 0.053
f4 0.542 0.081 6.699 0.000 0.542 0.542
f5 0.267 0.086 3.112 0.002 0.267 0.267
f6 0.244 0.085 2.861 0.004 0.244 0.244
f2 ~~
f3 0.300 0.077 3.904 0.000 0.300 0.300
f4 0.002 0.068 0.036 0.971 0.002 0.002
f5 0.205 0.085 2.409 0.016 0.205 0.205
f6 0.177 0.088 2.012 0.044 0.177 0.177
f3 ~~
f4 0.097 0.071 1.369 0.171 0.097 0.097
f5 0.433 0.067 6.490 0.000 0.433 0.433
f6 0.711 0.048 14.976 0.000 0.711 0.711
f4 ~~
f5 0.262 0.074 3.537 0.000 0.262 0.262
f6 0.264 0.074 3.583 0.000 0.264 0.264
f5 ~~
f6 0.681 0.058 11.657 0.000 0.681 0.681
Intercepts:
Estimate Std.Err z-value P(>|z|) Std.lv Std.all
.fb1 2.628 0.065 40.726 0.000 2.628 2.249
.fb2 3.497 0.054 64.223 0.000 3.497 3.546
.fb3 3.314 0.065 50.901 0.000 3.314 2.811
.fb14 3.869 0.054 71.807 0.000 3.869 3.965
.fb9 1.774 0.059 30.321 0.000 1.774 1.674
.fb4 1.649 0.058 28.602 0.000 1.649 1.579
.fb6 3.210 0.063 50.792 0.000 3.210 2.805
.fb7 2.814 0.068 41.563 0.000 2.814 2.295
.fb12 3.308 0.062 53.250 0.000 3.308 2.940
.fb13 3.271 0.061 53.645 0.000 3.271 2.962
.fb15 2.802 0.065 42.871 0.000 2.802 2.367
.fb16 3.055 0.062 48.993 0.000 3.055 2.705
.fb21 2.662 0.065 40.935 0.000 2.662 2.260
.fb18 2.741 0.066 41.450 0.000 2.741 2.289
.fb19 3.421 0.061 56.444 0.000 3.421 3.117
.fb20 3.192 0.064 49.787 0.000 3.192 2.749
.fb22 2.994 0.060 49.650 0.000 2.994 2.741
.fb23 3.497 0.062 56.016 0.000 3.497 3.093
f1 0.000 0.000 0.000
f2 0.000 0.000 0.000
f3 0.000 0.000 0.000
f4 0.000 0.000 0.000
f5 0.000 0.000 0.000
f6 0.000 0.000 0.000
Variances:
Estimate Std.Err z-value P(>|z|) Std.lv Std.all
.fb1 1.056 0.096 11.050 0.000 1.056 0.773
.fb2 0.773 0.082 9.421 0.000 0.773 0.795
.fb3 0.769 0.107 7.165 0.000 0.769 0.553
.fb14 0.688 0.080 8.649 0.000 0.688 0.722
.fb9 0.339 0.265 1.278 0.201 0.339 0.302
.fb4 0.674 0.165 4.071 0.000 0.674 0.618
.fb6 0.492 0.076 6.493 0.000 0.492 0.375
.fb7 0.476 0.086 5.553 0.000 0.476 0.317
.fb12 0.919 0.111 8.298 0.000 0.919 0.726
.fb13 -0.094 0.260 -0.363 0.717 -0.094 -0.077
.fb15 0.628 0.080 7.819 0.000 0.628 0.448
.fb16 0.525 0.073 7.211 0.000 0.525 0.412
.fb21 0.750 0.090 8.330 0.000 0.750 0.541
.fb18 0.744 0.071 10.503 0.000 0.744 0.519
.fb19 0.625 0.062 10.119 0.000 0.625 0.519
.fb20 0.732 0.067 10.985 0.000 0.732 0.543
.fb22 0.729 0.070 10.346 0.000 0.729 0.611
.fb23 0.935 0.079 11.804 0.000 0.935 0.732
f1 1.000 1.000 1.000
f2 1.000 1.000 1.000
f3 1.000 1.000 1.000
f4 1.000 1.000 1.000
f5 1.000 1.000 1.000
f6 1.000 1.000 1.000
Recommendations
En la presente investigación se estudia el afecto generado por las valoraciones cognitivas de los trabajadores sobre los eventos laborales. En esta relación se evalúan dos moderadores, la importancia atribuida a los eventos y el estilo explicativo del trabajador. Setenta y tres trabajadores contestaron un cuestionario de estilos explicativos (nivel...
Emotional labor is ubiquitous in service work, but little is known about what enables service employees to use desirable strategies such as deep acting. Applying conservation of resources theory, we hypothesized that being recovered is a crucial resource for deep acting, especially for employees with low customer orientation and low positive affect...
