Content uploaded by John G O'Neill
Author content
All content in this area was uploaded by John G O'Neill on Mar 30, 2017
Content may be subject to copyright.
1
“CRITIC AND CONSCIENCE OF SOCIETY: A REPLY TO JOHN HATTIE
1
Following the release of John Hattie’s book, Visible Learning (2009), we wrote a
paper which was published in this journal (Snook et al, 2009). We began: “The
announcement of the book has already led to a good deal of discussion both in New
Zealand and overseas and seems to have captured the attention of policy makers. It is,
therefore, important that members of the educational research community pay John
Hattie the courtesy of subjecting his conclusions to critical scrutiny in a spirit of
mutual truth seeking to ensure that: (1) discussions are based on a careful reading of
the book, rather than on half baked ‘reactions’ in the popular media; (2) the caveats
which Hattie himself sets out are carefully noted so that decisions are not made in
opposition to the message of this book and (3) the findings are not “appropriated” by
political and ideological interests and used in ways which the data do not substantiate.
We were, then, rather surprised that Hattie chose to include us in a reply to “his
critics” rather than examining our paper in terms of its stated purposes. Our
“sympathetic critique” was lumped in with the criticisms of many others, some
named, many unnamed, who in one way or another have attacked his writings, his
public statements and his personal motivation. Though we are sometimes named and
sections of our paper cited, it is not always easy to determine which of his criticisms
are directed at us and this is unusual in academic debate. It is particularly ironic that
the force of his reply is that his critics do not understand how difficult it is to be both
a researcher and a contributor to educational policy. We would point out, with
respect, that all of us have been involved in both sets of activities, in some cases for
more than 40 years. The role of “critic and conscience of society” is something that
we are very familiar with and, like Hattie, we have faced strong criticism from those
who do not agree with us. We too have found that it is common “for the person to be
subject to scrutiny, misrepresentations and even personal attacks.” (Hattie, 2010,
p85). To that extent, we understand how he feels.
A careful re-reading of our original article will reveal that we concentrated on his
ideas as expressed in his book and made no reference to his role in public debates.
There is, in our view, a pressing need in New Zealand for university academics to
openly debate the differences between intellectuals (Said, 1994) and policy
entrepreneurs (Mintrom, 1997). Indeed, in a small democracy like New Zealand
where funding for educational research is almost exclusively focused on Ministry
policy implementation or evaluation (OECD, 2001), it is a moot point whether it is
possible for public academics to be both inside and outside the system (Hattie, 2010,
p. 93). However, that debate is for another occasion.
We made no ‘personal attacks” and we do not deserve to be grouped with those who
“uncritically respond through new media such as blogs” (Hattie, 2010, p. 85) or with
those who “had not read the book (some even applauded that they had not read it.)”
(Hattie 2010, p.87). We read the book carefully and, we believe, presented a balanced
commentary. As he suggests “critique of ideas...is the essence of academia and can
1
Amended, accepted version by I. Snook et al published in New Zealand Journal of
Educational Studies; 2010, Vol. 45 Issue 2, p. 93
2
lead to debate which is more informed.” (Hattie, 2010, p. 87). We hoped to initiate
debate about his ideas in the pages of an academic journal and are disappointed that
he appears to have ignored one of his own “fundamentals of academic critique” and
focused on the personal rather than the ideas. He has declined to engage fully with us
despite the fact that our account was presented with the “courtesy and goodwill”
which he believes should characterise academic debate.
Despite the difficulty of determining which parts of his reply are directed at us, it is
possible to detect some of the concerns which relate to our analysis either directly (as
when we are cited) or indirectly (when he criticises points which we did make even if
he does not sheet them home to us.)
Hattie says:“Among the most common of criticisms of Visible Learning is
the place of non-school factors.” (Hattie, 2010, p 89). His response to such
criticisms is that he is well aware of them but preferred to ignore them and
to concentrate on what can be done in schools. We acknowledged this to
be his position and indeed quoted the very line which he quotes back at us:
We pointed out that he downplayed non-school factors “NOT because
they are unimportant, indeed they may be more important than many of
the issues studied in this book. It is just that I have not included these
topics in my orbit.” ( Hattie 2010, p. 89; Snook at al, 2009, p. 94-95), but
we went on to say “It is easy for those seeking to make policy decisions to
forget this significant qualification” (Snook et al, 2009. p. 98). We then
cited research which indicates that home background is indeed the most
important variable affecting school achievement and that Hattie’s own
work testifies to this.
Hattie argues that his comment about not writing a book about ‘criticism of
research’ “seems to have been misinterpreted as my paying no attention to
the quality of the research.” (Hattie, 2010, p. 88). He says that where
research quality relating to specific influences has been shown to make a
difference, this has been cited and where meta-analyses have been poor
this has been pointed out. However (1) In our original article, we carefully
noted that there are occasions on which he comments on the quality of
research, most notably in relation to ‘learning styles’ where we say that he
is “justifiably suspicious of the motives behind much of research and
appropriately sceptical about the results. (Snook et al, p.95) and we state
“His own approach acknowledged that a simple effect size analysis cannot
be maintained, when there are questions about the underlying quality of
some of the studies he includes.” (Snook et al, p. 95). (2) His original
disclaimer is actually much stronger than he acknowledges in his latest
paper: “I have deliberately not included much about moderators of research
findings based on research attributes (quality of study, nature of design)
again not because they are unimportant....but because they have been dealt
with elsewhere by others.” (Hattie, 2009, p. ix). We submit that we were
right to state that, despite some exceptions, in the main “he does not
subject the various studies to critical analysis” (Snook et al, 2009, p. 95)
and we note that he himself raised this as a problem with his analysis.
He states: “Snook at al (2009) claimed that I did not acknowledge that
‘bias is not normally controlled in meta-analysis’ but they failed to
acknowledge the rich literature on this topic which shows how controlling
3
bias is a major concern of this method” (Hattie, 2010, p. 88) . But the only
way in which Hattie defends his work against this charge is by repeating
our own point that “the studies are heavily weighted to published studies”
which (because of peer review) is one way of avoiding bias. But has been
often pointed out, this very method can also lead to bias in that studies
which do not show marked effects (or effects in the “wrong direction”)
frequently do not make it into publication. The possibility of bias is not
negated by talk of “the rich literature” on the topic.
He goes on to say that our claim that he ignored “the many potential
influences (such as quality, year of schooling) on the overall conclusion is
not correct, given I was careful to note these effects whenever they
occurred.” (Hattie, 2010, p. 88) But we did not claim that he ignored these
variables. On the contrary we acknowledged (eg in relation to homework)
that Hattie presents the data with all the relevant nuances (age of student,
ability of student, subject studied etc) but, in the end, he produces an
average effect size, (in this case .29) which means that a signal is sent out
to the profession (and certainly to media commentators) that “homework is
not important” even though his own data reveal that homework is very
significant for students in high schools, particularly low ability students.
Here it is glaringly obvious that a ban on homework might cause
substantial harm to many students. This reinforces our advice to be very
wary about deducing policies from meta-analyses.
He argues that “Snook et al claimed that New Zealand may be so different
from the rest of the Western world that we ‘should exercise great care in
relating the meta -analyses to New Zealand education.’ ”. (Hattie, 2010,
p.89). Once again, it was Hattie himself who drew attention to the problem
of generalising from largely USA sources to other different education
systems. We simply noted that “It could easily be that New Zealand
schools, teachers students, and curricula are sufficiently different from
those of the USA that the meta-analyses may not fully apply in this
country,” (Snook et al, 2009. pp. 97-98) This is a very modest and
reasonable claim
He says that we accused him of adopting an “arbitrary” cut off point. He
agrees that it is arbitrary but that it is not capricious. We did not suggest
that it was; we simply made the point that for many purposes a lower cut
off can be justified and went on to suggest that “Much depends on the
quality of the research studies in the various meta-analyses. If the sample is
large and random.... a ‘small’ effect size is of considerable significance.
On the other hand, large effect sizes from small samples are meaningless at
best and positively dangerous when lumped together with other studies to
produce an “average.” ( Snook et al, 2009, p.94).) We do acknowledge,
however, that our point about the .17 effect size from Outward Bound
programmes was based on a misunderstanding which we regret.
Hattie says: “Snook et al...claim that I am more concerned with surface
level achievement,” (Hattie, 2010, p.90). We did not say (or imply) that
Hattie is more concerned with surface level achievement. On the contrary
we noted that “conceptual knowing or understanding is what [Hattie]
thinks should be the result of good teaching.” (Snook et al, p. 95). Rather
we repeated his own reservation that so much of the research is based on
surface level achievement and concluded that “there is less to be drawn
4
from his synthesis than commentators have suggested.” (Snook et al, p.
85). Ours was a warning for policy makers and practitioners, not a claim
about Hattie’s position.
In relation to our discussion of class size, he accuses us of being ‘selective”
in choosing our studies and again defends the merits of focusing on
averages: “the major advantage of meta-analysis is that extreme findings
are moderated by a larger sample of studies.” (Hattie, 2010, p. 90). It
seems that we here have a real difference between us; we recognize the
usefulness of met- analyses but are extremely sceptical of too much
reliance on them in making practical judgments. The two cases we cited as
showing favourable results from lowered class sizes were (1) large-scale
(2) well designed (one involved a random design) and (3) carried out over
several years. We are not sure where Hattie would stand, but we put more
faith in such studies than in “averages” from studies many of which may
be of poor quality. In the Murname and Levy (1996) study, the authors
make the point that if the results from all schools had been averaged there
would have been “a small negative” result” but the two schools which
make other changes in addition to class size experienced striking results.
Such points may indeed be “well rehearsed” in the literature on meta-
analyses but, to our minds, they are compelling. It is, however, worth
noting that we and Hattie are in total agreement on one matter: smaller
classes allow but do not ensure more effective teaching.
We are rather surprised that Hattie did not comment on what is a (perhaps
the) major problem in using meta-analysis in relation to educational
research. The technique had its origin in medical science with its carefully
defined concepts and tight (usually experimental) research design. In
education, however, the variables being studied are often poorly
conceptualised and the studies often far from rigorous. How does one
clearly distinguish for research purposes between a classroom that is
teacher centred and one which is student centred and, in researching the
topic, how can one control all the variables in a noisy and busy classroom
with perhaps more than 30 participants? (Snook et al, p. 97). It is
interesting that after forty years of close-grained research in classrooms,
Graham Nuthall concluded that “although there are many published reports
of educational research, you need to be very selective in identifying those
that have something trustworthy to say about classroom teaching and
learning.” Nuthall, 2007, p16). In this connection it is interesting to
juxtapose the title of John Hattie’s book (Visible Learning) with that of
Graham Nuthall (The Hidden Lives of Learners).
Conclusion:
We would like to repeat some of the conclusions we came to at the end or our
original paper:
We believe that John Hattie’s book makes a significant contribution to understanding
the variables surrounding successful teaching and think that it is a very useful
resource for teacher education. We are concerned, however, that: (1)
despite his own frequent warnings, politicians may use his work to justify policies
which he does not endorse and his research does not sanction; (2) teachers and teacher
5
educators might try to use the findings in a simplistic way. Hattie, following Dewey
holds that “Evidence does not supply us with rules for action but only with hypotheses
for intelligent problem solving, and for making inquiries about our ends in education.”
(Hattie, 2009, p. 247). His findings cannot be directly applied to educational policy or
practice. (3) the quantitative research on “school effects” might be presented in
isolation from its historical, cultural and social contexts, and their interaction with
home and community backgrounds; (4) there may be insufficient discussion about the
aims of education and the purposes of schooling without which the studies have little
point.
“It is important that students preparing for teaching learn about the research process
and how easily it leads to error rather than truth. They need to respect research but be
acutely aware of its limitations. The research that they need to know about goes
beyond what happens in schools and classrooms. As this review has shown, what
students bring from their social class, family, culture, home background and prior
experiences is more important than what happens in the school, even though what
happens in the school (particularly what teachers are and do) is very important. The
secret of school improvement lies in the recognition of these factors and their
integration into a social, economic and educational programme.
References
Hattie, J. (2008). Visible learning: a synthesis of over 800 meta-analyses relating to
achievement. London: Routledge.
Hattie, J. (2010). “On Being a “Critic and Conscience of Society’: the Role of the
Education Academic in Public Debates.” New Zealand Journal of Educational Studies
45: 1, pp 85-96.
Mintrom, M. (1997). Policy entrepreneurs and the diffusion of innovation. American
Journal of Political Sciences, 41(3), 738-770.
Nuthall, G (2007) The hidden lives of learners. Wellington: NZCER Press.
Organization for Economic Cooperation and Development (2001, September). OECD
Review: Educational research and development policy in New Zealand: Examiners’
report. CERI/CD (2001)4. Paris: OECD.
Said, E. (1994). Representations of the intellectual. New York: Pantheon.
Snook, I., Clark., J, Harker R., O’Neill A-M., O’Neill, J. (2009). “Invisible
Learnings”: a commentary on John Hattie’s book visible learning: a synthesis of over
800 meta-analyses relating to achievement.” New Zealand Journal of Educational
Studies, 44:1, pp 93-106.
