ArticlePDF Available

Supporting 'quality' in novice teacher research: Data collection and data analysis

Authors:

Abstract

Lankshear, C. and Knobel, M. (2018). Education and “new literacies” in the middle years. Literacy Learning: The Middle Years, 26 (2): 7-16.
ETAS Journal 35/3 Summer 2018
45
Supporting ‘quality’
in novice teacher
research: Data
collection and
data analysis
COLIN LANKSHEAR A N D
MICHELE KNOBEL
Introduction
This article describes our approach to
supporting teachers enrolled in a Masters
of Literacy program in acquiring some
basics of becoming competent (novice)
producers of qualitative research in the
area of ‘new literacies’. Participants
typically have had no prior experience with
conducting research. The course is based
on a quite formal academic view of
qualitative research as a systematic,
methodical, coherent, rigorous, and
reflective process of inquiry and reporting
that takes the form of a sustained
argument – where the research findings
comprise the conclusions to the argument.
Research, in this sense, aims to generate
robust and defensible claims to knowledge.
This means there has to be a coherent
chain of reasoning that runs all the way
from the research purpose and questions
to the research findings; via appropriate
and adequate data collection and data
analysis processes that are informed by
relevant theory and concepts, and
subsequent discussion of
analytic results that is
informed by wider work in
the field as well as by the
study’s aims and
objectives. For the
research conclusions to
be taken seriously, the
reader/audience must be
satisfied that the reported
research provides sufficient
grounds for believing in the
quality of the research process.
This does not necessarily entail that a
reader/audience accept the conclusions as
compelling, but they must be able to see and
understand those points at which they agree
or disagree, and the extent to which claims
are warranted at each point in the argument.
In what follows, we indicate some key
features of the course structure and
process and provide some examples from
participants’ work that illustrate how they
respond to the challenge of ensuring
quality at the level of data collection and
data analysis.
The course approach
This Masters of Literacy course requires
participants to create a popular culture
digital media artefact of a kind they have
never created before (e.g. stop motion
animation, movie trailer remix); document
this using a range of data collection
methods and strategies; analyse their
data using some basic but powerful data
analysis methods; write a formal research
report of 5000-7000 words; and at the
end of the course make a formal
30-minute presentation under formal
conference conditions to their peers.
We have them study their own learning
and ‘making’ in order to avoid needing
formal ethical clearance from the
university to conduct research.
The fundamental key to the course is that
it is team-based, collaborative work.
This, of course, reflects the circumstances
of mature real-world research activity where
‘serious’ research is undertaken by teams,
rather than by individuals. From the outset,
participants are working together, bringing
their respective strengths – their reading
and understanding of the course resources
and the formal instruction sessions, their
digital skills and internet savvy, their
popular culture interests and knowledge,
and anything they might already believe
about qualitative research – to a common
cause. They are welcome to interact with
other teams, too; anything to help them
produce the best research result possible.
The course operates under two different
modes: an intensive mode of ten days
face-to-face work and ten of home-based
reading (before and between the
face-to-face meetings); and a semester-long
version with three face-to-face sessions,
and a month of home-based work
between sessions.
Using online resources is
likewise fundamental.
Participants access a
dedicated Google site a
month prior to the course
that hosts their readings
and other supporting
resources (e.g. lists of
media production options,
exemplar papers). Three main
online resources are introduced
in the first session: Google Docs,
the Google Scholar search engine
(including setting library preferences), and
their university’s online library archives.
Setting Google Scholar library preferences
to include their university’s online library
facilitates finding full text academic
materials. Each team also creates a Google
Doc and invites us to each document as
‘critical friends’. This role is grounded in
IN ENGLISH LANGUAGE TEACHING
Burns, A. (2005b). Action research: An evolving
paradigm? Language Teaching, 38(2), 57-74.
Burns, A. (2009). Action research in second
language teacher education. In A. Burns &
J. C. Richards (Eds.), The Cambridge guide
to research in language teaching and learning
(pp. 289-297). Cambridge, UK: Cambridge
University Press.
Burns, A. (2010). Doing action research in English
language teaching: A guide for practitioners.
London, UK & New York, NY: Routledge.
Cochran-Smith, M., & Lytle, S. L. (1999). The teacher
research movement: A decade later. Educational
Researcher, 28(7), 15-25.
Dikilitas, K., Smith, R., & Trotman, W. (2015).
Teacher-researchers in action. Faversham,
UK: IATEFL.
Dixit, K., Joshi, V., & Mane, M. (2016). Exploring
learners and learning of English. Nagpur,
India: AINET.
Edge, J., & Richards, K. (1994). Teachers develop
teachers research: Papers on classroom research
and teacher development. Portsmouth,
NH: Heinemann.
Kemmis, S., & McTaggart, R. (1988). The action
research planner. Waurn Ponds, Australia: Deakin
University Press.
McIntyre, A. (2008). Participatory action research.
London, UK: Sage.
Norton, L. (2009). Action research in teaching and
learning: A practical guide to conducting
pedagogical research in universities. London,
UK & New York, NY: Routledge.
Padwad, A., & Dixit, K. (2018). Developing as
researchers: Promoting teachers’ CPD through
engagement with research. In B. Tomlinson (Ed.).
Explorations: Teaching and learning English in India.
Kolkata, India: British Council.
Rebolledo, P., Smith, R., & Bullock, D. (2016).
Champion teachers: Stories of exploratory action
research. London, UK: British Council.
Roulston, K., Legette, R., Deloach, M., & Buckhalter
Pittman, C. (2005). What is research for teacher-
researchers? Educational Action Research, 13(2),
169-190.
Sagor, R. (2000). Guiding school improvement with
action research. Alexandria, VA: ASCD.
Schön, D. A. (1983). The reflective practitioner.
London, UK: Temple Smith.
Smith, R., Connelly, T., & Rebolledo, P. (2014).
Teacher research as continuing professional
development: A project with Chilean secondar y
school teachers. In D. Hayes (Ed.), Innovations in
the continuing professional development of English
language teachers (pp. 111-129). London, UK:
British Council.
About the Author
Amol Padwad lives and works in central India,
where he teaches tertiary-level English language
and literature courses. He has been working in
the areas of teacher development, CPD, ESL,
teacher networking, and teacher motivation for
over 20 years, with a recently added interest in
ELT history in India. Besides a long-standing
association with national and international
English teacher associations, he is also an ELT
consultant, trainer, and course designer.
amolpadwad@gmail.com
J. M. Patel College
For the research
conclusions to be
taken seriously, the
reader/audience must be
satisfied that the reported
research provides sufficient
grounds for believing in
the quality of the
research process.
ETAS Journal 35/3 Summer 2018
46
qualitative research traditions where such
friends contribute to the quality of a study by
asking key questions about methods,
processes, and interpretations that help
researchers gain some critical distance from
their work and to shore up the explicitness
and justification of their methods, processes
and claims (Merriam & Tisdell, 2015).
One key operating principle for everything done
in the course is “just in time and just in place,”
which is facilitated by Google tools. Feedback
and suggestions, additional resources, all
manner of sharing and updating, and so on,
can be made available on request at point of
need. As such, we do very little ‘lecturing’ on
‘how to do research’ in this course and work
directly with groups as their study unfolds. In
short, we set up a kind of ‘sandbox’ for
learning to become qualitative researchers,
with a particular eye to maximizing quality in
research. A ‘sandbox’ suggests a safe,
give-things-a-go space in which to engage
in activity within parameters set by the
affordances of the available resources
(Gee, 2007).
As indicated above, we provide an initial ‘kit’
of resources to guide their work. This includes
setting a research purpose in the form of an
in-common research question: e.g. “To what
extent and in what ways might your media
creating activity be understood in terms of
engaging in a new literacy practice?”; or “What
are some salient features of how you learned
to create a piece of digital media?” This ‘kit’
also includes some broad theoretical framing
for their research learning: e.g. a version of
social learning theory (e.g. Brown & Adler,
2008); a sociocultural conception of social
practices (e.g. Reckwitz, 2002); concepts
associated with new literacies (e.g. Black,
2009), etc. In addition, focused resources
pertaining to how to collect written, spoken,
and observed data are provided
(e.g. Lankshear & Knobel, 2004), together
with introductions to common forms of data
analysis like coding and category development
(Saldaña, 2016) and pattern matching
(Fetterman, 2009). In addition, participants
are told that “Google is your friend,” and at
every point they are encouraged to identify
additional resources to assist in all aspects
of the work, and to share these.
Gee (2007) makes the point about the limited
use of computing and games manuals for
learning how to ‘drive’ a computer or play the
game—he argues that these manuals make
most sense once the game has been played.
This principle applies to the direct instruction
we provide in this course. We explain that the
‘metalevel’ concepts and knowledge involved
in research only become clear when they are
experienced in sustained, material activity.
We aim to introduce technical/specialist
concepts as close as possible to their point
of application in practice, and as concretely
as possible, under an overarching idea of
research as a systematic and methodical
practice that aims to generate a quality
result (Lankshear & Knobel, 2004;
Lankshear & Knobel, 2011).
Focus on quality in data collection, data analysis,
and reporting research
Qualitative researchers have developed concepts and criteria designed to shore
up confidence in the quality of the research process and the claimed results of research.
They do not aim for ‘truth’ or ‘proof’ or approximations to ‘certainty’, so much as for
ideals like communicative validity, trustwor thiness, and defensible interpretation.
(a) Data collection and reporting
At the level of data collection this means showing how the collected data is appropriate
to and sufficient for addressing the research purposes, and how this was decided. We
provide participants with simple tools like a design-focused matrix (see Table 1) to help
with study planning.
Table 1: Data collection design matrix
Interviews, observations, relevant artefacts, and in situ ‘talk throughs’ of what is being
done in the moment generate data that can be cross-referenced to create
multidimensional accounts of what happened. Having multiple team members collecting
data at the same time, while others are working on the digital production – with
participants moving fluidly in and out of these roles – heightens confidence in data
quality, too.
In explicit instruction we emphasise the importance of recording speech and action
digitally. In the following examples, a participant used his phone to record speech, and
when the speech was too soft to record, he (re)spoke the exact words in situ and named
the speaker. He also spoke brief observations of the context and event into the recording
and noted the time of activity. These notes were included in his transcription (see Table 2).
Table 2: In situ talk (collecting spoken data)
This team’s fieldnotes documented written data resources (such as URLs) used in
learning-on-the-go how to create stop motion animation (see Table 3).
Table 3: Textual sources documented in situ (collecting written data)
It is worth noting that this data was collected within the first three hours of these
participants’ introduction to qualitative research practices.
We likewise emphasise the importance of making full and richly descriptive fieldnotes
(see Table 4). We provide readings on and examples of how to write detailed fieldnotes,
which include identifying time and location (e.g., to triangulate against another
participant’s fieldnotes of the same events/episodes/activity).
Research literacy Part 3:
S U P P O R T I N G T E A C H E R R E S E A R C H
SPECIAL
SUPPLEMENT
Means of
data collection
What kinds of
tools and methods
will provide the
appropriate data?
Kinds of data
Data needed for
addressing the
research purpose
Justification for data
to be collected
How does this kind
of data address the
research purpose?
Sources to inform and
guide data collection
Who shows how to
collect these kinds
of data well?
Date and time
11:30
Transcription
[S and A and R analysing video]
S - They caught one guy?
R - Did they dig it up?
A - They have love letters?
R - Look at the fire
[They watch the video three times and have a discussion]
A - Should we have the same script?
K - Let’s use Batman and Robin
A - Ya, Lego people
A - Bert and Ernie?
N - Ya, thats a great idea
S - Lego people?
R - I have lots of Lego
Time
11.55
Observations
K suggested to look up how many photos it is required to make
one minute of video. R google searched the key words “number of
pics for 1 sec stop motion.” R reviewed the website:
http://stopmotionexplosion.com/blogs/blog/6394410-6-stop-
motion-beginner-mistakes-and-how-to-avoid-them
She discovered that 12-24 “frames/sec is ideal” and shared the
information with the group.
ETAS Journal 35/3 Summer 2018
47
Table 4: Observation fieldnotes
Participants are required to produce a written research report, and this includes
detailing and justifying their data collection activity in ways that are accurate, honest,
and well-informed (see Table 5).
Table 5: Overview statement of data collection approach from a final report
(b) Data analysis and reporting
We introduce course participants to pattern matching and a basic coding process that
focuses on using in vivo and descriptive codes to develop categories and themes. For
example, the observation fieldnotes presented in Table 4 above were typical of a large
subset of fieldnotes coded by this team. Within the large set of data items initially coded
as “problem solving,” participants subsequently identified five sub-patterns of problem
solving behaviour: “Problem solving through messing about”; “Problem solving by using an
affinity space”; “Problem solving through group collaboration”; “Problem solving through
using a search engine”; and “Problem solving by other means” (for occasional, less
frequent types, such as consulting a book, emailing someone, asking children at home).
Participants were asked to devise a system such that they could subsequently ‘point to’
the location of data samples they used in their report. So, for example, in Table 4, Ann
notes that her fieldnotes organized as “7:31, 32, 33, and 34” are all instances of what
they categorized as “Problem solving through messing about.” This provides a kind of
‘audit trail’ for readers, saying in effect, “if you would like to check out our analysis to
see if you agree or not, these are the places in my fieldnotes where you could evaluate
our coding and subsequent category development.”
Our final example comprises a group describing their data analysis approach in their
research report (see Table 6). Their text recounts how they set about finding patterns
in their data, and subsequently used concepts associated with a sociocultural
understanding of ‘new literacies’ to assign patterns to pre-determined categories;
an analytic strategy that coheres perfectly with the kind of theory-guided research
they were learning to engage in.
Table 6: A group describing their data analysis approach
Needless to say, the final reports produced have
limitations. We do provide models of good quality
papers from previous classes and discuss academic
report writing conventions with them. However, time
is short, and prior experience practically non-existent.
Experienced researchers would likely disagree with
many of the analytic results claimed, and with the
level of sophistication of demonstrating data analysis.
We would be surprised, however, if they found serious
shortcomings in the quality of data collected by teams
like those represented in our examples, or in their
initial appreciation of what it means to aim for quality
in their research.
References
Beck, S., Coley, C., Conway, K., Hoven, D., & Maynard, P. (2009).
From collaboration to affinity space: Learning a new literacy
(Unpublished manuscript). Steady Brook, Newfoundland, Canada:
Research Report for Summer Program.
Black, R. (2009). Adolescents, fan communities, and twenty-first
century skills. Journal of Adolescent & Adult Literacy, 52(8), 688-697.
Booth, W., Colomb, G., Williams, J., Bizup, J., & Fitzgerald, W. (2016).
The craft of research (4th ed.). Chicago, IL: Chicago University Press.
Brown, J. S., & Adler, R. P. (2008). Minds on fire.
Educause Review, 43(1), 16-32.
Donovan, S., Hawley, J., & Whitty, S. (2010). Social learning resulting
from the production of a media artefact (Unpublished manuscript).
North Sydney, Nova Scotia, Canada: Research Report for
Summer Program.
Gee, J. (2007). What video games have to teach us about learning
and literacy. New York, NY: St Martin’s.
Fetterman, D. (2009). Ethnography: Step by step (3rd ed.). Newbury
Park, CA: Sage.
Lankshear, C., & Knobel, M. (2004). Handbook for teacher research.
Maidenhead, UK: Open University Press.
Lankshear, C., & Knobel, M. (2011). New literacies:
Everyday practices and social learning (3rd ed.). Maidenhead,
UK: Open University Press/McGraw-Hill.
Merriam, S., & Tisdell, E. (2015). Qualitative research: A guide
to design and implementation. San Francisco, CA: Jossey Bass.
Reckwitz, A. (2002). Toward a theory of social practices:
A development in social theorizing. European Journal of
Social Theory, 5(2), 245-265.
Saldaña, J. (2016). The coding manual for qualitative researchers
(3rd ed.). Thousand Oaks, CA: Sage.
About the Authors
Colin Lankshear and Michele Knobel jointly edit the New
Literacies book series for Peter Lang USA. Their collaborative
publications include Researching New Literacies (2017), A New
Literacies Reader (2013), New Literacies: Everyday Practices
and Social Learning (2011), and A Handbook for Teacher
Research (2004). Michele has also co-edited, with Judy
Kalman, New Literacies and Teacher Learning (2016).
Colin Lankshear is an educational researcher and writer who lives
in Mexico and is an adjunct professor at Mount St Vincent
University in Canada. His early research focused on the philosophy
of education before he encountered Paulo Freire’s work and
developed an interest in the sociocultural study of literacy. His
main research interests today are in the area of new literacies.
c.lankshear@yahoo.com
Mount St Vincent University
Michele Knobel is a Professor in the Department of Early
Childhood, Elementary, and Literacy Education at Montclair
State University, USA. She has worked in teacher education in
Australia, Canada, Mexico, and the US. Her research focuses
on people’s everyday literacy practices, especially in relation
to digital technologies.
knobelm@montclair.edu
Montclair State University
IN ENGLISH LANGUAGE TEACHING
5 July
11.15
I got back into YouTube.com to search for Haiti, started with search
‘President Obama on Haiti’. Parts of the speech I think we can use,
so I copy and paste the url into www.zamzar.com. I got error message
‘File has no extension’. I asked H and J if they got this error.
They didn’t, so I went back to YouTube.com. I repeated the process,
opening up a new Explorer window and it worked (code: “problem
solving through messing about”; also see Ann: 7:31, 32, 33, & 34).
Data were collected by three participants over five days … resulting in a total of 30
hours of observations, field notes, head notes, post facto notes taken by team
members individually at differing times. Field notes documented specific written details
of what was occurring, what was being created, and the emotions group members had
felt. Field notes involved writing descriptively, marking the time down regularly, and
developing a shorthand language. [When] it was difficult to write an observation down
as events were taking place, team members made ‘head notes’ and would write details
as post facto notes … as soon as possible after the observed event… Group members
copied and pasted URLs of websites that were useful sources of information and took
screenshots of those websites (text from Donovan, Hawley, & Whitty, 2010, p. 12).
The primary [analytic] method used was pattern finding … [where] the researcher
begins with a “mass of undifferentiated ideas and behaviour, and then collects bits
of information, comparing, contrasting, and sorting gross categories and minutiae”
(Fetterman, 2009, p. 92) until discernible patterns emerge. We used four questions
[derived from Fetterman] to guide our analysis of data:
1. What’s going on here?
2. Who is doing what?
3. Have I seen this particular event or action before? Is it significant? Why or why not?
4. What things are happening or being done more than once? What does this mean
or suggest?
Applying these questions to our data … we began to notice similarities in what we
had collected. We colour coded and identified patterns as they emerged from the data.
[This] was guided by our review of the literature; that is, we expected to find patterns …
that corresponded to key characteristics of what constitute new literacies
(e.g., participation, collaboration, distributed knowledge). At the same time, we
remained open to finding additional insights into what constitutes a new literacy
practice in our data as well (Beck, Coley, Conway, Hoven, & Maynard, 2009, pp. 11-12).
ResearchGate has not been able to resolve any citations for this publication.
Article
The abstract for this document is available on CSA Illumina.To view the Abstract, click the Abstract button above the document title.
From collaboration to affinity space: Learning a new literacy (Unpublished manuscript)
  • S Beck
  • C Coley
  • K Conway
  • D Hoven
  • P Maynard
Beck, S., Coley, C., Conway, K., Hoven, D., & Maynard, P. (2009). From collaboration to affinity space: Learning a new literacy (Unpublished manuscript). Steady Brook, Newfoundland, Canada: Research Report for Summer Program.
Adolescents, fan communities, and twenty-first century skills
  • R Black
Black, R. (2009). Adolescents, fan communities, and twenty-first century skills. Journal of Adolescent & Adult Literacy, 52(8), 688-697.
  • W Booth
  • G Colomb
  • J Williams
  • J Bizup
  • W Fitzgerald
Booth, W., Colomb, G., Williams, J., Bizup, J., & Fitzgerald, W. (2016). The craft of research (4th ed.). Chicago, IL: Chicago University Press. Brown, J. S., & Adler, R. P. (2008). Minds on fire. Educause Review, 43(1), 16-32.
Social learning resulting from the production of a media artefact (Unpublished manuscript)
  • S Donovan
  • J Hawley
  • S Whitty
Donovan, S., Hawley, J., & Whitty, S. (2010). Social learning resulting from the production of a media artefact (Unpublished manuscript). North Sydney, Nova Scotia, Canada: Research Report for Summer Program.
Handbook for teacher research
  • C Lankshear
  • M Knobel
Lankshear, C., & Knobel, M. (2004). Handbook for teacher research. Maidenhead, UK: Open University Press.
Qualitative research: A guide to design and implementation
  • S Merriam
  • E Tisdell
Merriam, S., & Tisdell, E. (2015). Qualitative research: A guide to design and implementation. San Francisco, CA: Jossey Bass.
Thousand Oaks, CA: Sage. About the Authors Colin Lankshear and Michele Knobel jointly edit the New Literacies book series for Peter Lang USA
  • J Saldaña
Saldaña, J. (2016). The coding manual for qualitative researchers (3rd ed.). Thousand Oaks, CA: Sage. About the Authors Colin Lankshear and Michele Knobel jointly edit the New Literacies book series for Peter Lang USA. Their collaborative publications include Researching New Literacies (2017), A New Literacies Reader (2013), New Literacies: Everyday Practices and Social Learning (2011), and A Handbook for Teacher Research (2004). Michele has also co-edited, with Judy Kalman, New Literacies and Teacher Learning (2016).