ArticlePDF Available

Using Computer-based Text Analysis to Integrate Qualitative and Quantitative Methods in Research on Collaborative Learning

Authors:

Abstract and Figures

This paper argues that there are great potential benefits in incorporating computer-based text analyses into methods for researching talk and educational activity in classrooms. The first part of the paper discusses the strengths and weaknesses of some existing approaches to the study of talk and collaborative activity. The second part of the paper suggests ways that computer-based analysis of transcribed talk can integrate qualitative and quantitative methods, and in so doing overcome some of their respective weaknesses. This integrated approach is illustrated by a recent study of primary school children's talk and joint activity while working at the computer.
Content may be subject to copyright.
Using
Computer-based
Text
Analysis
to
Integrate
Qualitative
and
Quantitative
Methods
in
Research
on
Collaborative
Learning
Rupert
Wegerif
and
Neil
Mercer
Centre for Language and Communications, School of Education, Open
University, Milton Keynes, MK7 6AA, UK.
This paper argues that there are great potential benefits in incorporating computer-
based text analyses into methods for researching talk and educational activity in
classrooms. The first part of the paper discusses the strengths and weaknesses of some
existing approaches to the study of talk and collaborative activity. The second part of
the paper suggests ways that computer-based analysis of transcribed talk can integrate
qualitative and quantitative methods, and in so doing overcome some of their respec-
tive weaknesses. This integrated approach is illustrated by a recent study of primary
school children’s talk and joint activity while working at the computer.
Introduction
Computer-based text analysis is one of the fastest growing areas of linguistics
(Stubbs, 1996). Some have even described its impact on research in linguistics as
a ‘revolution’ (Baker
et al
., 1993). To date computer-based analysis has mainly
been applied to the study of large corpora of written texts. In this paper, we
propose that such methods are also of great value for research on children’s talk
and the joint construction of knowledge in the classroom.
The paper begins with a brief discussion of the strengths and weaknesses of
some commonly-used methods for analysing classroom talk. One such method
involves the use of coding schemes to analyse talk into functional categories.
Coding schemes enable researchers to deal with large amounts of data, and
normally use explicit, publicly-verifiable criteria to make categorisations. How-
ever, they are of limited value for exploring the construction of knowledge
because they are not sensitive to the ways that the content and context of talk
develops over time. We will refer to such methods, which depend upon coding
and quantifying data according to a pre-established set of categories, as
‘quantitative’. On the other hand, there are also well-established ‘qualitative
methods of discourse analysis which generally eschew the categorical classifica-
tion and quantification of data and rely essentially on the interpretative analysis
of transcribed speech. These qualitative methods of discourse analysis can be
sensitive to content and context, but have been criticised for the apparent reliance
of their exponents on the detailed interpretation of short excerpts selected from
larger, unseen speech corpora. We believe that the incorporation of computer-
based methods into the study of talk offers a way of combining the strengths of
quantitative and qualitative methods of discourse analysis while overcoming
some of their main weaknesses.
271
0950-0782/97/04 0271-16 $10.00/0 ©1997 R. Wegerif & N. Mercer
LANGUAGE AND EDUCATION Vol. 11, No. 4, 1997
We will illustrate our claims for the value of computer-based discourse
analysis by applying it to the study of talk amongst pupils in primary school
classrooms. For this study we used a software package specifically developed for
analysis of transcripts of talk which offers the researcher the possibility of moving
quickly between a detailed focus on a small section of transcript and an analysis
of the contexts of all occurrences of selected linguistic features throughout the
whole transcript or series of transcripts. In the research described, this software
was used to link a detailed analysis of the quality of children’s talk with
quantitative data of the same children’s scores on a test of reasoning. Features of
the talk of groups of children were related to their success in working together
to solve reasoning problems. In this way the old dichotomies of process and
product, quantitative and qualitative, were at least to some extent transcended.
The
Paradigm
Debate
in
Classroom
Discourse
Analysis
Coding
and
counting
There are methods for analysing classroom talk, which have developed from
the well-established methodological tradition commonly called ‘systematic
observation’ (Croll, 1986), in which talk data is reduced to coded categories which
are then statistically compared. The particular set of categories employed vary
according to the focus of the research study. Teasley (1995) offers a recent
example of this type of method, applied to the study of collaborative learning. In
Teasley’s study the talk of children working in pairs on a problem-solving task
was transcribed, and each utterance attributed to one of fourteen mutually
exclusive categories. These categories included such functions as ‘prediction’ and
‘hypothesis’. Transcripts were coded independently by two coders and the level
of agreement measured to ensure reliability. A count of categories of talk in
different groups was correlated with outcome measures on the problem solving
activity in order to draw conclusions about the kinds of utterances which
promote effective collaborative learning. There have been many other studies of
collaborative learning which have used some version of this coding approach to
analysing talk. King (1989), for example, used measures, such as length of
utterance, as well as pragmatic functional categories to investigate variables
affecting the success of collaborations. Kruger (1993) counted utterances consid-
ered indicative of ‘transactive reasoning’, and correlated their incidence with
measures of the success of children’s problem solving. Barbieri & Light (1992)
similarly measured the incidence of plans and explanations expressed in talk,
while Azmitia & Montgomery (1993) looked for talk features indicative of
scientific reasoning. And, drawing on the neo-Piagetian concept of ‘socio-cogni-
tive conflict(Doise & Mugny, 1984, and Perret-Clermont, 1980), Joiner (1993)
counted the number and type of disagreements in interactions and related these
to problem-solving outcome measures.
These and other studies using similar coding methods have produced
interesting and valuable results. Their strength, as opposed to more qualitative
methods discussed below, lies in their capacity to handle large corpora of data;
to offer explicit criteria for comprehensively categorising the whole of a data set;
to offer a basis for making systematic comparisons between the communicative
272 Language and Education
behaviour of groups of children and, similarly, to offer a basis for relating this
behaviour to measures of the outcomes of collaborative activity. However, critics
of such approaches can point to certain weaknesses and limitations, which we
will now summarise.
Critiques
of
coding
approaches
to
the
analysis
of
talk
Both the usefulness and the validity of coding methods have been questioned.
Edwards & Mercer point out (1987: 11) that coded analysis is often presented as
a
fait accompli
and that the prior interpretative movement that generated the codes
from the data is often obscured or forgotten. Draper & Anderson (1991) go further
to identify four specific kinds of problem:
(1) Utterances are often ambiguous in meaning, making coding difficult or
arbitrary.
(2) Utterances may have indeed often have multiple simultaneous
functions , which is not recognised by most coding schemes which normally
involve the assignment of utterances to mutually-exclusive categories.
(3) The phenomena of interest to the investigator may be spread over several
utterances, and so any scheme based on single utterances as the unit of
analysis may not capture such phenomena.
(4) Meanings change, and are re-negotiated during the course of the ongoing
conversation.
It might be thought that using two or more independent coders, and
measuring their level of agreement, overcomes the question of validity raised as
the first problem above. Indeed, coding schemes are often used in preference to
other discourse analysis methods because they appear to offer a more ‘objective
basis for validity claims. But, as Potter & Wetherell (1994) point out, this
widely-held belief confuses the reliability of a measure with its validity. That two
or more coders can consistently agree on how to code different classes of
ambiguous utterances tells us only that they have a shared way of interpreting
utterances it tells us little, if anything, about the validity of their way of
interpreting utterances. If, as is often the case, such researchers offer no examples
of the utterances they have coded in their original discursive contexts, readers of
their research reports have to take the validity of any interpretations entirely on
trust. Moreover, Potter & Wetherell argue that talk is inevitably and necessarily
ambiguous in its meanings because it is a means by which shared meaning is
negotiated. Crook (1994) emphasises that this limitation with coding schemes is
particularly serious in the study of collaborative learning. He argues that the
study of collaborative learning should be the study of the development of shared
knowledge over time. Coding schemes for talk fail to capture this crucially
important temporal dimension in co-operative activity, and tend to reduce
collaborations into atemporal ‘inventories of utterances’ (Crook, 1994: 150).
In research on collaboration, coding schemes are often used to search for
correlations between the incidence of some kinds of talk behaviour and particular
outcomes of joint activity (for example, success or failure in solving problems).
But while coding methods can show a statistical relationship between two events
in time, (i.e. that event B generally follows event A), they are not good at
Computer-based Analysis of Collaborative Learning 273
demonstrating causal relations between two events, (i.e., how and why event A
led to event B). For example, King’s (1989) finding, that there is a statistical
correlation between the incidence of task-focused questions and group success
in problem-solving is interesting, and suggestive of a causal link; but that kind
of analysis does not in itself explain how such a link is achieved. To explain such
a relationship, a researcher would have to show exactly how asking questions
helped the groups of learners to solve the problems.
Qualitative
Approaches
to
the
Analysis
of
Talk
and
Collaborative
Activity
in
Classrooms
Douglas Barnes (Barnes, 1976; Barnes & Todd, 1978; 1995) was amongst the
first researchers to devise an analytic method for studying collaborative learning
in classrooms, which was sensitive to context and to the temporal development
of meanings. In contrast to the coding approaches described above, Barnes has
used detailed classroom observation and the interpretation of transcribed talk of
children engaged in normal classroom tasks to explore the processes through
which knowledge is shared and constructed. His approach is allied to ethnogra-
phy, in that it incorporates intuitive understanding gained through discussions
with teachers and children, and participation in the contexts described. His usual
method of reporting his research is to demonstrate and illustrate his analysis by
including transcribed extracts of talk, on which he provides a commentary. Since
Barnespioneering work, many other educational researchers have developed
similar methods of discourse analysis, and some have applied them to the study
of children’s talk and joint activity (e.g. Lyle 1993; Maybin, 1994; Swann, 1994;
Mercer, 1995).
Critique
of
qualitative
discourse
analysis
In their comprehensive review of methods for researching talk in classrooms,
Edwards & Westgate argue that the strength of Barnes’ early work lay in making
easily taken for granted aspects of classroom life ‘visible’, and so available for
reflection, and that the value of this can be seen in the recognition his insights
gained immediately from many teachers (Edwards & Westgate 1994: 58).
However they also quote many critics of such insightful observation’ methods
(Edwards & Westgate, 1994: 108). It is easy, they write, to pull transcript evidence
out of context in order to illustrate a case already made and so to offer ‘only the
illusion of proof’. Stubbs similarly argues that while studies based on the
presentation of fragments of recorded talk can be insightful and plausible, they
raise problems of evidence and generalisation’ (Stubbs, 1994). It is often not clear,
Stubbs continues, how such studies could be replicated and compared, or how
they could lead to cumulative progress in the field. While we are not convinced
that Stubbs’ own sociolinguistic analyses are appropriate to the investigation of
shared understanding, his criticisms of fragment-based discourse analysis are
particularly relevant to our concerns here and all the more so because they
lead him to advocate the use of computer-based text analysis.
Qualitative discourse analysis, in the tradition of Barnes, must rely on
presenting short selected texts. Yet educational research often seeks generalisa-
tions, and evaluative comparisons, which cannot rest only on these samples. This
274 Language and Education
is why, as Hammersley has argued, qualitative analysis can be effective for
generating theories but not so effective for rigorously testing them (Hammersley,
1992). In contrast, the quasi-experimental research designs which are often
associated with the use of coding schemes and other quantitative measures can
offer explicit tests of hypotheses and systematic comparisons between the
communicative behaviour and outcomes of target’ and ‘control’ groups.
The strengths of quantitative and qualitative methods of discourse analysis in
the field of collaborative learning appear on the surface to be complementary.
However, as Snyder points out in her discussion of integrating multiple
perspectives in classroom research (Snyder, 1995) different methodologies can
be taken to embody different views of the nature of meaning. The act of coding
a text into a limited number of discrete categories, for example, tends to imply a
view of the meaning of utterances as relatively stable, unambiguous and
independent of context. Many qualitative researchers, on the other hand, insist
that the meaning of any utterance depends upon the way it is interpreted by
participants in the collaboration and so is not only highly sensitive to context but
also necessarily always ambiguous (Potter & Wetherell, 1994; Graddol
et al
.,
1994). In the next section we argue that computer-based transcript analysis offers
a way beyond this apparent divide through enabling a more context sensitive
approach to combining qualitative analysis with systematic comparison and
evaluation.
Interrelating
Different
Levels
of
Data
Computer-based
text
analysis
Computer-based tools such as concordancers are increasingly used in
linguistics to explore changes in word meaning and create modern dictionary
entries (Graddol
et al
., 1994). Software is being developed making it possible to
apply similar techniques to transcripts of discourse without the need for complex
coding. !KwicTex, for example, a concordancer designed by Graddol, allows the
researcher to move almost instantly between full transcripts and lists of
utterances, or other contextual unit, containing key words in context. Graddol
argues that the use of Key Word In Context (KWIC) searches of electronically
stored text can dramatically speed up the iterative cycle of exploration and testing
involved in any analysis of discourse (Graddol, in preparation). This iterative
cycle often combines close exploratory ‘qualitative’ work with generalisation and
testing of hypotheses about linguistic features across the whole of a text or series
of texts. The use of such techniques for written transcripts of spoken language
offers the possibility of systematic comparisons of language-use in different
settings without losing sight of the relationship between particular linguistic
features and their context within transcripts.
Different
levels
and
types
of
abstraction
in
the
data
Data can be more or less abstract: more or less concrete. A count of different
functions of utterances in a transcript is more abstract, in the strict sense, than the
full transcript itself. That transcription is in turn an abstraction from an audio
recording which may in turn be an abstraction from a full video and audio
Computer-based Analysis of Collaborative Learning 275
recording. Any video/audio recording of events is the most concrete level of data
available to language researchers after an event. However, recorded data is
evidently itself abstracted from a more concrete original event. The very concept
of recording assumes that there is a more concrete original event or events which
is recorded through being abstracted in some way. The nature of this original
level of actual events, a level which we posit as degree zero on the scale of concrete
to abstract, is a matter for philosophical debate which need not concern us here.
In research we are always dealing with data which has some degree of
abstraction.
Not all types of data can be easily placed in relation to each other on this
continuum. A brief narrative account of a week’s training course in problem
solving, for example, is not evidently any more or less abstract than the numerical
results of a questionnaire survey taken at the end of the course. Here each type
of data draws out different aspects of the concrete event.
The time required for analysis and the space required for presentation mean
that there is a
de facto
relationship between degree of abstraction useful in the data
and the sample size of a study or the degree of generalisation. More concrete data
such as video-recordings of events cannot be used to generalise across a range of
events without abstracting and focusing on some key features from each event.
The
proposed
method
We have seen that quantitative coding methods which have emerged from the
‘systematic observation’ tradition of research on classroom talk have been
criticised for their failure to handle the development of contextualised meanings
which is at the essence of talk. Qualitative methods of discourse analysis, on the
other hand, have been criticised for not incorporating a systematic basis for
abstracting and generalising from specific episodes of discourse. It was also
suggested that while coding methods (when incorporated into experimental or
quasi-experimental research designs) offer an efficient means for correlating
features of observed events with outcomes, they were weak in explaining any
process by which such correlations might arise. Qualitative methods, in contrast,
are good for showing how specific events unfold and arrive at their actual
conclusions, but are not easily used to demonstrate the superiority of one theory,
or generalisable account of such events, over another. It seems, then, that the
difficulty in effectively combining the strengths of quantitative and qualitative
methods of discourse analysis in the study of collaborative learning is the
problem of integrating different levels of abstraction in the data. Coding operates
at too abstract a level of data, and so the explanatory power of an analysis of the
quality of the talk in particular events is lost; while ‘insightful observation’ and
other similar methods of discourse analysis lack a systematic means of moving
beyond the concrete and particular. The value of computer-based text analysis,
we suggest, lies in its ability to perform a mediating role between these two
positions. Concordancing software such as !Kwictex enables a rapid movement
between different levels of abstraction. Units of analysis such as words,
utterances or conversational turns can be abstracted from the transcript to form
a separate list. At the same time , these words or utterances are highlighted in the
main text, which can be returned to instantly at any time. Clicking on an
276 Language and Education
abstracted conversational turn in a list, for example, returns the analyst
immediately to its place in the full transcript. At one extreme this software offers
the possibility of a quantitative breakdown of speakers’ conversational turns, or
other selected features such as the occurrence of particular ‘key words or types
of utterance; while at the other it offers the possibility of carrying out a close,
detailed analysis of full transcripts. The use of such a tool therefore facilitates the
inter-relation of different levels of abstraction in the data.
An essential and attractive feature of the method we are proposing here is that
it maintains a connection between relatively concrete data, such as recordings of
events, and relatively abstract data such as word counts or test scores. In this way
the quantitative analysis of test scores, for example, does not replace a qualitative
analysis of collaborative interactions but both co-exist and contribute to an
overall understanding. With regard to the dissemination of research, this new
methodology enables discourse analysts to present their findings in ways that
should be more explicit and convincing to a critical audience. Any qualitative
analysis is not represented only by the illustrative interpretation of a very few
selected events but is supported by a more systematic analysis of the data as a
whole, and possibly also by a demonstration of statistical relationships between
features of linguistic events and other kinds of measures of collaborative activity
(in this case, scores on group reasoning tests).
A
study
illustrating
the
proposed
method
To support the claim we are making for the value of combining the use of
computer-based text analysis with more established qualitative methods of
discourse analysis it is necessary to look at how this approach works in practice.
To that end we present some aspects of a recent research project which used this
approach in the context of evaluating an intervention programme.
The intervention programme consisted of a series of eight lessons coaching
exploratory talk
, defined as talk in which reasons are given for assertions and
reasoned challenges made and accepted within a co-operative framework
orientated towards agreement (as discussed in more detail in Mercer, 1995; see
Wegerif & Mercer, 1996, for a fuller account of this study). Each lesson integrated
three stages. They began with the teacher explaining and modelling the type of
interaction being taught, continued through small group exercises designed to
encourage the children to practice the type of interaction being taught, and
finished with whole class discussions both reflecting back on what had been
learnt and using ‘exploratory talk’ in a realistic context. The series of lessons
began with exercises to encourage cooperation and listening to each other. One
early lesson, for example, had the children sitting back to back in such a way that
only one could see an object which she had to describe to her partner in order
that her partner could draw it. Later lessons coached ground-rules of exploratory
talk, for example, that reasons should be given for assertions; that all should be
asked for their views and listened to with respect; or that several alternative
possibilities should be generated and discussed before an answer to a problem
is reached. The intervention programme included some computer software
designed to present information and problems in a way that encouraged children
to formulate hypotheses, share information, question assumptions and reach
Computer-based Analysis of Collaborative Learning 277
joint decisions where the content of their discussions was directly relevant to the
National Curriculum for England and Wales. In the area of citizenship, for
example, moral dilemmas were presented on the screen with a number of
different possible views in the form of the thought of participants to support the
children in debating alternatives, and embedded within an engaging narrative
frame. In the area of science, the software both encouraged the children to
formulate predictions before changing variables in a simulation, and then asked
them for reasons why their predictions succeeded or failed. (Some principles
behind the design of this software are put forward in Wegerif, 1996a, and the
software itself is described more fully in Wegerif, 1996b).
The evaluation of this intervention programme combined an analysis of
classroom talk and interaction throughout the programme with the use of a
pre-and post-intervention comparison of children’s problem solving (as de-
scribed below). This pre- to post-intervention comparison used two kinds of data:
scores from a group reasoning test and an analysis of the recorded talk of certain
focal groups of children, video-taped while doing this reasoning test. The use of
video made it possible to relate the children’s talk to the answers they gave to
particular problems in the test. With this research design it was possible to
statistically link changes in test score measures to changes in linguistic features,
in a similar way to many coding and counting studies, but it was also possible to
relate extracts of transcripts of groups talking together to their work on specific
problems, the sort of study normally found in the qualitative discourse analysis
tradition. A ‘control’ class of children of the same age in a neighbouring school
were also given the group reasoning test, before and after the intervention had
taken place in the target school.
Pre-
and
post-intervention
reasoning
test
results
The reasoning test given to both the target class and a control class, at the
beginning of the intervention programme and again at the end, consisted of
problems taken from a wisely used psychological test of reasoning, Raven’s
Standard Progressive Matrices. Each Raven’s problem involves matching shapes
and patterns. (Wegerif & Mercer, 1996, contains more information about this test).
The children (all aged 9 and 10 years) worked together in groups of three. In the
target class, nine groups produced comparable pre- and post-intervention tests,
while in the (smaller) control class, five groups produced comparable pre- and
post-intervention tests. There was one question sheet and answer sheet per group
and children were encouraged to talk together to reach a joint solution. In each
case the first two questions were used to explain the tests and the tests did not
begin until it was clear that all in the class understood the procedure.
All the group scores in both target and control classes increased over the period
of the intervention programme. The target class group scores increased by 32%
while the control class group scores increased by 15%. The differences between
the pre- and post-intervention test scores for all groups in the target class were
compared to the differences between the pre- and post-intervention-test scores
for all groups in the control class, and it was found that this difference was
significant (Z = -1.87 p = 0.031. One-tailed Mann-Whitney test, corrected for ties).
278 Language and Education
(These results are presented in more detail in Wegerif, 1996, and Mercer &
Wegerif, in press).
Transcript
evidence
The test results appeared to show that group effectiveness in problem solving
had increased. However, on their own, these highly abstract statistics do not tell
us why this change has occurred. To demonstrate a connection between the
outcome measures and the coaching of exploratory talk it is necessary to look at
changes in quality of the interactions of the groups. Video-recordings of three
‘focal groups’ of children in the target class (i.e. the class which had experienced
the intervention programme) were made when they did the initial, pre-
intervention reasoning task and again when they did the same task again after
the intervention. It was thus possible to study the talk of the groups as they solved
problems the second time that they had failed to solve the first time. The
following two transcripts give an illustration of this more concrete and
contextualised type of analysis.
The
pre-intervention
talk
around
problem
A11
John: (Rude noise)
Elaine: How do you do that?
Graham: That one look
All: It’s that (Elaine rings 1 as answer for A9)
Mean score out of 25
Pre-test
Target Control
Post-test
22
21
20
19
18
17
16
15
14
13
Figure 1
Comparing the pre- to post-intervention change in the means of the target
and control group reasoning tests
Computer-based Analysis of Collaborative Learning 279
Elaine
:
No, because it will come along like that (
Elaine rings 5 as answer for
A11
)
John: Look it’s that one (
Elaine rings 2 as answer for B1
)
The
post-intervention
talk
around
problem
A11
John: Number 5
Graham: I think it’s number 2
John: No, it’s out, that goes out look
Graham: Yeh but as it comes in it goes this
Elaine: Now we’re talking about this bit so it can’t be number 2 it’s that one
Elaine: It’s that one it’s that one
Graham: Yeh ’cos look
Elaine: 4
Graham: I agree with 4 (Elaine rings 4 as answer for A11)
Commentary
In the pre-intervention task question A11 was answered wrongly in the
context of a series of several problems which were moved through very rapidly.
The other problems in this short series were answered correctly. Elaine’s second
utterance ‘No, because it will come along like that’ implies that one of the other
two group members had just pointed to a different answer on the page. She gives
a reason to support her view and this is not challenged. There is no evidence that
65
3
2
4
1
Figure 2
A11
280 Language and Education
agreement is reached before the answer is given. The group move on to the next
problem. Looking at the full transcript it is apparent that the children do not take
the task set very seriously and much of their talk is off-task.
In the post-intervention task episode much more time is spent by the group
on A11. Two alternatives are considered and rejected before the right answer is
found and agreed on. This is crucial. In the pre-intervention task example only
one alternative was considered and rejected before a decision was reached. The
structure of the problem is such that, to be sure of a right answer it is necessary
to consider at least two aspects of the pattern. John first spots the pattern of the
dark vertical lines moving outwards and so suggests answer 5. Graham
contradicts John saying the answer must be 2 presumably basing this on the
pattern of the lighter horizontal lines. Just as Graham’s reason means number 5
is wrong so John’s reason means that number 2 is wrong. In proposing number
4 Elaine is building on these two earlier failed solutions. Graham sees that she is
right and points to confirming evidence on the page. In the context of John’s vocal
objections to previous assertions made by his two partners his silence at this point
implies a tacit agreement with their decision.
Both episodes appear to contain talk of the kind that we have called
exploratory
.
That is, challenges are offered, reasons are given and the group appear to be
working co-operatively towards a shared goal. However the second episode
includes a much longer sustained sequence of exploratory talk about the same
shared focus.
It was generally found to be the case that the problems which had not been
solved in the pre-intervention task and were then solved in the post-intervention
task, leading to the marked increase in group scores, were solved as a result of
group interaction strategies associated with exploratory talk and coached in the
intervention programme.
Key
Word
in
Context
(KWIC)
Analysis
We have presented above, in some detail, our analysis of how one group of
children solved one problem in the post-intervention task which they had failed
to solve in the pre-intervention task. We have used this analysis to support a claim
that the children’s use of exploratory talk was a significant factor in their finding
the solution of the problem. However, for reasons discussed in the early part of
this paper, this kind of claim (even if elaborated) may not be convincing to some
educational researchers, who might argue that we simply chose such an example
of talk because it supported, rather than tested, our claim. This is where the value
of using a computer-based analysis to augment a qualitative study can be
appreciated. The software we used, !KwicTex, enabled us to extract all uses of
specific linguistic features across the full transcripts of this group of children in
the immediate context of utterances in which they occur. In the extracts shown
above, Focal Group 1 solving problem A11, the use of the word ‘because’ or ‘’cos’
was found in both the pre-intervention and the post-intervention example. In the
following transcripts all the occurrences of this key word in the pre-intervention
task and the post-intervention task for this group are listed within the context of
one conversational turn.
Computer-based Analysis of Collaborative Learning 281
Focal
Group
1
pre-intervention
task
use
of
‘’cos’
or
‘because
Elaine: It isn’t
’cos
look that’s a square
Graham: No
’cos
look watch there all down there and they are all at the side and
they are all up there
Elaine: Wait wait wait its that one
’cos
look its them two and them two () and
them two
John:
’Cos
look that goes out like that
Elaine:
’Cos
look that goes in
John:
’Cos
look that goes too far out
Graham: Look
’cos
thats got 4
Elaine: No
¼
not that one not that one
because
its got a little bit like that its
that one look it goes in and then it goes out
John: No its isn’t
because
its there
Elaine: No
because
it will come along like that
Elaine: Could be that one
because
look stops at the bottom and look
Elaine: It isn’t It isn’t
because
look
(12)
Focal
Group
1
post-intervention
task
use
of
‘’cos’
or
‘because
Graham: Number 6 ’cos 6 stops in there ’cos look if you
Elaine: It can’t be there ’cos look if you done that
Elaine: It is look if that goes like that and then it has another one ’cos those
two make
Elaine: He doesn’t say what they are ’cos he might be wrong
Graham: Yeh ’cos look
Elaine: ’Cos it would go round
John: It is ’cos it goes away ’cos look that one goes like that
Elaine: No it can’t be ’cos look ¼ with the square with the triangle you take
away the triangle so you’re left with the square so if you do just this
and then again take that away it’s going to end up, like that isn’t it?
Graham: Actually cos that’s got a square and a circle round it
John: Yeh ’cos it goes like that and then it takes that one away and does that
Elaine: No ’cos look
Elaine: Probably one in the circle cos there are only two circles
Graham: ’Cos if they are lines and then they are going like that it is because they
are wonky isn’t it
Graham: No actually it ain’t ’cos then
Elaine: Yeh its number 8 because those ones those two came that those two
make that
John: No because 1, 2, 3 1, 2, 3
John: No because that goes that way and that goes that way
Graham: No because it’s that one
(21)
Commentary
For the same group of children, there were more uses of ’cos or because’ to
282 Language and Education
link reasons to claims in the post-intervention task than in the pre-intervention
task. In the pre-intervention task two-thirds (eight) of the usages are collocated
with ‘look’ that is either because look’ or ‘’cos look’. In the post-intervention
task this collocation is less frequent. It occurs six times which is less than one third
of the total uses. When collocated with ‘look’, because’ or ‘’cos’ has a deictic
function akin to non-verbal pointing. In this case the reason or warrant is usually
to be found outside of the talk in the physical context, i.e. on the problem book
page. In these utterances there is a noticeable shift between the pre-intervention
condition and the post-intervention condition from using ‘because/’cos’ in this
way to using it to link claims to verbally elaborated reasons which are more
evident in the talk. In other words, there is a shift from reliance on the physical
context to greater use of the linguistic context which the children construct
together.
Our ‘key word in contextanalysis took one linguistic feature which a detailed
qualitative analysis had indicated was associated with one group’s solution of
one reasoning problem, and tested the generality of this association across the
transcripts of this group in the pre-intervention and post-intervention tests. The
generalisation was made possible through the partial abstraction of listing the
utterances in which the key word occurred. This showed a marked shift in the
number of times this key word was used from the pre-to the post-intervention
condition. It also showed an evolution in the way this key term was used in
context. The use of !KwicTex in this way makes it possible to explore rapidly the
contextualised use of such linguistic features.
A
Count
of
Key
Usages
Our ‘key word in context analysis showed how the general occurrence of
specific words can be tracked over a set of related transcripts (in this case, those
representing one group of children working on one task). In disseminating
research, it is difficult to present many such analyses in a way that does not take
up much space and overburden the reader. To generalise the analysis further to
include several key terms hypothesised as marking the occurrence of exploratory
talk, and to do so for all the transcripts available from the three focal groups, it
is necessary to move up a level of abstraction and make a count of key usages’.
A key usage is not simply a key word but a key word being used to serve a
particular function. A key word count alone is not adequate for words which can
have many functions and where there is reported speech in transcripts, for
example words being read from instructions. The use of !KwicTex to look at a
word in its immediate context makes it easier to ascribe a function to that word.
In the context of the pre- and post-intervention group reasoning tasks the
following list of key usages were found, through qualitative analysis of the kind
shown above, to be indicative of exploratory talk:
‘if’ used to link a reason to an assertion
‘so’ used to link a reason to an assertion
‘because/’cos’ used to link a reason to an assertion
Computer-based Analysis of Collaborative Learning 283
questions used both to challenge, as in ‘why?’ questions, and to be socially
inclusive, as in what do you think?’ type questions.
Table 1 shows counts of these key usages for the talk of the three focal groups
when doing the standard group test together.
Summary
of
the
‘key
word
in
context’
analysis
results
Computer-based text analysis using !KwicTex demonstrates a difference
between the post-intervention task talk of the three focal groups and their
pre-intervention task talk. This difference takes the form of more talk, more
reason clauses using the linguistic forms because/’cos’, so and ‘if’ and more
questions being asked. This marked change in the use of language occurs in
association with an improvement in test-scores which was in line with the mean
improvement in all the groups of the target class (see Figure 1).
Summary
In the first half of this paper we proposed a method for the study of
collaborative learning which was illustrated in the second half of the paper. The
basis of the proposed method was interrelating different levels and types of data
to produce an overall interpretation which integrated qualitative and quantita-
tive dimensions. It was argued that the use of computer-based methods of text
analysis facilitated this integration through allowing the researcher to move
rapidly between different levels of abstraction in dealing with transcript data.
The illustration of this method given in the second half of the paper used two
kinds of data and four levels of abstraction. The most concrete level of data was
represented by the full transcript of each recorded event, (in the full study this
was enhanced by other information from video-recording and field notes). The
use of specialised computer software facilitated generation of data at the next two
levels. These were the selected ‘key words and other linguistic features in the
utterances of one focal group, and a count of ‘key usages’ for all three focal groups
in the target class. The final level of abstraction and generality, was afforded by
the results of the group reasoning tests. Interrelating these four levels of analysis
Pre-test Post test
Gp1 Gp2 Gp3 Total Gp1 Gp2 Gp3 Total
Test score 15 18 19 52 23 22 22 67
Questions 2 8 7 17 9 33 44 86
Because/’cos 12 18 9 39 21 34 40 95
So 6 3 1 10 6 5 7 18
If 1 1 0 2 13 8 14 35
Total words 1460 1309 715 3484 2166 1575 2120 5761
Table 1
Key usage count for the pre- and post-intervention tests of the focal groups
in the target class
284 Language and Education
enabled us to draw two conclusions. First, that the exploratory features of talk
responsible for solving the problem illustrated through a qualitative analysis
were features generally found more in the post-intervention talk of the children
doing a reasoning test task than in the pre-intervention talk of the children doing
the same task. Second, that this change in the style of talk of the children towards
exploratory talk was matched by an increase in group reasoning test scores.
Conclusion
Using computer-based transcript analysis to help combine qualitative and
quantitative methods in the study of collaborative learning can produce an
overall interpretation which is more convincing than either qualitative or
quantitative accounts can be if used alone. Computer-based transcript analysis
has a special role to play in bringing data representing different levels of
abstraction together into dynamic relationship, in which different kinds of
analysis reflect upon and inform each other. The computer-based method we
propose can facilitate this approach because it enables the abstraction of different
levels of linguistic data without ever leaving behind the original linguistic
contexts of the actual words spoken.
References
Azmitia, M. and Montgomery, R. (1993) Friendship, transactive dialogues, and the
development of scientific reasoning. Social Development 2 (3), 202–21.
Baker, M., Francis, G. and Tognin-Bonelli, E. (eds) (1993)
Text and Technology
. Amsterdam
and Philadelphia: Benjamin’s.
Barbieri, M. and Light, P. (1992) Interaction, gender and performance on a computer-based
task.
Learning and Instruction
2 (3): 199213.
Barnes, D. (1976) From Communication to Curriculum. Harmondsworth: Penguin Books.
Barnes, D. and Todd, F. (1978) Communication and Learning in Small Groups. London:
Routledge & Kegan Paul.
Barnes, D. and Todd, F. (1995) Communication and Learning Revisited. New York:
Heinemann.
Croll, P. (1986)
Systematic Classroom Observation
. Lewes, Sussex: The Falmer Press.
Crook, C. (1994) Computers and the Collaborative Experience of Learning. London and New
York: Routledge.
Doise, W. and Mugny, G. (1984)
The Social Development of Intellect
. Oxford: Pergamon Press.
Draper, S. and Anderson, A. (1991) The significance of dialogue in learning and observing
learning.
Computers and Education
17 (1), 93–107.
Edwards, A. and Westgate, D. (1994)
Investigating Classroom Talk
. London: Falmer Press.
Edwards, D. and Mercer, N. (1987) Common Knowledge: The Development of Understanding
in the Classroom
. London: Methuen/Routledge.
Graddol, D. (in preparation) !KWICTex a computer-based tool for discourse analysis.
Occasional Paper, Centre for Language and Communication, Open University.
Graddol, D., Cheshire, J. and Swann, J. (1994)
Describing Language
. Milton Keynes: Open
University Press.
Graddol, D., Maybin, J. and Stierer, B. (1994) Researching Language and Literacy in Social
Context
. Clevedon: Multilingual Matters.
Hammersley, M. (1992) What’s Wrong with Ethnography. London: Routledge.
Joiner, R. (1993). A dialogue model of the resolution of inter-individual conflicts:
Implications for computer-based collaborative learning. Unpublished PhD thesis, The
Open University.
King, A. (1989) Verbal interaction and problem solving within computer aided learning
groups.
Journal of Educational Computing Research
5, 1–15.
Computer-based Analysis of Collaborative Learning 285
Kruger, A. (1993) Peer collaboration: Conflict, cooperation or both?
Social Development
2
(3).
Maybin, J. (1994) Children’s voices: Talk, knowledge and identity. In Graddol, Maybin,
and Stierer (eds)
Researching Language and Literacy in Social Context
. Clevedon:
Multilingual Matters.
Mercer, N. (1995) The Guided Construction of Knowledge: Talk Amongst Teachers and Learners.
Clevedon: Multilingual Matters.
Mercer, N. and Wegerif, R. (in press) Is exploratory talk productive talk? In P. Light, and
K. Littleton (eds) Productive Interactions Around Computers. London: Routledge.
Perret-Clermont, A.N. (1980)
Social Interaction and Cognitive Development in Children
.
London: Academic Press.
Potter, J. and Wetherell, M. (1994) Discourse Analysis and Social Psychology. London: Sage.
Lyle, S. (1993) An investigation into ways in which children ‘talk themselves into
meaning’. Language and Education 7 (3), 181–97.
Snyder, I. (1995) Multiple perspectives in literacy research: Integrating the quantitative
and qualitative.
Language and Education
9 (1).
Swann, J. (1994) What do we do about gender? In B. Stierer and J. Maybin (eds) Language:
Literacy and Learning in Educational Practice. Clevedon: Multilingual Matters.
Stubbs, M. (1994) Grammar, text and ideology: Computer-assisted methods in the
linguistics of representation. Applied Linguistics 15 (2), 202–23.
Stubbs, M. (1996) Text and Corpus Analysis. Oxford: Blackwell
Teasley, S. (1995) The role of talk in children’s peer collaborations.
Developmental
Psychology 31 (2), 207–20.
Wegerif, R. (1996a) Collaborative learning and directive software. Journal of Computer
Assisted Learning
12 (1), 22–32.
Wegerif, R. (1996b) Using computers to help coach exploratory talk across the curriculum.
Computers and Education
26 (1–3), 51–60.
Wegerif, R. and Mercer, N. (1996) Computers and reasoning through talk in the classroom.
Language and Education 10 (1), 47–64.
286 Language and Education
... Another example of research on educational dialogues where the mediating role of digital technology has become more prominent in recent years is the seminal work on productive and exploratory talk (Wegerif & Mercer, 1997b;Wegerif & Scrimshaw, 1997). Here, a set of 'ground rules' guides the talking and working together in a group by using productive dialogue while collaborating (Mercer & Littleton, 2007). ...
Thesis
Full-text available
In the last decade or two, initiatives engaging the public in scientific activities have become increasingly popular. For example, in air pollution monitoring with Do-it-Yourself (DIY) low-cost sensors. It is a relatively new practice that emerged due to the falling costs of sensor technology and components, combined with enhanced access to communities for sharing information and support. The engagement in DIY monitoring is here described as a maker-based civic engagement where collective action towards civic goals is reached through a peer-based, interactive, and social practice adopted from the culture of the maker movement. Through a multi-sited ethnography this dissertation contributes with two perspectives on DIY monitoring; how an institutionally organized initiative perceives outcomes of public engagement and how a grassroot civic mobilization initiative acts and learns while DIY monitoring. The sites cover two approaches to involve people: top-down as an institutionally organised public engagement and bottom-up as a grassroot-driven civic engagement. To unpack the creation and sharing of meaning in this empirical setting, I draw on research on productive and exploratory dialogue, combined with research on online communities where people collectively engage in meaningful participation. The institutionally organized initiative plans for public participation and wants to influence people. However, they prioritize getting sensors up and running since not knowing how to address issues of empowerment. The members of the grassroot initiative do not engage in building common community knowledge around issues of air pollution the way the institutionally organised project wants. Instead, their civic and collective actions are intended to generate hyperlocal, open, real-time air pollution data and ensure that data will continue to be delivered. The maker-based civic engagement seen in this dissertation enables a specific form of interest-driven learning. The maker-based social media setting allows meaningful participation where members make sense of sensing through exploratory dialogues and scaffolding common community knowledge. This dissertation suggests that establishing social relationships through non-productive and purely social conversations may be of substantial value for the meaning that participants ascribe to participation in a community. https://hdl.handle.net/2077/70614
... Their findings indicate how a computer-based space can develop higher-order thinking by developing exploratory talk, that is, talk that leads children to explore their problem-solving skills. Exploratory talk is characterised by reflection, joint problem solving and constructive criticism, where students challenge their peers by offering different hypotheses or solutions to problems, thereby developing their reasoning skills (Mercer, 2000;Mercer, Wegerif & Dawes, 1999;Mercer & Fisher, 1997a;Wegerif & Mercer, 1997b;2000;Wegerif & Dawes, 1997;Wegerif & Scrimshaw, 1997). The research reported by SLANT certainly indicates that talk within a computer-based environment can develop a student cognitively, but it depends very much on the type of talk that characterises interactions. ...
... However, equally important are the socio-constructive processes that are realised by learners' actual language and discourse practices Wegerif & Mercer, 1997;Wise & Paulus, 2016). Several researchers have argued that learners need to be aware of discourse practices that are suitable for online communications, especially for negotiation in online discussions (Herring, 2004). ...
... However, equally important are the socio-constructive processes that are realised by learners' actual language and discourse practices Wegerif & Mercer, 1997;Wise & Paulus, 2016). Several researchers have argued that learners need to be aware of discourse practices that are suitable for online communications, especially for negotiation in online discussions (Herring, 2004). ...
Chapter
Full-text available
... However, equally important are the socio-constructive processes that are realised by learners' actual language and discourse practices (Vygotsky, 1978;Wegerif & Mercer, 1997;Wise & Paulus, 2016). Several researchers have argued that learners need to be aware of discourse practices that are suitable for online communications, especially for negotiation in online discussions (Herring, 2004). ...
... For example, the same phrase, 'vamos a ver' that apparently led to learning in the High-Scope data was only used in the offi cial classroom data to turn students' eyes to the blackboard where the answer was written for them. This dynamic combination of topdown and bottom-up analysis of the data, facilitated by electronic text analysis with the use of concordance software, was presented as the 'Dynamic-Inverted-Pyramid' or 'DIP' method (Wegerif & Mercer 1997b ;Wegerif, Mercer & Rojas-Drummond, 1999 ). I mentioned briefl y above how Hammersley noted a research cycle in which exploratory bottom-up research sometimes led to theories which could be tested in a second stage of top-down research. ...
... For example, the same phrase, 'vamos a ver' that apparently led to learning in the High-Scope data was only used in the offi cial classroom data to turn students' eyes to the blackboard where the answer was written for them. This dynamic combination of topdown and bottom-up analysis of the data, facilitated by electronic text analysis with the use of concordance software, was presented as the 'Dynamic-Inverted-Pyramid' or 'DIP' method (Wegerif & Mercer 1997b ;Wegerif, Mercer & Rojas-Drummond, 1999 ). I mentioned briefl y above how Hammersley noted a research cycle in which exploratory bottom-up research sometimes led to theories which could be tested in a second stage of top-down research. ...
... For example, the same phrase, 'vamos a ver' that apparently led to learning in the High-Scope data was only used in the offi cial classroom data to turn students' eyes to the blackboard where the answer was written for them. This dynamic combination of topdown and bottom-up analysis of the data, facilitated by electronic text analysis with the use of concordance software, was presented as the 'Dynamic-Inverted-Pyramid' or 'DIP' method (Wegerif & Mercer 1997b ;Wegerif, Mercer & Rojas-Drummond, 1999 ). I mentioned briefl y above how Hammersley noted a research cycle in which exploratory bottom-up research sometimes led to theories which could be tested in a second stage of top-down research. ...
... Even though the framework was developed for classroom contexts, there are arguments for using it to study online conversations. Wegerif and Mercer (1997), for example, argue that the use of computer-based transcripts 'enables the abstraction of different levels of linguistic data without ever leaving behind the original linguistic contexts of the actual words spoken ' (p.285). There are also arguments addressing the way we communicate online and face to face. ...
Article
Full-text available
In the relatively understudied area of online informal scientific engagements, this paper aims to unpack the process of creating and sharing knowledge and understanding through dialogue. This is done through online ethnography in an initiative where members with their own air pollution monitors are discussing matters of air pollution, monitoring, data, and technical issues in a public Facebook group. By analysing a sample of text-based dialogue through the framework of exploratory talk, this paper addresses how knowledge and changes of understanding are negotiated, and situated, through social interaction within the community. The findings provide detailed and empirical insights into the ways in which the members of this informal online scientific community create and share meaning through social interaction. The use of low-cost sensors, active participation in the Facebook group, and a shared interest in understanding their own air pollution data are here shown to mediate further engagement and informal learning. Through social interaction and by sharing resources, members are enabled to engage in productive dialogue where common knowledge is created and scaffolded in the community over time. By unpacking the content and context dependent aspects of dialogue, this paper argues for the importance of meaningful participation for productive dialogues.
Article
Background: Studies on the writing of students with intellectual disabilities have been scarce and unrepresentative. Therefore, the purpose of this research was to analyse this group of students' abilities to plan their texts METHOD: A collective case study was carried out, with an eminently qualitative approach, although aided by an initial quantitative analysis. Fifteen students with unspecified intellectual disabilities were interviewed using open-ended questions. For data interpretation, content analysis and quantification of responses were used. Results: The results revealed the abilities and limitations of these students regarding the knowledge and use of one of the key processes of written expression (writing planning). Conclusion: The cognitive operations of planning, in which these students admitted the greatest problems, were the ordering and recording of ideas, and textual structuring.
Article
Full-text available
We propose an approach to teaching reasoning through talk in classrooms which integrates group work around computers into curriculum-related activities. The rationale for this approach lies in a communicative, sociocultural perspective on intellectual development and the process of education in schools. We also draw on recent research on collaborative learning which points to the importance of the quality of classroom talk for the construction of shared knowledge and for enabling children to join educated communities of discourse. Finally, we describe a programme of research in which, using specifically designed software, the computer is used to encourage and direct primary school children's explicit, joint reasoning through talk, in curriculum-related activities.
Article
This paper investigates the validity of claims that children provided with opportunities to work cooperatively and collaboratively in small groups will use talk to help them make meaning and enhance their cognitive understanding. Transcripts of children's talk in small groups are analysed to see how they use language to construct their understanding. The transcript shows children actively engaged in their own learning, arguing rationally, hypothesising, reflecting and evaluating their ideas. It shows how the children utilise their own ‘commonsense’ knowledge of the world and use it as a basis for developing a more explicit, systematised, integrated knowledge. The paper concludes that styles of structured, collaborative learning such as those described in the paper can provide a context for teaching and learning which takes account of the understanding we have of how children can use talk to make meaning for themselves in an educational setting.
Article
Verbal interaction and problem-solving behavior of small cooperative peer groups were observed while these groups worked on a computer-assisted (non-programming) problem-solving task. The purpose of the study was to identify problem-solving behaviors which relate to success within this context. Thirty-six fourth grade students were assigned to groups of three to form six groups of high and six of average academic ability. All groups used a nonprogramming version of Logo turtle graphics to reproduce a given line design on the computer screen. Results indicate that there was no relationship between success and ability, and that successful groups asked more task-related questions, spent more time on strategy, and reached higher levels of strategy elaboration than did unsuccessful groups. High ability groups made a greater number of long task statements than did average groups, bindings are discussed within the theoretical frameworks of social cognition and modeling. Instructional implications, including those for the development of computer-assisted learning materials for peer group problem solving, are also discussed.