Developments and trends in synthesizing diverse forms of evidence: Beyond comparisons between distance education and classroom instruction

Abstract

This paper describes a range of models and methods for synthesizing diverse forms of research evidence. Although this topic is applicable to many contexts (e.g. education, health care), the focus here is on the research traditions of distance education and online learning. Thirteen models and methods are described, and seven examples from distance education and online learning are used to illustrate them. The models are classified as systematic, purposeful, and mixed and are described and compared in terms of purpose, methodological aspects, and expected outcomes.

Full text

PLEASE SCROLL DOWN FOR ARTICLE
This article was downloaded by:
[Canadian Research Knowledge Network]
On:
16 November 2010
Access details:
Access Details: [subscription number 783016864]
Publisher
Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-
41 Mortimer Street, London W1T 3JH, UK
Distance Education
Publication details, including instructions for authors and subscription information:
http://www.informaworld.com/smpp/title~content=t713412832
Developments and trends in synthesizing diverse forms of evidence:
beyond comparisons between distance education and classroom instruction
Edward C. Bethel
a
; Robert M. Bernard
a
a
Centre for the Study of Learning and Performance, Concordia University, Montreal, Canada
Online publication date: 11 October 2010
To cite this Article Bethel, Edward C. and Bernard, Robert M.(2010) 'Developments and trends in synthesizing diverse
forms of evidence: beyond comparisons between distance education and classroom instruction', Distance Education, 31:
3, 231 — 256
To link to this Article: DOI: 10.1080/01587919.2010.513950
URL: http://dx.doi.org/10.1080/01587919.2010.513950
Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf
This article may be used for research, teaching and private study purposes. Any substantial or
systematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply or
distribution in any form to anyone is expressly forbidden.
The publisher does not give any warranty express or implied or make any representation that the contents
will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses
should be independently verified with primary sources. The publisher shall not be liable for any loss,
actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly
or indirectly in connection with or arising out of the use of this material.

Distance Education
Vol. 31, No. 3, November 2010, 231–256
ISSN 0158-7919 print/ISSN 1475-0198 online
© 2010 Open and Distance Learning Association of Australia, Inc.
DOI: 10.1080/01587919.2010.513950
http://www.informaworld.com
Developments and trends in synthesizing diverse forms of evidence:
beyond comparisons between distance education and
classroom instruction
Edward C. Bethel* and Robert M. Bernard
Centre for the Study of Learning and Performance, Concordia University, Montreal, Canada
Taylor and FrancisCDIE_A_513950.sgm
(Received 10 May 2010; final version received 15 June 2010)
10.1080/01587919.2010.513950Distance Education0158-7919 (print)/1475-0198 (online)Original Article2010Open and Distance Learning Association of Australia, Inc.313000000November 2010Edward C.Betheledcbeth@gmail.com
This paper describes a range of models and methods for synthesizing diverse
forms of research evidence. Although this topic is applicable to many contexts
(e.g. education, health care), the focus here is on the research traditions of distance
education and online learning. Thirteen models and methods are described, and
seven examples from distance education and online learning are used to illustrate
them. The models are classified as systematic, purposeful, and mixed and are
described and compared in terms of purpose, methodological aspects, and
expected outcomes.
Keywords: distance education; research synthesis; systematic review; online
learning; e-learning
Introduction
From presidents to principals, policy-makers at all levels have been asking the same
questions: ‘Does it work?’ and ‘If it works, is it working the way it’s supposed to
work?’ Researchers in distance education and online instruction have asked and
attempted to answer these ‘what works’ questions by comparing distance education
(DE) or online learning (OL) to their most likely alternative, classroom instruction
(CI). As a result, hundreds of DE studies have been completed, providing policy-
makers an abundance of evidence upon which to base policy and practice recom-
mendations. Given the often-contradictory nature of primary research, however, this
abundance of evidence poses its own challenges to ‘evidence-based policy’, the idea
that decisions about educational interventions are or should be made on the basis of
scientific evidence of effectiveness. In many ways, evidence-based policy is not new
at all. Policy-makers have always sought evidence to support their decision-making.
As often as not, however, policy-makers and their advisors used ad hoc, unsystem-
atic methods of evidence gathering and synthesis. Two things were missing:
evidence-gathering procedures to ensure that all the evidence was considered, and
methods for summarizing and synthesizing this evidence base. In response to these
twin needs, modern methods of research synthesis have evolved (Cooper, Hedges, &
Valentine, 2009).
Researchers have attempted to synthesize this continually growing body of
DE literature in order to provide general answers to questions related to differences in
*Corresponding author. Email: edcbeth@gmail.com
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

232 E.C. Bethel and R.M. Bernard
achievement, attitude/satisfaction, and student retention between DE and classroom-
based courses. One of the challenges of synthesizing DE research is that the research
base is diverse, incorporating studies that span the range of research design and meth-
odology. The purpose of this paper is to explore the different approaches that have
been used to synthesize DE research, diverse as it is, and to provide a theoretical
framework for understanding diverse evidence synthesis in general. We will not
recommend a particular method, but instead try to lay out for the reader the strengths
and weaknesses of each approach and the conditions, relating to the type of question,
the type of data, the type of answer that is desired and the purpose that it will serve,
for which each model is most appropriate.
We begin by describing several approaches that have been used to synthesize
research on DE and OL in different ways to illustrate the differences and similarities
between the synthesis methods. Next, we explore challenges facing the synthesis of
diverse evidence. Finally, for the purpose of description of particular methodologies,
we will use a classification structure that contrasts three types of syntheses: systematic
synthesis, purposeful synthesis, and mixed synthesis.
Research synthesis and the ‘what works’ question
Research synthesis is the process through which two or more research studies are
assessed with the objective of summarizing the evidence relating to a particular ques-
tion. The term also relates to the product of such a process. This practice of synthesiz-
ing research has a long history, but emphasis upon it has increased because the corpus
of primary research studies has increased exponentially. Whether it is a synthesis of
quantitative or qualitative evidence, a research synthesis typically adheres to a set of
characteristics that initially help guide the synthesis and later are used to convince
others that it can be trusted as a source of evidence. As we will see in the following
sections, different approaches to synthesis emphasize these characteristics differen-
tially. These general characteristics are as follows (adapted from Bernard & Abrami,
2009; Cooper et al., 2009, p. 13):
● Systematic – step-by-step iterative approach to the method
● Comprehensive – encompasses all available relevant research
● Critical – appraises the nature of evidence and selects appropriately
● Objective – minimizes bias at each step
● Rigorous – applies the strictest standards or states why exceptions apply
● Transparent – subjects every important detail to scrutiny
● Replicable – repeatable by other researchers to see if conclusions hold
● Integrative – reaches general conclusions or states why this cannot be
accomplished
● Explanatory – relates findings to theory/practice
● Relevant – adds value to researchers, theorists, practitioners and/or policy-makers.
The goal of a research synthesis that attempts to answer the ‘what works’ question is
to aggregate the existing research on a particular question and draw general conclu-
sions from the research evidence that it encompasses. Traditionally, these questions
have focused on the effectiveness of interventions, and hence quantitative studies,
in particular those employing true experimental or high-quality quasi-experimental
designs have been targeted for synthesis (Dixon-Woods, Agarwal, Jones, Young, &
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

Distance Education 233
Sutton, 2005; Goldsmith, Bankhead, & Austoker, 2007; Mays, Pope, & Popay, 2005).
Some see carefully designed experiments as the only ‘scientific’ evidence that can
yield valid conclusions about whether an intervention caused desired outcomes (e.g.
Torgerson & Torgerson, 2008). Typically, syntheses that address ‘what works’ ques-
tions deal exclusively with quantitative studies, although we will see that there are
qualitative approaches to synthesis that also include quantitative studies.
Meta-analysis
Meta-analysis is a well-established methodology for synthesizing quantitative
research findings. Since 2000, at least 15 meta-analyses or quantitative DE syntheses
have been completed (Allen, Bourhis, Mabry, Burrell, & Mattrey, 2002; Allen et al.,
2004; Bernard, Abrami, Lou, Borokhovski, Wade et al., 2004; Cavanaugh, 2001;
Cavanaugh, Gillan, Kromrey, Hess, & Blomeyer, 2004; Cook et al., 2008; Jahng,
Krug, & Zhang, 2007; Lou, Bernard, & Abrami, 2006; Machtmes & Asher, 2000;
Shachar & Neumann, 2003; Sitzmann, Kraiger, Stewart, & Wisher, 2006; Ungerleider
& Burns, 2003; US Department of Education, 2009; Williams, 2006; Zhao, Lei, Lai,
& Tan, 2005). Results suggest that DE and CI are roughly equivalent (i.e. an average
effect size of about zero), but thoses studies vary tremendously, from highly positive
effects (i.e. d ≈ +1.5SD), suggesting that DE outperforms CI, to highly negative results
(i.e. ≈ −1.5SD), suggesting the reverse. Overall, this body of literature has yielded
very little in terms of answering the ‘what works in DE’ question, largely because of
the nature of the question itself (see Bernard, Abrami, Lou, & Borokhovski, 2004 for
a discussion of these issues). For example, Bernard, Abrami, Lou, Borokhovski,
Wade, et al. (2004) conducted a meta-analysis of the comparative DE literature
between 1985 and 2002. The aim was to determine average effect size in studies that
compared DE and CI. In total, they analyzed 232 studies containing 688 independent
achievement, attitude, and retention (opposite of dropout) outcomes. Overall results
indicated effect sizes of essentially zero on all three measures and wide variability.
This suggests that many applications of DE outperform their classroom counterparts
and that many perform more poorly. Dividing achievement outcomes into synchro-
nous and asynchronous forms of DE produced a somewhat different impression. In
general, mean achievement effect sizes for synchronous applications favored CI,
while effect sizes for asynchronous applications favored DE. However, significant
heterogeneity remained in each subset. They assessed moderator variables for media
use, pedagogy, subject matter, and grade level in an attempt to explain variability
among studies.
More recently, Bernard et al. (2009) embarked on a new course by meta-analyzing
the results of three interaction treatments (i.e. student–student, student–teacher and
student–content) within a collection of 74 DE vs DE studies. The results suggest a
moderate overall effect for interaction treatments (g+ = +0.38) and generally support
some of Anderson’s (2003) hypotheses about the nature of interaction and the combi-
nation of interaction treatments. Perhaps more importantly, the approach that was
developed may represent a more useful means for answering questions about DE more
directly, without reference to CI.
The term meta-analysis was coined by Gene Glass (1976) to distinguish this method
of research synthesis from primary analysis of a single sample. Meta-analysis, like vote
counting (to be described next), is a study of samples. But this is where the similarity
ends. Whereas vote counting relies on inferential test statistics, meta-analysis uses
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

234 E.C. Bethel and R.M. Bernard
descriptive statistics (i.e. means, standard deviations, and sample sizes) as the raw mate-
rial for judging the magnitude of difference between groups. The outcome, referred to
as an effect size, is generally expressed as the standardized difference (i.e., expressed
in units of standard deviation) between the means of the experimental or intervention
condition and the control condition (i.e. ). An effect size,
unlike a test statistic, is relatively independent of sample size, so that derivatives of
large samples are relatively equal to derivatives of small samples. Effect sizes can be
averaged based on the findings of many experimental studies to determine how effec-
tive an intervention is, on average. Cohen (1988) describes an effect size of up to 0.20
standard deviation units as being small, up to 0.50 as being moderate, and any effect
size above 0.80 as being large. Additionally, moderator variable analysis derived from
coded study features can be performed to compare conditions within the overall meta-
analysis, such as different age levels or content areas, in an attempt to explain why stud-
ies vary (Abrami & Bernard, 2006). Analysis of variance and regression analogues are
used to investigate differences or establish relationships between predictors and
outcomes. Meta-analysis has its detractors too (e.g. Eysenck, 1978, comparing apples
and oranges being the most common complaint), but meta-analysis still stands as the
most effective and fully developed method for synthesizing quantitative experimental
studies.
One important challenge has been that meta-analysis is not discriminating enough.
Robert Slavin (1986) has argued that the accuracy and validity of meta-analytic find-
ings are compromised by the inclusion of studies of varying methodological quality.
As an alternative, he suggests that the principles of best evidence should be applied;
only the highest-quality relevant studies should be included in the synthesis, as
these will produce the most reliable findings. Lower-quality studies should only be
included in the absence of high-quality studies and only when potential threats to
validity are accounted for (Slavin, 1986). A problem that sometimes arises is that
extreme selectivity leads to low statistical power that limits the exploration of subsid-
iary characteristics. Best evidence synthesis can be thought of as a more restrictive
form of meta-analysis (Slavin, 1986).
Vote counting
Thomas L. Russell’s (2001) well-known collection of studies of media and DE is a
good example of a vote count. Russell proclaims that because the number of no-
significance studies so vastly outweighs the number of significant findings there must
be no advantage for technology integration or DE studies compared to CI. The prob-
lems with this study are threefold. First, it is not a systematic research synthesis
because of the method of collecting evidence (i.e., Russell invites his readers to submit
studies for inclusion in the database). Second, its very title proclaims a bias that
undoubtedly influences the conclusion of ‘no significant difference’ and possibly the
additional evidence that is sent to him by readers. Third, and apropos to this descrip-
tion, the method of vote counting tends to misclassify studies because of technical
limitations relating to large and small sample sizes.
Vote counting is one of the simplest methods of aggregating a large number of
experimental studies. Typically, this involves sorting studies, based on a test statistic
(e.g. t-test, ANOVA), into three categories:
(1) studies producing significant differences favoring the treatment
dXXSD
Cohen s E C pooled'
/=−
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

Distance Education 235
(2) studies producing significant differences favoring the control
(3) studies producing no significant differences (i.e., favoring neither the treat-
ment nor the control group).
The category with the greatest frequency wins, by virtue of having the most votes. In
Russell’s (2001) study the ‘winner’ was the no-difference category. A much more
detailed description of this method can be found in Cooper et al. (2009) and in Hedges
and Olkin (1980). A particularly scathing discussion can be found in ‘Why vote count
reviews don’t count’ (Friedman, 2001).
Challenges of research synthesis
Meta-analysis is accepted as the most reliable way of synthesizing multiple quantitative
studies. Similarly, methods for synthesizing qualitative research, while newer, have
evolved and gained a certain level of acceptance in the research community. Not only
are methods to synthesize diverse evidence still evolving (Abrami, Bernard, & Wade,
2006; Dixon-Woods et al., 2005; Goldsmith et al., 2007; Mays et al., 2005), but there
is also controversy over the legitimacy or desirability of attempts to synthesize disparate
evidence sources (Pope, Mays, & Popay, 2007).
Epistemological issues
The epistemological divisions between qualitative and quantitative researchers,
debated endlessly over the last 20 years, are repeated in synthesis methodologies. In
the quantitative approach, knowledge emanates from the systematic application of
scientific procedures to ensure objectivity and minimize bias. By contrast, in the qual-
itative approach, knowledge emanates from purposeful exploration and interpretation
of data in order to gain deeper understanding. The key contrast here is between
systematic and purposeful approaches to the conduct of research. On the one hand, in
the purposeful approach, data sampling, collection and analysis are explicitly tailored
to explore and explain a certain phenomenon. On the other hand, in the systematic
approach, data sampling and collection strives to be as representative of the popula-
tion in question and data analysis to be as objective as possible. In this regard the two
approaches can be thought of as endpoints on a research design continuum. One might
easily imagine traditional, narrative reviews approaching the ‘purposeful’ end of the
continuum, while meta-analysis, developed specifically to overcome the inherent
danger of bias in purposeful narrative reviews, approaches the ‘systematic’ endpoint
(see Figure 1).
Figure 1. Continuum of synthesis methods.
This is not to say that quantitative research is not purposeful, or that qualitative
research is not systematic, but rather that while all research is both systematic and
purposeful, the emphasis given to one or the other will reflect the epistemology of the
research paradigm. Green and Skukauskait
[edot ] (2008) underscored the need for transpar-
ency in research so that readers can easily judge where a report is purposeful and
where it is systematic. Credibility is undermined when the two approaches are
conflated (Green & Skukauskait
[edot ] e, 2008). At the same time, however, theoretical and
epistemological differences are often diminished by the realities of conducting actual
research. Sandelowski, Voils, and Barroso (2007) argue that few studies in the social
sciences meet the criteria of qualitative or quantitative orthodoxy. For example,
grounded theory studies often turn out to be qualitative descriptive studies, lacking in
e˙
e˙
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

236 E.C. Bethel and R.M. Bernard
the subtle insights typical of grounded theory, while experimental studies turn out to
be observational studies lacking in controls or randomization. Synthesis in these
instances may be straightforward, as there is less to lose either in converting the
words to numbers in the case of qualitative to quantitative or vice-versa. Beyond this
qualitative–quantitative divide, further specific challenges must be overcome.
Qualitative challenges
Some qualitative researchers hold that the very nature of qualitative research resists
synthesis and aggregation. They are wary of the loss of uniqueness that synthesis
would bring to their studies, and the inevitable thinning out of the thick descriptions
that are the hallmark of qualitative research (Sandelowski, Docherty, & Emden, 1997).
Moreover, the diversity of qualitative methodologies works against the possibility of
successful research synthesis. Because these differences reflect epistemological rather
than procedural differences, the challenge of reaching a working reconciliation is
daunting (Sandelowski et al., 1997).
Another concern is the application of quality criteria for the inclusion of qualitative
studies in a synthesis. Study quality has traditionally been one of the more important
inclusion criteria, as this directly impacts the credibility of the study findings. For quan-
titative research, assessment of study quality has meant the degree to which studies
meet various validity standards (internal, external, construct, statistical analysis), even
though researchers may disagree over the relative importance to assign to each of these
standards. For qualitative research, however, there is no consensus about what consti-
tutes quality criteria. Indeed, there are those who advocate abandoning what they call
‘criteriology’ altogether (Dixon-Woods, Shaw, Agarwal, & Smith, 2004; Freeman,
deMarrais, Preissle, Roulston, & St Pierre, 2007; Maxwell, 1992; Sandelowski et al.,
1997). This approach may be a reaction to the various ‘hierarchies of evidence’ and
their reification of randomized control trials (RCTs) as the ‘gold standard’ of research
on the one hand and their relegation of qualitative research to the bottom rungs of the
hierarchy on the other (Dixon-Woods, Shaw et al., 2004; Evans, 2003; Petticrew &
Roberts, 2003).
At the same time, many in the qualitative community recognize the important
contribution qualitative evidence can make to research synthesis and appreciate the
need for broader relevance of the studies beyond their particular frames of reference
(Dixon-Woods, Agarwal, Young, Jones, & Sutton, 2004; Sandelowski, 2004;
Figure 1. Continuum of synthesis methods.
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

Distance Education 237
Sandelowski et al., 1997). They suggest that the question of quality may be best
addressed by the ‘horses for courses’ approach – the expression stems from the fact
that a racehorse performs best on a racecourse specifically suited to it – that is, the
quality, or more correctly, the appropriateness of a study is dependent on the research
question to be answered. Different types of research questions will demand different
types of evidence (Abrami, Bernard, Wade et al., 2006; Gough, 2007; Petticrew &
Roberts, 2003; Sandelowski et al., 2007). This approach may be the one that bears the
most fruit as it reflects the needs of policy-makers to heed diverse points of view when
framing policy.
Quantitative challenges
Despite the recognized importance of incorporating qualitative research into research
syntheses of evidence, there remain powerful advocates for limiting evidence-based
policy to the results of experimental or ‘high-quality’ quasi-experimental studies (Best
Evidence Encyclopedia, n.d; Coalition for Evidence-Based Policy, 2008; Slavin,
2008; What Works Clearinghouse, 2008). At the same time, however, some of
the organizations traditionally committed to quantitative synthesis are recognizing the
benefits of including more diverse evidence sources in their reviews. Both the
Cochrane Collaboration (http://www.cochrane.org/) and the Campbell Collaboration
(http://www.campbellcollaboration.org), review bodies dedicated to evidence-based
health and social programs respectively, emphasize the importance of trials-based
quantitative reviews, but acknowledge that certain questions need to be answered by
reviews of other types of evidence (Higgins & Green, 2008; Shadish & Myers, 2004).
Both the Cochrane and Campbell Collaborations are following the lead of the
Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre,
2006–2009), a part of the Social Science Research Unit at the Institute of Education,
University of London.
Although the Cochrane and Campbell Collaborations have acknowledged the need
to include diverse evidence to answer complex questions, the EPPI-Centre has already
developed a methodology for dealing with these sorts of questions. Researchers at the
EPPI-Centre have published several works outlining how the methodology has been
used to produce reviews on several issues in the health sciences (Brunton et al., 2005;
Goldsmith et al., 2007; Harden et al., 2004; Rees et al., 2006; Thomas et al., 2004). In
fact, the EPPI-Centre is responsible for the lion’s share of published literature on the
synthesis of diverse evidence. Another promising approach, the Argument Catalogue,
is being developed by the Centre for the Study of Learning and Performance (CSLP;
http://doe.concordia.ca/cslp/), a research consortium of Montreal area universities and
colleges (Abrami, Bernard, & Wade, 2006). A more detailed discussion of both these
methods follows later.
Of course, there are many important questions, especially ones involving social
policy decisions that are unanswerable or very difficult to answer using any method-
ology, whether qualitative or quantitative. In an interview with Daniel H. Robinson
(2004), when asked about the US policies that demand evidence based on randomized
trials, Gene Glass queried, ‘where is the randomized experiment that proves that
randomized experiments are the royal road to truth? There is none, of course.’ He
followed this with: ‘If the federal government wishes to be consistent … then they will
have to back off their policies on smoking, coal dust, speeding on interstate highways,
and a host of other things that have never been verified by randomized experiments.’
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

238 E.C. Bethel and R.M. Bernard
(p. 26). The point Glass was making is demonstrated by Smith (2003) in a mock meta-
analysis and article entitled ‘Parachute use to prevent death and major trauma related
to gravitational challenge: Systematic review of randomised controlled trials.’ After a
thorough search of major databases, where no randomized trials were found, Smith
concluded that there is no ‘scientific’ evidence that parachutes work to prevent death.
He suggested: ‘that everyone might benefit if the most radical protagonists of evidence
based medicine organised and participated in a double blind, randomised, placebo
controlled, crossover trial of the parachute.’ (p. 1459). The point is well taken and self-
evident.
Methodologies of research evidence synthesis
In this section, we identify the important factors that need to be considered when
attempting diverse evidence synthesis and briefly describe the systematic, purposeful,
and mixed approaches to research synthesis. Where possible, we give examples of
syntheses from DE or OL to help illustrate the descriptions.
Nature of the data
Diverse evidence is just that – diverse. Synthesis methods are usually tailored to a partic-
ular type of evidence, for example meta-analysis aggregates and averages difference
findings in experimental or quasi-experimental studies, whereas meta-ethnography
aggregates findings from ethnographic studies. Moreover, as syntheses broaden their
scope to include policy documents and opinion pieces, the need for and complexity of
a new approach increases. There have been three broad approaches: quantitative,
qualitative (see Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou, 2004), and the
‘horses for courses’ approach (Petticrew & Roberts, 2003). Both the quantitative and
qualitative approaches involve data reduction, however, and important findings can be
lost in translation. These three approaches are discussed in more detail below.
Sampling
Sampling techniques in quantitative studies are chosen for very different reasons than
in qualitative studies. Quantitative samples reflect an attempt to make the findings as
generalizable as possible, whereas qualitative samples reflect an attempt to identify
and describe uniqueness. Study findings reflect this difference in scope and focus.
Successful inclusive syntheses must include systematic translation of the specific to
the general and vice versa.
Many approaches to the synthesis of diverse evidence have emerged. Some have
been developed precisely for this purpose, while others extend primary methodologies
to literature synthesis. Synthesis methods can be effectively characterized and
compared using the systematic-to-purposeful continuum discussed above.
Research design
Qualitative and quantitative synthesis methods place different emphasis on research
design. In the quantitative tradition, where causal inferences are an important
outcome, true experiments (often called RCTs) and high-quality quasi-experiments
are prized for their potential for eliminating or reducing threats to internal validity. All
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

Distance Education 239
threats cited by Campbell and Stanley (1963; 1966) are reduced if random assignment
of participants to conditions can be accomplished. Quasi-experiments reduce selection
bias through pretesting, matching or other procedures. Whenever possible, quantita-
tive synthesis methods attempt to reduce bias by selecting or controlling for the qual-
ity of included studies. However, as controls are introduced, threats to external
validity, or (the representativeness of the study to real life conditions) may arise. In
education, this is particularly important because so many of the aspects of practice
cannot be easily manipulated, and when they can, do not translate easily into practice.
Although it is possible to implement experimental designs that limit threats to both
internal and external validity, it is difficult and expensive to do so.
Qualitative research syntheses place greater emphasis on the representativeness or
authenticity of the setting, participants, and so on (external validity). As a result, stud-
ies that are rich in the details of process and context are prized, and data are often
wholly verbal in nature. Some have argued (e.g., Lincoln & Guba, 1985) that it is
differences in epistemology and the differential emphasis on research methodology
and the form of data lead to incompatibility at the level of synthesis. However, as we
will see, there is cause for optimism.
Steps in performing syntheses
According to Gough (2007) all reviews share a common structure consisting of
several steps:
(1) problem statement/statement of research question
(2) search, retrieval, and selection of studies
(3) analysis of studies
(4) synthesis of studies.
Each of these steps can be characterized as systematic, purposeful, somewhere in
between the two, or a combination of the two. In Table 1, the steps of a variety of
methodologies are compared on the systematic-to-purposeful continuum.
This breakdown helps locate each of these methodologies on the systematic-to-
purposeful continuum, remembering that these are general guidelines, not fixed clas-
sifications. Many of the qualitative methods have incorporated systematic search and
selection methods in an attempt to minimize actual or potential bias. In this regard, the
gap between the ends of the continuum is narrowing. Nonetheless, a clear distinction
can be made between studies that use purposeful versus systematic synthesis methods.
When the purpose is to explore and explain, the synthesis itself will be purposeful, even
though all other steps may be systematic. For this reason, the synthesis step was the
litmus test to determine whether studies were placed on the systematic or purposeful
end of the continuum. Note that two methodologies use both systematic and purposeful
methods in the synthesis. In both the EPPI and Argument Catalogue methods, different
types of evidence are synthesized using appropriate methods for those types. The find-
ings from each of these smaller syntheses are brought together for a grand synthesis.
Synthesis design
Sandelowski et al. (2006) described diverse evidence synthesis as having one of three
designs, depending on how evidence is incorporated into the synthesis: the segregated
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

240 E.C. Bethel and R.M. Bernard
design, the integrated design, or the contingent design. In the segregated design, the
research question is divided into sub-questions that are best answered by specific
types of evidence (the ‘horses for courses’ approach). Separate syntheses are
conducted to answer those specific questions. In the end, all of the smaller syntheses
are themselves synthesized in a grand review. The EPPI and Argument Catalogue
methods are good examples of segregated syntheses. By contrast, in the integrated
method, all evidence is considered to be essentially compatible and is synthesized
together to answer a single research question. The case survey method is a good exam-
ple of an integrated method. Finally, in the contingent method, the initial research
question will determine the form of the initial synthesis. The results of this study will
raise additional questions to be answered by further syntheses. These questions may
be best answered by a second synthesis, which in turn will result in yet more ques-
tions. Those questions will need to be answered by another synthesis, and so on until
the researchers are satisfied that the overall research problem has been addressed
sufficiently. The Bayesian meta-analysis and realist synthesis typify the contingent
design (Sandelowski et al., 2006).
Description and examples of systematic review syntheses
Systematic synthesis methodologies all rely on mathematical analytical methods to
aggregate evidence. These approaches aggregate outcome findings (meta-analysis,
vote counts, Bayesian meta-analysis), enumerate common themes (content analysis,
case survey) or use common themes to explain outcome findings (qualitative compar-
ative analysis). Though quite different in focus, they are classified together here
Table 1. Steps of different synthesis methodologies.
Methodology Search Selection
Analysis of
studies
Synthesis of
studies Discussion
Meta-analysis S S S S P
Vote count S S Depends S P
Case survey S S S S P
Content analysis S S P-S S P
Qualitative comparative Analysis P P S S P
Bayesian meta-analysis S P P S P
EPPI method S S S S P
Argument Catalogue S S S S P
Thematic analysis S-P P P P P
Grounded theory P P P P P
Meta-study S S S-P S P
Meta-synthesis S S P P P
Realist synthesis P P P P P
Traditional narrative review P P P P P
Meta-ethnography P P P P P
Notes. P: Purposeful;
S: Systematic;
S-P: From systematic to purposeful;
S & P: Both systematic and purposeful;
P-S: From purposeful to systematic.
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

Distance Education 241
because of their reliance on objective and systematic methods to extract meaning
from a body of evidence. Aggregation methods seek to reconcile variable findings of
quantitative research by averaging (meta-analysis) or tabulating (vote counts)
outcomes. Thematic methods rely on frequency counts of common themes to distill
meaning from bodies of evidence. In qualitative comparative analysis (QCA) the two
methods are combined. The presence or absence of specific themes or combinations
of themes helps to explain outcomes. Finally, in Bayesian meta-analysis, qualitative
and quantitative findings are analyzed and synthesized in terms of the probable
impact on decision-making. More detailed descriptions of the methodologies can be
found in Table 2.
As described earlier, both meta-analysis and vote counting methods have been
used to synthesize DE or OL studies. A study by So and Kim (2005) is an example of
the case +3study methodology applied to qualitative research in DE. The case survey
method as developed by Yin and Heald (1975) and later refined by Larsson (1993) was
Table 2. Systematic synthesis methodologies.
Methodology Description of method Comments
Meta-analysis Meta-analysis aims for greater
confidence in estimating the true
impact of interventions by
pooling results to establish one
grand average effect size that is
representative of the entire body
of evidence.
Pros: Can deal with large numbers of
studies; bias is minimized; provides
answers to large questions; moderator
analysis can begin to address effect
size variability. Cons: Does not
account for contextual impacts; only
includes high-quality quantitative
studies.
Vote count Studies are evaluated as reporting
positive or not positive findings
and tabulated. Studies aggregated
in terms of numbers of positive
findings versus numbers of not
positive findings.
Pros: Can deal with large numbers of
studies; bias is reduced, provides
answers to large questions; can
integrate quantitative and some
qualitative studies. Cons: Ignores
differences in magnitude of findings;
ignores differences in sampling; does
not account for context; is
reductionist.
Case survey Each study is analyzed according to
a variety of survey items, like
study feature coding in meta-
analysis, then the aggregate
survey data is analyzed.
Case survey can deal with large numbers
of diverse studies and is systematic in
its application. In addition, coder
reliability is easily established. On the
other hand, the case survey approach
is reductionist and gives equal weight
to all studies regardless of sample
size. This method is useful for scoping
reviews and for theory building.
Content
analysis
Content analysis involves the
reduction of text or other media
into content categories based on
strict rules of objective and
neutral coding (Weber, 1990;
Stemler, 2001). Code frequencies
are tabulated then analyzed using
a variety of quantitative
techniques.
Codes are precise so that results of
coding are replicable across coders,
thereby increasing validity and
reliability. Content analysis is suited
to diverse evidence synthesis, and can
be quite useful for theory building. Its
drawbacks include a reliance on
frequency of codes, a tendency toward
reductionism rather than importance,
context, or interpretation.
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

242 E.C. Bethel and R.M. Bernard
intended as a method for synthesizing large numbers of case study reports. This
method is analogous to survey research in that a survey is applied to each of the reports
and the survey data analyzed. Study feature coding in meta-analysis can be understood
to fall within this framework. Case survey has great potential for cross-genre synthesis,
can deal with large numbers of studies and is systematic in its application, and coder
reliability is easily established. On the other hand, like content analysis, the case
survey approach will always be reductionist. Additionally, because each study is being
treated as a single unit of analysis, the weighting issue between large-sample quanti-
tative studies and small-sample qualitative studies must be addressed. This method is
useful for scoping large bodies of outcome data and for theory building.
The purpose of the So and Kim (2005) study was to identify instructional methods
for collaborative learning in computer-supported learning environments that would be
useful for computer-supported collaborative learning (CSCL) researchers, instructional
designers, and online instructors. This study critically reviewed and analyzed 10 case
studies to identify instructional goals, methods, effectiveness, and conditions of collab-
orative OL. Twenty-three instructional methods, identified from the synthesis and
comparison of cases, were grouped into five categories representing commonalities:
(a) grouping; (b) collaborative tasks; (c) team-building; (d) computer-mediated
communication, and (e) instructor. It was found that some methods are equally impor-
tant for both face-to-face and CSCL environments. Instructional methods related
to group composition, synchronous interaction, and communication modes are
particularly critical for collaborative OL.
Table 2. (Continued).
Methodology Description of method Comments
Qualitative
comparative
analysis
This method involves determining
necessary and sufficient
conditions and combinations of
conditions for outcomes by
mapping the presence or absence
of variables onto attainment or
otherwise of the desired outcome
in a truth table, using Boolean
logic.
QCA is methodologically transparent
and systematic, and can incorporate
diverse types of studies. Like realist
synthesis, QCA allows for the testing
of competing theories of causality.
This method can be reductionist and
ignores any qualitative
interpretations. Like realist
synthesis, QCA allows for the testing
of competing theories of causality.
This method can be reductionist and
ignores any qualitative
interpretations.
Bayesian meta-
analysis
This approach treats research
findings in terms of their
potential impact on beliefs and
decision-making. Qualitative
findings are converted into
quantitative weights that impact
decision-makers’ prior beliefs,
which are then graphed in a
probability distribution. Using
Bayes’s theorem, these beliefs
are combined with the
quantitative data to determine
final probability distribution
describing probable outcomes.
Bayesian meta-analysis relies on
translating actors’ beliefs and
qualitative research findings into
quantitative weights. In this regard
the method lacks transparency and
practicality.
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

Distance Education 243
Description and examples of purposeful review syntheses
Although there are many purposeful methodologies, there is perhaps less diversity in
approach. Thematic synthesis is common to most of the purposeful methodologies.
Thematic analysis, grounded theory, meta-synthesis, narrative review, and meta-
ethnography, all synthesize research by comparing and contrasting common and
unique themes. These methodologies differ, if at all, in the way themes are extracted
and compared and contrasted. Some approaches require reviewers to compare themes
as they appear in the original studies (thematic synthesis). Others require that themes
be translated from one study to the next (meta-ethnography, meta-synthesis). Yet other
approaches look for broader, theoretically driven categories to capture commonalities
in themes (grounded theory, narrative review).
Theme-based synthesis
In thematic analysis, reviewers identify recurrent themes in the literature and summa-
rize findings according to thematic headings. In this regard, thematic analysis shares
common methods with content analysis, narrative summary, and grounded theory. One
of the key questions is whether themes must emerge and stay true to the text or whether
theory-driven themes are acceptable (Dixon-Woods et al., 2005). As for narrative
summary, efforts are being made to systematize the study selection process. Addition-
ally, researchers at the EPPI-Centre are developing systematic methods of coding and
data extraction for thematic synthesis of qualitative texts (Thomas & Harden, 2008).
Texts are coded line-by-line, and descriptive themes are extracted and collated.
Though transparent and systematic, it is unclear whether Thomas and Harden’s method
could be applied to quantitative studies. Moreover, line-by-line coding is unlikely to
be practical for large numbers of studies. The thematic analysis approach is developing
into a flexible and structured way to analyze potentially diverse evidence types.
Thematic synthesis has the potential to provide a well-organized way of describing
large and potentially diverse evidence bases, but it is not intended to develop explana-
tions of observed data patterns (Dixon-Woods, Agarwal et al., 2004). Table 3 is a
description of the methods that are classified as purposeful.
A study by Waight, Willging, and Wentling (2004) is an example of a theme-based
synthesis, or a narrative analysis, as the authors call it. Fifteen major e-learning reports
were used to provide a basic insight on what government, business, and professional
organizations are saying about e-learning. There is no pretense of systematicity or
comprehensiveness, as the reports were gathered through limited searches within a
narrow timeframe. The researchers were trying to determine the purpose of e-learning,
the features of e-learning, and the trends affecting e-learning. In this respect the study
was similar to the Argument Catalogue approach used by Abrami, Bernard, and Wade
(2006). The reviewers used a four-step process for developing and analyzing themes:
(1) the first three pages of each report were read to determine e-learning sources
(2) software called Theme Weaver was used to identify key words and phrases
(3) software called NVivo was used to locate the specific paragraphs containing
the themes identified by Theme Weaver
(4) the research questions were used to identify ‘emergent themes’.
In all, seven trends appeared in all 15 reports. These trends all capitalize on technol-
ogy, education, human capital, and the intertwined impact of these three. E-learning
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

244 E.C. Bethel and R.M. Bernard
Table 3. Purposeful synthesis methodologies.
Methodology Description of method Comments
Thematic analysis Reviewers identify recurrent themes
in the literature and summarize
findings according to thematic
headings. In this regard, thematic
analysis shares common methods
with content analysis, narrative
summary and grounded theory.
Texts are coded line by line, and
descriptive themes are extracted
and collated.
The thematic analysis approach is
developing into a flexible and
structured way to analyze
potentially diverse evidence
types. Thematic synthesis has
potential to provide a well-
organized way of describing
large and potentially diverse
evidence bases, but it is not
intended to develop explanations
for observed data patterns. It is
unclear whether this could be
applied to quantitative studies.
Moreover, line-by-line coding is
unlikely to be practical for large
numbers of studies.
Grounded theory The constant comparison method of
Glaser and Strauss’s (1967) and
later Strauss and Corbin’s (1998)
grounded theory methodology is
a well-established and rigorous
method that could be used for
literature synthesis. Instead of
original texts and transcripts,
primary studies become the unit
of analysis.
Grounded theory has only been
used to synthesize qualitative
studies, though more diverse
data sources could be included.
Grounded theory is particularly
interesting as a review method as
it seeks to provide explanations
for observed phenomena.
Additionally, sampling criteria
are made explicit, and sampling
to theoretical saturation has the
benefit of limiting the sample
size for the review. Grounded
theory is still dependent on the
reviewer for critical analytical
decisions. Further, the term
grounded theory has become a
catch phrase for a variety of
methodologies. Grounded theory
is best suited to building testable
hypotheses about the nature and
scope of a problem.
Meta-study The reviewer conducts a three-part
overview: meta-data synthesis,
meta-method synthesis, and
meta-theory synthesis, and then
synthesizes the ideas that have
emerged. Meta-data refers to the
synthesis of report data: ‘what
happened’. Meta-method
synthesis analyzes how various
methodologies impact
interpretation and understanding.
Meta-theory synthesis explores
the theoretical underpinnings of
the research (Dixon-Woods et al.,
2005).
Meta-study is highly systematic,
but flexible regarding methods,
and can include all types of
evidence. In this regard,
however, meta-study depends on
the methodological quality of
included studies. Meta-study
synthesis is best suited for
building theories about how we
understand interventions and
perceive them to be working or
otherwise.
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

Distance Education 245
Table 3. (Continued).
Methodology Description of method Comments
Meta-synthesis Meta-synthesis uses both qualitative
and quantitative studies as data or
unit of analysis. It seeks to
understand and describe key
points and themes. Meta-
synthesis enables the researcher
to represent and account for
differences in conceptualizations
and measurements of the research
problem. It can also be used for
synthesizing syntheses (see
section on Summative Methods).
The steps of meta-synthesis include
(1) search for articles; (2) make
decision on inclusion; (3)
appraise studies; (4) analyze
studies including ‘translation’ of
different conceptualizations and
comparisons, and (5) synthesize
findings (Walsh & Downe,
2005). Though originally
intended for the synthesis of
qualitative research, this method
can be used for synthesis of
qualitative, quantitative and
mixed methods studies.
Realist synthesis A realist synthesis seeks to
understand the theories that drive
interventions and to test how well
those theories work in a given
contexts and for whom Five
questions are asked of the
theories: (1) What are the theories
to be tested?; (2) Does the
intervention work as intended?;
(3) Which theories fit best?; (4)
How does the intervention work
in different settings with different
groups?; (5) How does the
intended intervention actually
translate into practice? The
theories to be tested serve as
guides for the sampling of
intervention reports to be
included in the synthesis. Both
confirmatory and contradictory
data sought to help determine
program limits (Pawson et al.,
2005).
This approach to synthesis is well
suited to and depends on the
analysis of diverse types of
evidence. No specific directions
guide the selection of evidence
for inclusion (Dixon-Woods et
al., 2005). Though realist
synthesis appears to hold real
promise, only its creator has
actually used it in practice.
Realist syntheses are best suited
for analyzing the interaction of
interventions and contexts (Mays
et al., 2005).
Traditional
narrative
review
This method involves the ‘informal
selection, assembly and summary
of studies for review’, and usually
includes an interpretive
commentary (Dixon-Woods,
Agarwal et al., 2004). Before the
evolution of systematic reviewing
techniques, narrative summary
was the most common method of
literature review.
Although systematic literature
selection techniques have been
incorporated, summary and
interpretive techniques can vary
from study to study and can
depend on the skill and bias of
the reviewer (Dixon-Woods et
al., 2005). On the other hand,
because of its flexibility, this
methodology can adapt to
accommodate large bodies of
potentially diverse evidence.
Narrative reviews are best suited
to answer questions related to
stakeholder opinions and
attitudes (Mays et al., 2005).
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

246 E.C. Bethel and R.M. Bernard
was also identified in 12 reports as a mechanism for sustaining the growth of the intel-
lectual capital while 10 reports cited the introduction of business models on e-learning
by investors.
Realist synthesis
A realist synthesis seeks to understand the theories that drive interventions and to test
how well those theories work in a given context and for whom (Pawson, Greenhalgh,
Harvey, & Walshe, 2005). Five questions are asked of the theories:
(1) What are the theories to be tested?
(2) Does the intervention work as intended?
(3) Which theories fit best?
(4) How does the intervention work in different settings with different groups?
(5) How does the intended intervention actually translate into practice?
The theories to be tested serve as guides for the sampling of intervention reports to be
included in the synthesis. As for grounded theory, both confirmatory and contradic-
tory data are sought to help determine program limits (Pawson et al., 2005). This
approach to synthesis is well suited to and depends on the analysis of diverse types of
Table 3. (Continued).
Methodology Description of method Comments
Meta-ethnography Meta-ethnography as developed by
Noblit and Hare (1988) involves
three major strategies:
(1) Reciprocal translation analysis
(RTA). The key metaphors,
themes, or concepts in each
study are identified. An attempt
is then made to translate these
into each other. Some analogies
can be drawn between RTA and
content analysis.
(2) Refutational synthesis. Key
metaphors, themes or concepts
in each study are identified, and
contradictions between the
reports are characterised.
Possible ‘refutations’ are
examined and an attempt made
to explain them.
(3) Lines of argument synthesis
(LOA). This involves building a
general interpretation grounded
in the findings of the separate
studies. Some analogies can be
drawn between LOA and the
constant comparative method.
(Dixon-Woods et al., 2005,
p. 48).
Meta-ethnography is a systematic
approach to synthesis that has
been successfully demonstrated
with qualitative studies. Meta-
ethnography could be applied to
quantitative studies as well.
Meta-ethnography works best
when the component studies are
methodologically similar.
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

Distance Education 247
evidence. No specific directions guide the selection of evidence for inclusion (Dixon-
Woods et al., 2005). Though realist synthesis appears to hold real promise, only its
creator has actually used it in practice. Realist syntheses are best suited for analyzing
the interaction of interventions and contexts (Mays et al., 2005).
Wong, Greenhalgh, and Pawson (2010) performed a realist synthesis with the
intention of producing theory-driven criteria to guide the development and evaluation
of Internet-based medical courses. They describe a realist synthesis as ‘a qualitative
systematic review method whose goal is to identify and explain the interaction
between context, mechanism and outcome.’ (p. 1). The authors searched 15 electronic
databases and references of included articles, trying to identify theoretical models of
how the Internet might support learning from empirical studies that (a) used the Inter-
net to support learning; (b) involved doctors or medical students, and (c) reported a
formal evaluation. All study designs and outcomes were considered, including quan-
titative primary research reports and meta-analyses. Using immersion (i.e. becoming
very familiar with studies) and interpretation (i.e. identifying and ‘testing’ theories
qualitatively), they tested theories by considering how well they explained the differ-
ent outcomes achieved in different educational contexts. They found 249 papers that
met their inclusion criteria, from which they identified two main theories of the
course-in-context that explained variation (i.e. as judged by the researchers) in learn-
ers’ satisfaction and outcomes: Davis’s technology acceptance model (learners were
more likely to accept a course if it offered a perceived advantage over available non-
Internet alternatives, was easy to use technically, and was compatible with their values
and norms) and Laurillard’s model of interactive dialogue (‘interaction’ led to effec-
tive learning only if learners were able to enter into a dialogue with an instructor, other
students or virtual tutorials, and gain formative feedback).
Meta-ethnography
Meta-ethnography was originally developed by Noblit and Hare (1988) as an alterna-
tive to meta-analysis, which they saw as conceptually inappropriate for the synthesis
of ethnographic studies. Meta-analysis depends on aggregation of data across studies
and hence can strip context from and obscure the uniqueness of individual studies.
In practice, meta-ethnography involves three major strategies:
(1) Reciprocal translation analysis (RTA). The key metaphors, themes, or concepts
in each study are identified. An attempt is then made to translate these into each
other. Some analogies can be drawn between RTA and content analysis.
(2) Refutational synthesis. Key metaphors, themes or concepts in each study are
identified, and contradictions between the reports are characterised. Possible
‘refutations’ are examined and an attempt made to explain them.
(3) Lines of argument synthesis (LOA). This involves building a general interpre-
tation grounded in the findings of the separate studies. Some analogies can
be drawn between LOA and the constant comparative method. (Dixon-Woods
et al., 2005, p. 48).
Britten et al. (2002) tested this methodology on four qualitative studies examining the
lay impressions of medicine. Key concepts were tabulated and translated from one
study to another. Second- and third-order interpretations were constructed, enabling
the authors to propose new hypotheses.
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

248 E.C. Bethel and R.M. Bernard
Meta-ethnography is a systematic approach to synthesis that has been successfully
demonstrated with qualitative studies. Although theoretically, meta-ethnography
could be applied to quantitative studies as well, this has not been demonstrated in
practice. Furthermore, meta-ethnography works best when the component studies are
methodologically similar. Meta-ethnography is best applied to questions of reaction
to, and acceptance of, interventions (Mays et al., 2005).
Major (2010) conducted a study of online teaching using meta-ethnography to
examine nine original qualitative studies (involving 23 researchers conducting inter-
views with 117 university faculty members with online teaching experience) to
provide a focused and detailed picture of faculty perceptions of teaching online. The
purpose of this research was to ‘make meaning’ of individual studies that investigated
faculty experiences of teaching online by considering the studies as an aggregate. The
researcher used analysis techniques common to meta-ethnographic approaches,
including reciprocal translation analysis (translating themes into each other), refuta-
tions synthesis (attempts to explain variations and contradictions), and lines-of-
argument analysis (building a general interpretation from findings of separate studies
through reliance on qualitative analysis such as constant comparison). The findings
reveal that teachers believe that online teaching changes the way instructors approach
and think about teaching, course design, time, instruction and students.
Description and examples of mixed-method review syntheses
Both the EPPI-Centre and Argument Catalogue methodologies take a ‘horses for
courses’ approach to evidence synthesis (Petticrew & Roberts, 2003). That is, in both
methods, evidence is categorized by type, and then categories are synthesized, creat-
ing several smaller syntheses. At the end all these smaller syntheses are reconciled
into a grand synthesis of syntheses. The main difference between the EPPI method
and the Argument Catalogue is that the EPPI method focuses on two syntheses: a
meta-analysis and a synthesis of qualitative evidence (usually from stakeholders’
viewpoints). The qualitative synthesis is then used to explain the findings of the meta-
analysis. The Argument Catalogue, on the other hand, collects and synthesizes all
available evidence on a particular topic, including research reports and studies, trade
articles, policy documents, magazine and newspaper articles, and websites. Like-
evidence is synthesized together and a final synthesis attempts to draw lessons out of
a body of syntheses. The main purposes of the Argument Catalogue is to provide an
outline of all the evidence available, highlight new directions for research, and act as
a policy guide. Table 4 is a description of the purposes and methodologies associated
with mixed-method syntheses.
Of the two mixed method syntheses approaches, the Argument Catalogue has been
used to review DE/OL literature. The Argument Catalogue is a tool for compiling
evidence systematically on a topic of interest. An Argument Catalogue goes beyond a
standard review of the research literature. It represents a means for analyzing and
characterizing documents from multiple sources, including the print media (a reflec-
tion of current public opinion), practitioners, policy-makers, reviewers of research,
and primary researchers. We use a recent state-of-the-field review of e-learning in
Canada to illustrate the steps in undertaking the development of an Argument
Catalogue (Abrami, Bernard, Wade, Schmid et al., 2006). These include defining the
problem, establishing inclusion/exclusion criteria, identifying the literature and proce-
dures for document retrieval, creating a codebook and coding documents, analyzing
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

Distance Education 249
data both quantitatively and qualitatively, analyzing and interpreting data, and dissem-
inating the results. A more detailed account of the methodology is described in
Abrami, Bernard, and Wade (2006).
This method was developed by researchers at the CSLP, to provide a compre-
hensive review of all the evidence, of whatever ilk, in a given domain (Abrami,
Bernard, & Wade, 2006). Evidence from the popular press, practitioner documents,
policy documents, reports of primary research, and reviews of primary research
were analyzed using techniques appropriate to the evidence type. The Argument
Catalogue sought to provide a rough sketch of the evidence, gaps, and promising
directions in e-learning from 2000 onwards, with a particular focus on Canada. The
findings were comprehensive, documenting mean effect sizes of outcome studies,
an analysis of the distribution of research efforts, a reflection of media opinion, and
an outline of policy direction. Methods included meta-analysis, vote counting,
content and thematic analysis, and narrative summary. What was particularly illu-
minating was the comparison and contrasts between the research findings, policy,
and media reports. Analyses of this sort can prove invaluable for policy-makers in
the future.
Syntheses of syntheses
There comes a time when the weight of multiple syntheses becomes so heavy that a
more general answer is needed to a research question or given topic. This is especially
the case when multiple syntheses have reached somewhat different conclusions or
there is a desire to cap a particular era of research with a summative conclusion. It is
arguable that this needs to be done after the last meta-analysis was performed in the
DE/OL classroom comparative literature (US Department of Education, 2009), since
it is unlikely that this form of research will prevail in the future as it has in the past.
Many of the methods previously described might be used to synthesize syntheses, but
two have appeared recently.
Table 4. Description of mixed-method synthesis techniques.
Methodology Description of method Comments
EPPI method This method involves a meta-
analysis of quantitative evidence
(intervention studies), a
qualitative synthesis of
perceptions (non-intervention
studies) and a cross-study
synthesis that allows the theories
developed in the qualitative
synthesis to inform and provide
context for the meta-analysis.
Qualitative synthesis could involve any
of the techniques discussed for
synthesizing qualitative data. The
EPPI method is well suited to
synthesis of diverse evidence as only
like types of evidence are directly
combined.
Argument
Catalogue
This method is designed to provide a
comprehensive review of all the
evidence in a given domain.
Different types of evidence are
analyzed using appropriate
techniques. Then syntheses are
themselves synthesized.
Methods include but are not limited to
meta-analysis, vote counting, content
and thematic analysis, and narrative
summary. Particularly useful is the
potential for comparison and contrasts
between the different types of
evidence including research findings,
policy, and media reports.
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

250 E.C. Bethel and R.M. Bernard
The first that we will describe is referred to as ‘second-order meta-analysis’ and
its intention is to collect and meta-analyze meta-analyses rather than studies. Tamim,
Bernard, Borokhovski, Abrami, and Schmid (2009) summarized 25 meta-analyses
dating back to 1990 that attempted to answer the question ‘Does technology integration
in classrooms affect performance outcomes?’ The studies were synthesized quantita-
tively and an overall average effect size and estimate of heterogeneity was produced.
In addition, overlap (i.e. the same study in more than one meta-analysis) was reduced
among studies to 25%, and the meta-analyses were evaluated qualitatively for meth-
odological quality. The average effect size was +0.32 (low to moderate) and the collec-
tion was significantly heterogeneous. After the comprehensive meta-analysis was
completed, a validation study was conducted among 574 primary studies drawn from
the meta-analyses. The average effect size was +0.30 and heterogeneous. This
suggested that the comprehensive meta-analysis had provided an accurate picture of
the state of this kind of study.
Strobel and van Barneveld (2009) used a qualitative alternative approach (i.e. an
adaptation of meta-synthesis) to synthesizing meta-analyses. See Table 3 for a
description of the method as it is usually used to synthesize primary studies. This was
a qualitative assessment of meta-analyses conducted since 1992 asking about the
effectiveness of problem-based learning (PBL) as compared with traditional teaching.
The research questions were:
(1) How do differences in (a) the definition of learning and (b) the measurement
of learning contribute to the inconclusiveness of the different meta-analyses
with regard to the effectiveness of PBL?
(2) Taking the differences into consideration, what generalizable value statements
about the effectiveness of PBL can be made and are supported by the majority
of meta-analyses? (p. 45)
The methods in this meta-synthesis mirrored those of a meta-synthesis of study-level
data (i.e. question description, inclusion/exclusion criteria, searching, and retrieval).
How learning was conceptualized helped to explain divergent results: findings from
eight studies that met the inclusion/exclusion criteria revealed that PBL is more effec-
tive for achieving long-term learning goals (i.e. recall, skill development) and student/
teacher satisfaction, while traditional instruction was more effective for short-term
retention as measured on standardized board exams.
Brief reviews of evidence (aka rapid reviews)
There is a growing desire for reviews that target specific issues and that can be done
in a short period of time. Policy-makers, in particular, often need information quickly,
but there is always the concern, when time is of the essence, that the review will not
be as accurate as a carefully conceived and executed synthesis. The EPPI-Centre, in
particular, has been a leader in this domain. Recently, Abrami et al. (2010) addressed
this issue by attempting to expose the range of issues and concerns (e.g. substantive,
methodological) that might arise when compromises must be introduced into the
review process. They define ‘brief review’ in the following way:
A brief review is an examination of empirical evidence that is limited in its timeframe
(e.g., six months or less to complete) and/or its scope, where scope may include the
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

Distance Education 251
breadth of the question being explored (e.g., a review of one-to-one laptop programs
versus a review of technology integration in schools); the timeliness of the evidence
included (e.g., the last several years of research versus no time limits); the geographic
boundaries of the evidence (e.g., inclusion of regional or national studies only versus
international evidence); the depth and detail of analyses (e.g., reporting only overall find-
ings versus also exploring variability among the findings); and otherwise more restrictive
study inclusion criteria than might be seen in a comprehensive review. (Abrami et al.,
2010, p. 372).
One of the alternatives for producing an answer to a question quickly is to conduct a
second-order meta-analysis or a meta-synthesis, as was described in the previous
section. The time saving in either of these approaches, as compared with a full synthe-
sis of primary literature, is considerable. Another is to perform a ‘Stage I’ review from
a random sample of studies from the literature. Schmid et al. (2009) conducted this
kind of review of the primary literature addressing technology integration in higher
education. They found an overall weighted effect size of +0.28 (based on 231 studies
and 310 effect sizes) surrounded by wide variability. This is similar to the average
effect reported by Tamim et al. (2009), discussed in the previous section. It is clear
that there are many issues to be discussed and reconciled regarding the nature and
quality of brief reviews.
Conclusion
Not unlike other areas in education, the research literature of DE and OL has become
increasingly diversified. No longer are classroom comparative studies, so numerous in
the years since about 1990, accepted as the best way of studying DE and OL. Simi-
larly, no longer is meta-analysis accepted as the only way of synthesizing the large
number of studies emanating from research efforts worldwide. We have come to real-
ize that the growing body of non-experimental quantitative and qualitative studies is
in need of methods of synthesis that reflect the various purposes and diverse kinds of
evidence they provide. In response, the number and variety of research synthesis
methodologies continue to increase.
As has been demonstrated, many new approaches have been proposed, each of
which is best suited to a particular evidence type or mix of types. Two factors should
guide the choice of synthesis methodology: the nature of the evidence and the purpose
of the synthesis. The systematic-to-purposeful continuum is one way of organizing a
potentially confusing array of methods by aligning them with the intent of the synthe-
sis and the types of evidence to which they are best suited. In general, the greatest care
must be taken with integrative synthesis methods, as these involve the direct combi-
nation of potentially incompatible data. Ironically, integrative methods recommend
themselves to policy-makers, as they tend to produce more definitive, less ambiguous
findings than the multi-layered segregated and contingent approaches. This is not the
final word, however, as synthesis methods continue to evolve. Rather this should be
seen as a continuing effort to bring order to an emerging approach to research.
Notes on contributors
Edward C. Bethel is a doctoral candidate in the Ph.D. program in educational technology in the
Department of Education at Concordia University. His dissertation (in progress) is a systematic
review of one-to-one K-12 laptop programs in order to identify the factors that influence
program outcomes and cost.
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

252 E.C. Bethel and R.M. Bernard
Robert M. Bernard is a professor of education, member of the Centre for the Study of Learning
and Performance at Concordia University, and the Systematic Review Theme Leader for the
CSLP. His research interests include DE, OL, and technology applications for teaching and
learning.
References
Abrami, P.C., & Bernard, R.M. (2006). Research on distance education: In defense of field
experiments. Distance Education, 27(1), 5–26. doi: 10.1080/01587910600653116
Abrami, P.C., Bernard, R.M., & Wade, C.A. (2006). Affecting policy and practice: Issues
involved in developing an Argument Catalogue. Evidence & Policy: A Journal of
Research, Debate and Practice, 2(4), 417–437. doi: 10.1332/174426406778881737
Abrami, P.C., Bernard, R., Wade, A., Schmid, R.F., Borokhovski, E., Tamin, R., Surkes, M.,
Lowerison, G., Zhang, D., Nicolaidou, I., Newman, S., Wozney, L., & Peretiatkowicz, A.
(2006). A review of e-learning in Canada: A rough sketch of the evidence, gaps and prom-
ising directions. Canadian Journal of Learning and Technology/La revue Canadienne de
l’apprentissage et de la technologie, 32(3). Retrieved 1 September 2010 from http://
www.cjlt.ca/index.php/cjlt/article/view/27/25
Abrami, P.C., Borokhovski, E., Bernard, R.M., Wade, C.A., Tamim, R., Persson, T., Bethel,
E.C., Hanz, K., & Surkes, M.A. (2010). Issues in conducting and disseminating brief
reviews of evidence. Evidence and Policy: A Journal of Research, Debate and Practice,
6(3), 371–389. doi: 10.1332/174426410X524866
Allen, M., Bourhis, J., Mabry, E., Burrell, N., & Mattrey, E. (2002). Comparing student
satisfaction with distance education to traditional classrooms in higher education: A meta-
analysis. American Journal of Distance Education, 16(2), 83–97. doi: 10.1207/
S15389286AJDE1602_3
Allen, M., Mabry, E., Mattrey, M., Bourhis, J., Titsworth, S., & Burrell, N. (2004). Evaluating
the effectiveness of distance learning: A comparison using meta-analysis. Journal of
Communication, 54(3), 402–420. doi: 10.1111/j.1460-2466.2004.tb02636.x
Anderson, T. (2003). Getting the mix right: An updated and theoretical rationale for interac-
tion. International Review of Research in Open and Distance learning, 4(2). Retrieved
7 September 2010 from http://www.irrodl.org/index.php/irrodl/article/view/149/230
Bernard, R.M., & Abrami, P.C. (2009). Meta analysis of distance education and online learn-
ing research: Past, present and future. Invited presentation at the 15th annual Sloan-C
International Conference on Online Learning, 28–30 October 2009, Orlando, FL.
Bernard, R.M., Abrami, P.C., Borokhovski, E., Wade, A., Tamim, R., Surkes, M., & Bethel,
E.C. (2009). A meta-analysis of three interaction treatments in distance education. Review
of Educational Research, 79(3), 1243–1289. doi: 10.3102/0034654309333844
Bernard, R.M., Abrami, P.C., Lou, Y., & Borokhovski, E. (2004). A methodological morass?
How we can improve the quality of quantitative research in distance education. Distance
Education, 25(2), 176–198. doi: 10.1080/0158791042000262094
Bernard, R.M., Abrami, P.C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., Wallet, P.A.,
Fiset, M. & Huang, B. (2004). How does distance education compare with classroom
instruction? A meta-analysis of the empirical literature. Review of Educational Research
3(74), 379–439. doi: 10.3102/00346543074003379
Best Evidence Encyclopedia. (n.d.). Criteria for inclusion in the Best Evidence Encyclopedia.
Baltimore, MD: Johns Hopkins University. Retrieved from http://www.bestevidence.org/
methods/criteria.htm
Britten, N., Campbell, R., Pope, C., Donovan, J., Morgan, M., & Pill, R. (2002). Using meta-
ethnography to synthesize qualitative research: A worked example. Journal of Health
Services Research and Policy, 7(4), 209–215. doi: 10.1258/135581902320432732
Brunton, G., Thomas, J., Harden, A., Rees, R., Kavanagh, J., Oliver, S., et al. (2005). Promot-
ing physical activity amongst children outside of physical education classes: A systematic
review integrating intervention studies and qualitative studies. Health Education Journal,
64(4), 323–338. doi: 10.1177/001789690506400404
Campbell Collaboration. (n.d.). The Campbell Collaboration. Retrieved from http://www.
campbellcollaboration.org
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

Distance Education 253
Campbell, D.T., & Stanley, J.C. (1963). Experimental and quasi-experimental designs for
research. Chicago, IL: Rand McNally.
Campbell, D.T., & Stanley, J.C. (1966). Experimental and quasi-experimental designs for
research. Chicago, IL: Rand McNally.
Cavanaugh, C.S. (2001). The effectiveness of interactive distance education technologies in
K-12 learning: A meta-analysis. International Journal of Educational Telecommunica-
tions, 7(1), 73–88. Retrieved from http://www.editlib.org/j/IJET
Cavanaugh, C., Gillan, K.J., Kromrey, J., Hess, M., & Blomeyer, R. (2004). The effects of
distance education on K-12 student outcomes: A meta-analysis. Naperville, IL: Learning
Point Associates.
Coalition for Evidence-Based Policy. (2008). Social programs that work. Retrieved from
http://www.evidencebasedprograms.org/
Cochrane Collaboration. (2010). The Cochrane Collaboration. Retrieved from http://
www.cochrane.org/
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence
Erlbaum Associates.
Cook, D.A., Levinson, A.J., Garside, S., Dupras, D.M., Erwin, P.J., & Montori, V. (2008).
Internet-based learning in the health professions: A meta-analysis. Journal of the American
Medical Association, 300(10), 1181–1196. doi: 10.1001/jama.300.10.1181
Cooper, H.M., Hedges, L.V., & Valentine, J.C. (2009). The handbook of research synthesis
and meta-analysis. Thousand Oaks, CA: Sage.
Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., & Sutton, A. (2005). Synthesising qual-
itative and quantitative evidence: A review of possible methods. Journal of Health
Services Research & Policy, 10(1), 45–53.
Dixon-Woods, M., Agarwal, S., Young, B., Jones, D., & Sutton, A. (2004). Integrative
approaches to qualitative and quantitative evidence. London: Health Development
Agency.
Dixon-Woods, M., Shaw, R.L., Agarwal, S., & Smith, J.A. (2004). The problem of appraising
qualitative research. Quality and Safety in Health Care, 13(3), 223–225. doi: 10.1136/
qshc.2003.008714
Evans, D. (2003). Hierarchy of evidence: A framework for ranking evidence evaluating
healthcare interventions. Journal of Clinical Nursing, 12(1), 77–84. doi: 10.1046/j.1365-
2702.2003.00662.x
Evidence for Policy and Practice Information and Co-ordinating Centre. (2006–2009). Welcome
to EPPI-Centre. Retrieved from http://eppi.ioe.ac.uk/cms/
Eysenck, H.J. (1978). An exercise in mega-silliness. American Psychologist, 33(5), 517. doi:
10.1037/0003-066X.33.5.517.a
Freeman, M., deMarrais, K., Preissle, J., Roulston, K., & St Pierre, E.A. (2007). Standards of
evidence in qualitative research: An incitement to discourse. Educational Researcher,
36(1), 25–32. doi: 10.3102/0013189X06298009
Friedman, L. (2001). Why vote-count reviews don’t count. Biological Psychiatry, 49(2),
161–162. Retrieved from http://www.journals.elsevierhealth.com/periodicals/bps/home
Glaser, B.G., & Strauss, A.L. (1967). The discovery of grounded theory. New York, NY:
Aldine de Gruyter.
Glass, G.V. (1976). Primary, secondary and meta-analysis of research. Educational
Researcher, 5(10), 3–8. doi: 10.3102/0013189X005010003
Goldsmith, M.R., Bankhead, C.R., & Austoker, J. (2007). Synthesising quantitative and
qualitative research in evidence-based patient information. British Medical Journal, 61(3),
262–270. doi: 10.1136/jech.2006.046110
Gough, D. (2007). Weight of evidence: A framework for the appraisal of the quality and
relevance of evidence. Research Papers in Education, 22(2), 213–228. doi: 10.1080/
02671520701296189
Green, J.L., & Skukauskait , A. (2008). Comments on Slavin: Becoming critical readers: Issues
in transparency, representation, and warranting of claims. Educational Researcher, 37(1),
30–40. doi: 10.3102/0013189X08314828
Greenhalgh, T., Robert, G., MacFarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of
innovations in service organizations: Systematic review and recommendations. The
Milbank Quarterly, 82(4), 581–629. Retrieved from http://www.milbank.org/quarterly.html
e˙
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

254 E.C. Bethel and R.M. Bernard
Harden, A., Garcia, J., Oliver, S., Rees, R., Shepherd, J., Brunton, G., & Oakley, A. (2004).
Applying systematic review methods to studies of people’s views: An example from
public health research. Journal of Epidemiology & Community Health, 58(9), 794–800.
doi: 10.1136/jech.2003.014829
Hedges, L.V., & Olkin, I. (1980). Vote-counting methods in research synthesis. Psychological
Bulletin, 88(2), 359–369. doi: 10.1037/0033-2909.88.2.359
Higgins, J.P.T., & Green S. (Eds.). (2008). Cochrane handbook for systematic reviews
of interventions version 5.0.1 [updated September 2008]. Retrieved from http://www.
cochrane-handbook.org
Jahng, N., Krug, D., & Zhang, Z. (2007). Student achievement in online education compared
to face-to-face education. European Journal of Open, Distance and E-Learning. Retrieved
from http://www.eurodl.org/materials/contrib/2007/Jahng_Krug_Zhang.htm
Larsson, R. (1993). Case survey methodology: Quantitative analysis of patterns across case
studies. The Academy of Management Journal, 36(6), 1515–1546. Retrieved from http://
www.jstor.org/stable/256820
Lincoln, Y.S., & Guba, E.G. (1985). Naturalistic inquiry. Newbury Park, CA: Sage
Publications.
Lou, Y., Bernard, R.M., & Abrami, P.C. (2006). Media and pedagogy in undergraduate
distance education: A theory-based meta-analysis of empirical literature. Educational
Technology Research & Development, 5(2), 141–176. doi: 10.1007/s11423-006-8252-x
Machtmes, K., & Asher, J.W. (2000). A meta-analysis of the effectiveness of telecourses in
distance education. American Journal of Distance Education, 14(1), 27–46. doi: 10.1080/
08923640009527043
Major, C.H. (2010). Do virtual professors dream of electric students? University faculty expe-
riences with online distance education. Teachers College Record, 112(8). Retrieved from
http://www.tcrecord.org
Maxwell, J.A. (1992). Understanding and validity in qualitative research. Harvard Educational
Review, 62(3), 279–300. Retrieved from http://www.hepg.org/her
Mays, N., Pope, C., & Popay, J. (2005). Systematically reviewing qualitative and quantitative
evidence to inform management and policy-making in the health field. Journal of Health
Services Research & Policy, 10(Supplement 1), 6–20. doi: 10.1258/1355819054308576
Noblit, G.W., & Hare, R.D. (1988). Meta-ethnography: Synthesizing qualitative studies.
Thousand Oaks, CA: Sage.
Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005). Realist review – A new method
of systematic review designed for complex policy interventions. Journal of Health
Services Research and Policy, 10(Supplement 1), 21–34. doi: 10.1258/1355819054308530
Petticrew, M., & Roberts, H. (2003). Evidence, hierarchies, and typologies: horses for
courses. British Medical Journal, 57(7), 527–529. doi: 10.1136/jech.57.7.527
Pope, C., Mays, N., & Popay, J. (2007). Synthesizing qualitative and quantitative health
research: A guide to methods. Milton Keynes: Open University Press.
Rees, R., Kavanagh, J., Harden, A., Shepherd, J., Brunton, G., Oliver, S., & Oakley, A. (2006).
Young people and physical activity: A systematic review matching their views to effective
interventions. Health Education Research, 21(6), 806–825. doi: 10.1093/her/cyl120
Robinson, D.H. (2004). An interview with Gene V. Glass. Educational Researcher, 33(3),
26–30. doi: 10.3102/0013189X033003026
Russell, T.L. (2001). The no significant difference phenomenon. Retrieved from http://www.
nosignificantdifference.org/
Sandelowski, M. (2004). Using qualitative research. Qualitative Health Research, 14(10),
1366-1386. doi: 10.1177/1049732304269672
Sandelowski, M., Docherty, S., & Emden, C. (1997). Qualitative metasynthesis: Issues and
techniques. Research in Nursing & Health, 20(4), 365–371. doi: 10.1002/(SICI)1098-
240X (199708)20:4<365::AID-NUR9>3.0.CO;2-E
Sandelowski, M., Voils, C.I., & Barroso, J. (2006). Defining and designing mixed research
synthesis studies. Research in the Schools, 13(1), 29–40. Retrieved from http://
www.msera.org/rits.htm
Sandelowski, M., Voils, C.I., & Barroso, J. (2007). Comparability work and the management
of difference in research synthesis studies. Social Science & Medicine, 64(1), 236–247.
doi: 10.1016/j.socscimed.2006.08.041
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

Distance Education 255
Schmid, R.F., Bernard, R.M., Borokhovski, E., Tamim, R., Abrami, P.C., Wade, C.A.,
Surkes, M.A., & Lowerison, G. (2009). Technology’s effect on achievement in higher
education: A Stage I meta-analysis of classroom applications. Journal of Computing in
Higher Education, 21(2), 95–109. doi: 10.1007/s12528-009-9021-8
Shachar, M., & Neumann, Y. (2003). Differences between traditional and distance education
academic performances: A meta-analytic approach. International Review of Research in
Open and Distance Learning, 4(2). Retrieved 8 September 2010 from http://www.irrodl.
org/index.php/irrodl/article/view/153/234
Shadish, W., & Myers, D. (2004). Campbell collaboration research policy brief: Methods
policy briefs. Retrieved from http://camp.ostfold.net/artman2/uploads/1/Research_Design_
Policy_Brief.pdf
Sitzmann, T., Kraiger, K., Stewart, D., & Wisher, R. (2006). The comparative effectiveness of
web-based and classroom instruction: A meta-analysis. Personnel Psychology, 59(3),
623–664. doi: 10.1111/j.1744-6570.2006.00049.x
Slavin, R.E. (1986). Best-evidence synthesis: An alternative to meta-analytic and traditional
reviews. Educational Researcher, 15(9), 5–11. doi: 10.3102/0013189X015009005
Slavin, R.E. (2008). Perspectives on evidence-based research in education – What works?
Issues in synthesizing educational program evaluations. Educational Researcher, 37(1),
5–14. doi: 10.3102/0013189X08314117
Smith, G.C.S. (2003). Parachute use to prevent death and major trauma related to gravitational
challenge: Systematic review of randomised controlled trials. British Medical Journal,
327, Number 7429, 1459–1461. doi: 10.1136/bmj.327.7429.1459
So, H.-J., & Kim, B. (2005). Instructional methods for CSCL: Review of case studies. In
Proceedings of the 2005 conference on Computer support for collaborative learning:
learning 2005: The next 10 years! 30 May– 4 June 2005 Retrieved from http://
portal.acm.org/toc.cfm?id= 1149293
Stemler, S. (2001). An overview of content analysis. Practical Assessment, Research &
Evaluation, 7(17). Retrieved from http://PAREonline.net/getvn.asp?v=7&n=17
Strauss, A.L., & Corbin, J.M. (1998). Basics of qualitative research: Techniques and proce-
dures for developing grounded theory. Thousand Oaks, CA: Sage.
Strobel, J., & van Barneveld, A. (2009). When is PBL more effective? A meta-synthesis of
meta-analyses comparing PBL to conventional classrooms. The Interdisciplinary Journal
of Problem-based Learning, 3(1), 44–58. Retrieved from http://docs.lib.purdue.edu/ijpbl/
Tamim, R., Bernard, R.M., Borokhovski, E., Abrami, P.C., & Schmid, R.F. (2009). Effects of
technology on students’ achievement: A second-order meta-analysis. Paper presented at
the Campbell Collaboration Ninth Annual Colloquium, 18–20 May 2009, Oslo, Norway.
Thomas, J., & Harden, A. (2008). Methods for the thematic synthesis of qualitative research
in systematic reviews. BMC Medical Research Methodology, 8(1), 45. Retrieved from
http://www.biomedcentral.com/1471-2288/8/45
Thomas, J., Harden, A., Oakley, A., Oliver, S., Sutcliffe, K., Rees, R., Brunton, G., &
Kavanagh, J. (2004). Integrating qualitative research with trials in systematic reviews.
British Medical Journal, 328(7446), 1010–1012. doi: 10.1136/bmj.328.7446.1010
Torgerson, D.J., & Torgerson, C.J. (2008). Designing randomised controlled trials in
health, education and the social sciences: An introduction. Basingstoke: Palgrave
Macmillan.
US Department of Education. (2009). Evaluation of evidence-based practices in online learn-
ing: A meta-analysis and review of online learning studies. Washington, DC: Office of
Planning, Evaluation, and Policy Development. Retrieved from http://www2.ed.gov/
rschstat/eval/tech/evidence-based-practices/finalreport.doc
Ungerleider, C., & Burns, T. (2003). A systematic review of the effectiveness and efficiency
of networked ICT in education. A state of the field report to the Council of Ministers
of Education, Canada and Industry Canada (Unpublished report). Retrieved from http://
www.cmec.ca/Publications/Lists/Publications/Attachments/55/SystematicReview2003.
en.pdf
Waight, C.L., Willging, P., & Wentling, T.L. (2004) Recurrent themes in e-learning: A narra-
tive analysis of major e-learning reports. Quarterly Review of Distance Education, 5(3),
195–203. Retrieved from http://www.infoagepub.com/Quarterly-Review-of-Distance-
Education.html
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010

256 E.C. Bethel and R.M. Bernard
Walsh, D. & Downe, S. (2005). Meta-synthesis method for qualitative research: A literature
review. Journal of Advanced Nursing, 50(2), 204–211. doi: 10.1111/j.1365-2648.2005.
03380.x
Weber, R.P. (1990). Basic content analysis (2nd ed.). Newbury Park, CA: Sage.
What Works Clearinghouse. (2008). WWC procedures and standards handbook (version 2.0).
Princeton, NJ: US Department of Education, Institute of Education Sciences. Retrieved
from http://ies.ed.gov/ncee/wwc/references/idocviewer/doc.aspx?docid=19&tocid=1
Williams, S.L. (2006). The effectiveness of distance education in allied health science
programs: A meta-analysis of outcomes. American Journal of Distance Education, 20(3),
127–141. doi: 10.1207/s15389286ajde2003_2
Wong, G., Greenhalgh, T., & Pawson, R. (2010). Internet-based medical education: A realist
review of what works, for whom and in what circumstances. BMC Medical Education,
10(1), 12. Retrieved from http://www.biomedcentral.com/1472-6920/10/12
Yin, R.K., & Heald , K.A. (1975). Using the case survey method to analyze policy studies.
Administrative Science Quarterly, 20(3), 371–381. Retrieved from http://www.johnson.
cornell.edu/publications/asq/
Zhao Y., Lei, J., Lai, B.Y.C., & Tan, H.S. (2005). What makes the difference? A practical
analysis of research on the effectiveness of distance education. Teachers College Record,
107(8), 1836–1884. doi: 10.1111/j.1467-9620.2005.00544.x
Downloaded By: [Canadian Research Knowledge Network] At: 22:23 16 November 2010