ArticlePDF Available

Abstract and Figures

Problem-based learning (PBL) has been utilized for over 40 years in a variety of different disciplines. Although extensively researched, there is heated debate about the effectiveness of PBL. Several meta-analyses were conducted that provided a synthesis of the effects of PBL in comparison to traditional forms of instruction. This study used a qualitative meta-synthesis approach to compare and contrast the assumptions and findings of the meta-analytical research on the effectiveness of PBL. Findings indicated that PBL was superior when it comes to long-term retention, skill development and satisfaction of students and teachers, while traditional approaches were more effective for short-term retention as measured by standardized board exams. Implications are discussed.
Content may be subject to copyright.
The Interdisciplinary Journal of Problem-based Learning volume 3, no. 1 (Spring 2009)
6–28
The Interdisciplinary Journal of Problem-based Learning volume 3, no. 1 (Spring 2009)
44–58
When is PBL More E ective? A Meta-synthesis of Meta-analyses
Comparing PBL to Conventional Classrooms
Johannes Strobel and Angela van Barneveld
Abstract
Problem-based learning (PBL) has been utilized for over 40 years in a variety of di erent
disciplines. Although extensively researched, there is heated debate about the e ective-
ness of PBL. Several meta-analyses have been conducted that provide a synthesis of the
e ects of PBL in comparison to traditional forms of instruction. This study used a quali-
tative meta-synthesis approach to compare and contrast the assumptions and  ndings
of the meta-analytical research on the e ectiveness of PBL. Findings indicated that PBL
was superior when it comes to long-term retention, skill development and satisfaction of
students and teachers, while traditional approaches were more e ective for short-term
retention as measured by standardized board exams. Implications are discussed.
When is PBL More E ective? A Meta-synthesis of Meta-analyses
Comparing PBL to Conventional Classrooms
Problem-based learning (PBL) has been utilized as a design methodology for teaching in
domains like medicine, engineering, science, and economics for over 40 years. Although
a large number of research studies have been conducted to investigate the e ectiveness
of PBL instruction compared to other forms of instruction, there is no consensus on the
value of PBL, but rather a heated debate on its e ectiveness. A recent widely debated ar-
ticle in Educational Psychologist by Kirschner, Clark, and Sweller (2006) argued particularly
vehemently of the failure of PBL due to the claimed failure of minimal guidance. While it
is beyond the scope of this particular paper to debate all of the implications of Kirschner
et al.s paper (2006), this article addresses the speci c claim that PBL is unsuccessful and
ine ective for learning.
Several meta-analyses, conducted over the past 15 years, have speci cally investi-
gated and quanti ed the e ectiveness of PBL compared to traditional instruction (Albanese
When is PBL More E ective? 45
volume 3, no. 1 (Spring 2009)
& Mitchell, 1993; Vernon & Blake, 1993; Kalaian, Mullan, & Kasim, 1999; Dochy, Segers, Van
den Bossche, & Gijbels, 2003; Newman, 2003; Gijbels, Dochy, Van den Bossche, & Segers,
2005). Although de ning PBL similarly, these studies were not consistent in their  nd-
ings, particularly because of di erences in de ning e ectiveness of learning, and how
e ectiveness was measured.
Purpose
The purpose of this study was to synthesize the di erent meta-analyses, compare and
contrast di erent conceptualizations of learning and how it was measured, and identify
common and generalizable  ndings across the meta-analyses with regard to the e ec-
tiveness of PBL.
Our research questions were:
1. How do di erences in (a) the de nition of learning and (b) the measurement of
learning contribute to the inconclusiveness of the di erent meta-analyses with regard to
the e ectiveness of PBL?
2. Taking the di erences into consideration, what generalizable value statements
about the e ectiveness of PBL can be made and are supported by the majority of meta-
analyses?
What is Problem-based Learning?
PBL in its current form originated as a response to low enrollments and general dissatis-
faction with medical education (Barrows, 1996). Since its origin, PBL has been used in a
variety of disciplines and educational levels (see Savery [2006] for a history; see Savery &
Du y [1996] for an introduction; see Hung, Jonassen & Liu [2007] for a summary of the
research).
As Barrows (1996) noted, PBL has taken on a myriad of de nitions, pushed in part by
institutions wanting to re ne their particular approach. Maudlsey (1999) cautioned us not
to assume that those making use of the term, problem-based learning were all referring to
the same concept, especially since the use of problems as a teaching strategy does not
necessarily constitute a PBL-oriented instructional methodology.
One of Barrows’ most recent de nitions (2002) identi ed the following key compo-
nents of PBL:
Ill-structured problems are presented as unresolved so that students will generate
not only multiple thoughts about the cause of the problem, but multiple thoughts
on how to solve it.
A student-centered approach in which students determine what they need to learn.
It is up to the learners to derive the key issues of the problems they face, de ne
their knowledge gaps, and pursue and acquire the missing knowledge.
The Interdisciplinary Journal of Problem-based Learning
46 Johannes Strobel and Angela van Barneveld
Teachers act as facilitators and tutors, asking students the kinds of meta-cognitive
questions they want students to ask themselves. In subsequent sessions, guidance
is faded.
Authenticity forms the basis of problem selection, embodied by alignment to
professional or ‘real world’ practice.
For our study, we were guided by the de nition of problem-based learning put
forth by Barrows, as described above, and by Savery (2006) who indicated that it is “an
instructional (and curricular) learner-centered approach that empowers learners to con-
duct research, integrate theory and practice, and apply knowledge and skills to develop
a viable solution to a de ned problem” (p. 9).
In contrast to PBL, we considered traditional learning approaches to be large-class,
instructor-driven, lecture-based deliveries within a curriculum, which compartmentalized
the content (e.g., in medicine, the instruction would be broken down into pharmacology,
anatomy etc.) (Barrows, 2002).
Methodology
To answer our research questions about how the di erences in the de nitions and
measurements of learning contribute to the inconclusiveness about the e ectiveness of
PBL, we conducted a meta-synthesis (Bair, 1999) of existing meta-analyses. The goal was
to determine which generalizable value statements about the e ectiveness of PBL were
supported by the majority of meta-analyses.
A meta-synthesis is a qualitative methodology that uses both qualitative and quan-
titative studies as data or unit of analysis. It is primarily concerned with understanding
and describing key points and themes contained within a research literature on a given
topic” (Bair, 1999, p. 4). Since we gave emphasis to the interpretation of data and to the
understanding of the di erences in the conceptualizations of what constitutes e ective-
ness in PBL, we opted not to do a meta-meta-analysis, which would have meant quanti-
tatively synthesizing all e ect sizes into a single one (see Spitzer, 1991, for an introduction
to meta-meta-analysis). Rather, we chose a meta-synthesis approach because it allowed
us to represent and account for di erences in the conceptualizations and measurements
of PBL e ectiveness due to the qualitative orientation of the approach.
According to Walsh and Downe (2005), the steps of meta-synthesis include (a) search
for articles, (b) make decision on inclusion, (c) appraise studies, (d) analyze studies includ-
ing “translation” of di erent conceptualizations and comparisons, and  nally (e) synthesize
ndings. While meta-syntheses are traditionally used to synthesize qualitative research
ndings exclusively, Bair (1999) expanded the use to include the qualitative comparison
of quantitative, qualitative, and mixed-method studies.
When is PBL More E ective? 47
volume 3, no. 1 (Spring 2009)
Sample and process of selection
Our unit of analysis was a meta-analysis or systematic review. A meta-analysis is a
process of quantitatively synthesizing research results by using various statistical methods
to retrieve, select, and combine e ect sizes and results from previously separate but related
studies (see Bernard, Abrami, Lou, & Borokhovski, 2004, for a methodological discussion).
Meta-analysis uses e ect size as a metric for judging the magnitude of the standardized
di erence between a treatment and control condition in a large set of studies and may
also be used to judge the magnitude of the relationship (r2 or R2) between measured
variables in a large set of studies (see Leandro, 2005, for more details).
Four research databases, ERIC, PubMEd, PsychInfo, and Web of Science were searched
using the terms problem-based learning (in a variety of di erent forms), PBL, and meta-
analysis. The date parameter was set to include studies since 1992 (the  rst recorded
meta-analysis for PBL that met the date parameter appeared in 1993). Twenty- ve records
that met these three broad search parameters were selected for an initial review. Abstracts
were reviewed to ensure that articles met the additional criteria for inclusion, which were
(1) a meta-analysis or a systematic literature review (general term for studies that provide
overviews of primary studies that used explicit and reproducible methods [Greenhalgh,
1997]) that assessed multiple studies and either calculated or reported e ect sizes, (2)
contained comparisons of the e ectiveness of PBL versus traditional learning approaches,
or (3) focused on individual outcome measures rather than program evaluation. This  rst
review of the 25 selected records resulted in eight studies that met the inclusion criteria—
four meta-analyses and four systematic reviews. Then, the reference sections of those eight
studies were reviewed in search of meta-analytical studies that may have been missed in
the database searches. This resulted in  nding two additional meta-analyses. Out of these
ten studies, we excluded Norman and Schmidt (2000), because it was a response to an
existing meta-analysis without providing new data or newly analyzing existing data, and
Smits, Verbeek, and De Buisonje (2002), because the study did not compare traditional
instruction to PBL-oriented training. The total number of studies included in this paper
was eight meta-analyses and systematic reviews.
The research base on the e ectiveness of PBL is particularly rich and strong in the
eld of medicine. Similarly well developed is assessment in the  eld of medicine, which
allows comparisons of di erent instructional interventions on situated and standardized
test environments. Not surprisingly, the meta-analyses dealing with PBL draw heavily
from primary studies conducted in medicine, but contain studies from other domains
(e.g., economy, computer science) to warrant a rather generalizable statement on the
e ectiveness of PBL.
The Interdisciplinary Journal of Problem-based Learning
48 Johannes Strobel and Angela van Barneveld
Analysis
We looked for the reported quantitative  ndings and the narrative description of results
in the meta-analytic studies and synthesized, qualitatively, the  ndings that assessed ef-
fectiveness of PBL versus traditional approaches.
Instead of prede ning codes of e ectiveness, we used an open-coding approach
(Denzin & Lincoln, 2005) whereby the selected codes were derived from the research stud-
ies. We focused on the narrative reports of the  ndings to evoke the codes. In alignment
with our research questions, we sought categories that coded the conceptual de nitions
of learning linked to the measurement strategies and outcome patterns used to determine
e ectiveness of the teaching and learning approach. We focused on the interpretation
of narrative and qualitative reporting to support our meta-synthesis approach. Two ad-
ditional coding categories emerged. One addressed overall satisfaction with the learning
experience, and the other included patient management, a mix of knowledge and skills
beyond basic science knowledge and in-situ clinical performance. We created a correlation
matrix that captured the measures of e ectiveness and modifying variables reported in
each study and the speci c orientation of e ect sizes (positive or negative) of each vari-
able. After all the measures were identi ed, we grouped them into categories based on
similarity of measurement intent. To assist with the categorization, we made use of Dochy
et al.s (2003) de nitions that knowledge tests measured “the knowledge from facts and
the meaning of concepts and principles” (p. 537) and skill tests measured “to what extent
students can apply their knowledge” (p. 537).
We excluded data from our coding strategy if only one study reported results, and if
results focused on program evaluation rather than individual learning outcomes. Finally,
we looked for overall patterns in measures that tended to favor PBL, indicated by a positive
e ect size, and those that tended to favor traditional approaches to learning/teaching,
indicated by a negative e ect size.
Details on meta-analysis and systematic reports
With regard to de nitions of PBL, the reported meta-analyses consistently employed a
de nition of PBL in congruence with Barrows (1996) and Savery (2006), which guided the
conceptual framework of this paper as well.
Summarized here is an overview of each of the eight meta-analyses selected for
this paper, describing the research questions, the selection criteria that the researchers
employed, and the  ndings. All of the meta-analyses included additional research ques-
tions, which were not pertinent to our particular investigation, for example, the cost of
PBL compared to traditional classrooms. We summarized only the research revolving
around the e ectiveness of PBL as a learning strategy compared to the traditional class-
room approach.
When is PBL More E ective? 49
volume 3, no. 1 (Spring 2009)
Albanese and Mitchell (1993) focused on the English-language international litera-
ture from 1972 to 1992 to gain insight into the e ectiveness of PBL within the domain of
medical education. They reviewed ten studies that provided data on outcome measures
of basic science knowledge, measured by the National Board of Medicine Exam (NBME
1), and seven studies that reported outcome measures of clinical knowledge and perfor-
mance (NBME 2). NBME 1 assesses understanding and application of important concepts
of the sciences basic to the practice of medicine, with special emphasis on principles and
mechanisms underlying health, disease, and modes of therapy. NBME 2 assesses applica-
tion of medical knowledge, skills, and understanding of clinical science essential for the
provision of patient care under supervision and includes emphasis on health promotion
and disease prevention.
Research questions by Albanese and Mitchell (1993) included: Do PBL students
develop the cognitive sca olding necessary to easily assimilate new basic sciences in-
formation? To what extent are PBL students exposed to an adequate range of content?
Does faculty dislike PBL because of the concentrated time commitment required? Results
of assessments of basic science knowledge indicated an overall negative e ect size (ES),
meaning that students engaged in the traditional classroom learning approach tended
to perform better on the standardized tests (NBME 1). The authors augmented the results
with two additional points. The  rst was that standardized examinations “have been
criticized for providing only a measure of the examinees ability to recognize the correct
answer from a limited list of potentially correct answers and of being heavily oriented
toward recall” (p. 56). The second point was that, although the ES favored the traditional
approach and the expectation was that PBL students would not perform as well in the
area of basic science knowledge assessments, this assumption was “not always true
(p. 57). However, the authors took this tendency as evidence of support for inadequate
cognitive sca olding development on the part of PBL students, as well as support for the
idea that PBL students may not have adequate exposure to a range of content. Interest-
ingly, though, the results also indicated that PBL graduates did perceive that they were
disadvantaged relative to their traditional learning counterparts. However, they viewed
themselves as better prepared in self-directed learning skills, problem-solving, information
gathering, and self-evaluation techniques. Results also indicated that the rates at which
PBL graduates were selected for their  rst choice residency positions were higher than
for traditional program graduates.
Vernon and Blake (1993) focused on 22 studies within the period from 1970 to
1992. Their study selection parameters included all identi able research on health-related
educational programs that contained signi cant PBL emphasis. That is, the studies used
quantitative methods, provided data that compared PBL with more traditional educational
methods, and measured outcomes that were of an evaluative nature. They excluded stud-
ies that were only descriptive or provided no comparison of the two learning approaches,
The Interdisciplinary Journal of Problem-based Learning
50 Johannes Strobel and Angela van Barneveld
PBL and traditional. The purpose of their research was to summarize all available data that
compared PBL with more traditional education methods, to analyze variations via meta-
analytic techniques, and to review perceived strengths and weaknesses of research in this
eld. The results indicated that, in terms of academic achievement (knowledge tests), the
results for standardized NBME 1 assessment outcomes showed signi cant trends favoring
students engaged in the traditional learning approach. For clinical knowledge and per-
formance outcomes (NBME 2), results slightly favored the PBL students, while assessment
outcomes of clinical performance (observation-based supervisor ratings) signi cantly
favored the PBL students.
Berkson (1993) did a narrative review of 10 pre-1992 studies, seeking evidence of
the e ectiveness of the PBL curriculum in medical education. Her research questions
included: Does PBL teach problem solving better than traditional schools? Does PBL im-
part knowledge better than traditional schools? Does PBL enhance motivation to learn
medical science better than traditional schools? Does PBL promote self-directed learning
(SDL) skills better than traditional schools? Berkson’s review indicated that there was no
evidence to suggest that a PBL approach taught problem solving better than the traditional
approach. Results did not demonstrate an advantage of one approach over the other for
imparting knowledge. However, results indicated that students and faculty favored PBL.
In addition, academic achievement and knowledge assessment favored the traditional
approach, while clinical assessments favored PBL. With regard to academic process, PBL
students placed more emphasis on meaning (understanding) rather than reproduction
(memory), which was the opposite pattern from students engaged in traditional learning
methods. Berkson concluded that it was unlikely students will su er detrimental conse-
quences from participation in PBL programs.
Kalaian, Mullan, and Kasim (1999) focused on medical education studies from 1970
to 1997—22 studies on NBME 1 outcome measures, and 9 studies on NBME 2 outcome
measures. The purpose of the research was to examine outcomes from primary research
comparing impact of PBL and traditional curricula on NBME 1 and NBME 2. The set of
primary studies reviewed included studies examined by previous reviews, augmented
through online searches for studies within the 1970 to 1997 time parameter, and manual
searches of medical education journals published in 1997. The exclusion criteria eliminated
studies that did not provide data needed to compute ES for PBL and traditional learning
approaches, as well as studies that examined only speci c subtests of the NBME, rather
than the overall NBME. The researchers found negative ES for NBME 1, and positive ES for
NBME 2, which was consistent with previous  ndings that traditional learning approaches
tended to produce better results for basic science knowledge, while PBL tended to produce
better results for clinical knowledge and skills.
Colliver (2000) reviewed the medical education literature, starting with three reviews
published in 1993 (Albanese & Mitchell; Vernon & Blake; Berkson) and included studies
When is PBL More E ective? 51
volume 3, no. 1 (Spring 2009)
published from 1992 to 1998 comparing PBL to the traditional curriculum. The purpose
was to focus on the credibility of claims about ties between PBL intervention and edu-
cational outcomes, particularly achievement (knowledge and skills), and on e ect sizes
of the intervention on said outcomes. As a study selection strategy, Colliver’s search was
limited to those articles that involved a comparison of curriculum tracks or schools. Where
e ect sizes were not provided, Colliver calculated them himself. Results indicated that
there was no convincing evidence that PBL improved the knowledge base and clinical
performance, at least not to the extent that may be expected for a PBL curricular inter-
vention. Colliver acknowledged that PBL may provide a more challenging, motivating
and enjoyable approach to medical education, as noted in the earlier research  ndings
on student and faculty satisfaction and motivation, but claimed that its educational ef-
fectiveness, compared to traditional methods, remained to be seen.
Dochy, Segers, Van den Bossche, and Gijbels (2003) reviewed 43 studies, where
33 of them measured knowledge e ects and 25 of them measured application of knowl-
edge e ects. Their study selection criteria stipulated that the work had to be empirical.
Although nonempirical literature and literature reviews were selected as sources of relevant
research, this literature was not included in the analysis. The characteristics of the learn-
ing environment had to  t the core model of PBL. The dependent variables used in the
study had to comprise an operationalization of the knowledge or skills (i.e., knowledge
application) of the students. The subjects of study had to be students in tertiary educa-
tion. Also, the study had to be conducted in a real-life classroom or programmatic setting
rather than under more controlled laboratory conditions. Research questions were: What
are the e ects of PBL on knowledge and skills? What are the moderators on the e ects
of PBL? Results indicated that assessment methods that focus more on recognition (e.g.,
NBME 1), showed signi cant negative e ects for almost all knowledge and favored the
traditional learning approach. Assessment methods that focused more on application of
knowledge (e.g., NBME II) showed larger e ects for PBL versus traditional learning environ-
ments. Researchers stated that the better an instrument was able to evaluate students’
skills, the larger the ascertained e ects of PBL.
Newman (2003) selected studies cited in the following papers which provided evi-
dence of the e ectiveness of PBL: Albanese and Mitchell (1993); Vernon and Blake (1993);
Berkson (1993); Smits, Verbeek, and De Buisonje (2002a); and Van Den Bossche, Gijbels,
and Dochy (2000). The  nal count was12 studies with extractable data in the medical
education domain. The minimum criteria for study selection consisted of only including
participants in postschool education programs. Study designs had to be controlled tri-
als; studies that used only qualitative approaches were excluded. The minimum meth-
odological inclusion criteria across all study designs were the objective measurement of
student performance and behavior or other outcomes. The minimum inclusion criteria
for interventions consisted of a cumulative integrated curriculum, learning via simula-
The Interdisciplinary Journal of Problem-based Learning
52 Johannes Strobel and Angela van Barneveld
tion formats that allowed free enquiry (i.e., not problem solving learning), small groups
with either faculty or peer tutoring, and an explicit framework implemented in tutorials.
Research questions included: Does PBL result in increased participant performance when
compared to other non-PBL teaching and learning strategies? Does an authentic PBL cur-
riculum deliver a greater improvement in performance than “hybrid” curricula? Results
indicated that knowledge related outcomes favored the traditional learning environment.
Also consistent with previous  ndings, study approaches and student satisfaction tended
to favor PBL. However, improvements in applied practice returned mixed results, whereas
previous studies reported better outcomes in a PBL environment.
Gijbels, Dochy, Van den Bossche, and Segers (2005) reviewed 40 studies that were
published between 1976 and 2000. Study selection parameters stipulated that each study
had to be empirical. Second, the characteristics of the problem-based learning environ-
ment had to  t the previously described core model of PBL (Barrows, 1996). Third, each
study had to include some course or curriculum comparison between a PBL environment
and a more traditional educational setting. Fourth, the study subjects had to be students
in higher education. Finally, each study had to be conducted in a real-life classroom or
programmatic setting rather than under more controlled laboratory conditions. The re-
search question was: What are the e ects of PBL when the assessment of its main goals
focuses, respectively, on (1) understanding concepts, (2) understanding principles that
link concepts, and (3) linking of concepts and principles to conditions and procedures for
application? Results indicated that PBL students performed better at knowledge levels
that emphasized principles (understanding the link between concepts) and application
knowledge structures. The e ect size of PBL interventions was larger when the assess-
ment strategy focused on the understanding of principles that link concepts. Most studies
reported positive outcomes of the traditional classroom approach on conceptual knowl-
Figure 1. Map of learning outcomes.
When is PBL More E ective? 53
volume 3, no. 1 (Spring 2009)
edge assessment, but when weighted average ES was taken in to account, PBL students
performed at least as well as students in a traditional environment. This demonstrated
the potential in uence of the assessment strategy and tool on outcome measures. The
authors stated that the better the capacity of an instrument to evaluate the application
of knowledge by the student, the greater the ascertained e ect of PBL.
In summary, the  rst general tendency of noted in the research was that traditional
learning approaches tended to produce better outcomes on assessment of basic science
knowledge but, according to Albanese and Mitchell (1993), not always. A second trend
noted was that a PBL approach tended to produce better outcomes for clinical knowl-
edge and skills. Interestingly, more recent research studies revealed that the assessment
strategy and tool in uence outcome measures.
Results and Discussion
We grouped and collapsed the data and established four high-level categories based on
the assessment of learning outcomes. These four categories included:
Non-performance, non-skill-oriented, non-knowledge-based assessment
Knowledge assessment
Performance or skill-based assessment
Mixed knowledge and skill-based assessment
A map of e ectiveness measures is shown in Figure 1. A detailed correlation matrix can
be found in the appendix.
Trends in e ect sizes were reported as overall tendencies based on the data, where
the (+) symbol indicates that e ect sizes favored PBL, while the (-) symbol indicates that
e ect sizes favored the traditional teaching and learning approach.
In the category coded as Non-performance, non-skill, and non-knowledge-based, which
included student and faculty satisfaction measures, as well as successful assignment of
rst choice of residency, all the reported e ect sizes favored PBL.
For the Knowledge assessment category, measures of short-term knowledge acquisition
and retention returned mixed results, but tended to favor traditional learning approaches.
With assessments delivered immediately post-course (Albanese & Mitchell, 1993; Dochy
et al., 2003), outcomes of knowledge measures such as NBME 1 (assesses understanding
and application of important concepts of the sciences basic to the practice of medicine),
multiple choice questions, progress assessments using 250 True/False questions favored
the traditional learning approach (Newman, 2003, who only discusses NBME 1). However,
outcomes of knowledge measures that focused more on recall over recognition, such as
free recall, where students were asked to write down everything they remembered on a
topic, and short answer, which allowed for elaboration of answers, favored PBL (reported
by all other systematic reviews which discussed both NBME 1 and 2).
The Interdisciplinary Journal of Problem-based Learning
54 Johannes Strobel and Angela van Barneveld
Knowledge assessment that focused on long-term knowledge retention, described
by Albanese and Mitchell (1993) as a comparison of immediate post-course results and
results of the same test applied after a period of between 12 weeks to 2 years, returned
e ect sizes that consistently favored PBL. Dochy et al. (2003) looked only at whether a
retention period existed and compared knowledge outcomes on the basis of retention
period or no retention period. Long-term knowledge retention favored PBL.
The Performance or skill-based assessment category included observations with clini-
cal ratings (formative assessment by supervisor during and at the end of performance)
and case analysis measures. Clinical ratings favored PBL. The case analysis sub-category,
which included measures from the NBME 2 (assesses application of medical knowledge,
skills, and understanding of clinical science), patient simulations, and elaborated assess-
ments such as essay questions and case studies, also favored PBL.
The  nal category, Mixed knowledge and skill, captured results that required both
knowledge and skill for performance—oral examinations and the USMLE 3 (assesses
application of medical knowledge and understanding of biomedical and clinical science
essential for the unsupervised practice of medicine, with emphasis on patient manage-
ment). The outcomes in this category favored PBL.
Speci cally, to answer our  rst research question of how di erences in (a) the de ni-
tion of learning, and (b) the measurement of learning contribute to the inconclusiveness
of the di erent meta-analyses with regard to the e ectiveness of PBL, the discrepancy
in reported results on the e ectiveness of PBL for knowledge retention seemed to stem
particularly from the di erences in seeing learning as long-term (PBL favorable) and
short-term retention of knowledge (traditional teaching methods favorable). Addition-
ally, conceptualizations and consequently measurements of learning, which focused
on the performance and were skill-oriented, indicate that PBL students outperformed
traditionally taught students. The focus on short-term learning gains as a measurement
of PBL seem a particular mismatch considering that learning within an authentic context
is a key criterion of the de nition of PBL (Barrows, 2002).
Overall, students and sta indicated greater satisfaction with the PBL approach to
learning. Standardized tests that measured knowledge of basic science focusing on short-
term acquisition and retention (primarily the medical board exams in their di erent ver-
sions) favored the traditional approach across all studies. However, when the method used
to assess basic science knowledge required a level of elaboration beyond multiple-choice
or true/false questions, results signi cantly favored the PBL approach. Standardized tests
and other assessment methods that evaluated skill-oriented application of knowledge,
mixed knowledge and long-term retention of knowledge, skills, and clinical performance
signi cantly favored PBL.
As to our second research question, several value statements can be made about the
e ectiveness of PBL that were supported by the majority of the meta-analyses reviewed:
When is PBL More E ective? 55
volume 3, no. 1 (Spring 2009)
PBL instruction was e ective when it came to long-term retention and performance im-
provement. PBL students were overall slightly underperforming when it came to short-
term retention. Ultimately, the goal of instruction should be performance improvement
and long-term retention. Therefore, preference should be given to instructional strategies
that focus on students’ performance in authentic situations and their long-term knowl-
edge retention, and not on their performance on tests aimed at short-term retention of
knowledge.
Conclusion
Instruction is often designed based on the assumption that learning is “a similar process
in all individuals and for all tasks and thus many people feel a common instructional ap-
proach should su ce” (Clark 2000, p. 31). PBL is not the only successful strategy to achieve
e ective learning of ill-structured and complex domains. The results of these qualitatively
synthesizing meta-analyses of PBL for preparation for the workplace indicate, however,
that PBL is signi cantly more e ective than traditional instruction to train competent and
skilled practitioners and to promote long-term retention of knowledge and skills acquired
during the learning experience or training session.
Future directions
The vast majority of research on the e ectiveness of PBL has been conducted in the
training of professionals in the  eld of medicine. Similarly solid research base is needed
in other disciplines and contexts such as K-12 education, history, or engineering, to (a)
expand the use of PBL in the learning environment and (b) to more clearly de ne the
boundaries of its use.
Since the evidence suggests that PBL works in particular contexts, especially for
workplace learning with a focus on skills and long-term retention, the focus should shift
from researching e ectiveness of PBL versus traditional learning, and should refocus on
studying the di erences in e ectiveness of support structures to  nd optimal sca olding,
coaching, and modeling strategies for successful facilitation of PBL.
Acknowledgements
The authors want to thank the three anonymous reviewers for their feedback.
The Interdisciplinary Journal of Problem-based Learning
56 Johannes Strobel and Angela van Barneveld
References
Albanese, M. A., & Mitchell, S. (1993). Problem-based learning: A review of literature on its
outcomes and implementation issues. Academic Medicine, 68, 52-81.
Bair, C. R. (1999). Meta-Synthesis: A new research methodology. Paper presented at the An-
nual Meeting of the Association for the Study of Higher Education, November 18-21, San
Antonio, Texas, 26p (ERIC document no. ED 473 866).
Barrows, H. S. (1986). A taxonomy of problem-based learning methods. Medical Education,
20(6), 481-486.
Barrows, H. S. (1996). Problem-based learning in medicine and beyond: A brief overview. New
directions for teaching and learning, (68), 3-12.
Barrows, H. S. (2002). Is it Truly Possible to Have Such a Thing as dPBL? Distance Education,
23(1), 119-122.
Berkson, L. (1993). Problem-based Learning: Have the expectations been met? Academic
Medicine, 68(10), S79-S88.
Bernard, R. M., Abrami, P. C., Lou, Y. & Borokhovski, E. (2004). A methodological morass? How
we can improve the quality of quantitative research in distance education. Distance
Education, 25(2), 175-198.
Clark, R.C. (2000). Four Architectures of Instruction. Performance Improvement, 39(10), 31-38.
Colliver, J. A. (2000). E ectiveness of problem-based learning curricula: Research and theory.
Academic Medicine, 75(3), 259-266.
Denzin, N.K. & Lincoln, Y.S. (Eds.) (2005) The SAGE Handbook of Qualitative Research. Thousand
Oaks, Sage.
Dochy, Filip, Segers, Mien, Van den Bossche, Piet, & Gijbels, David (2003). E ects of problem-
based learning: a meta-analysis. Learning and Instruction, 13, 533–568.
Gijbels, David, Dochy, Filip, Van den Bossche, Piet, & Segers, Mien (2005). E ects of Problem-
Based Learning: A Meta-Analysis from the Angle of Assessment. Review of Educational
Research, 75(1), 27-61.
Greenhalgh, T. (1997) How to read a paper: Papers that summarise other papers (systematic
reviews and meta-analyses), British Medical Journal, 315, 672-675
Hung, W., Jonassen, D.H., & Liu, R. (2007). Problem-based learning. In J.M. Spector, J. G. van
Merriënboer, M.D., Merrill, & M. Driscoll (Eds.), Handbook of research on educational com-
munications and technology (pp. 1503-1581). 3rd Ed. Mahwah, NJ: Lawrence Erlbaum
Associates.
Kalaian, Hripsime A., Mullan, Particia B., & Kasim, Rafa M. (1999). What can studies of problem-
based learning tell us? Synthesizing and modeling PBL e ects on National Board of
Medical Examination performance: Hierarchical Linear Modeling meta-analytic approach.
Advances in Health Sciences Education, 4, 209-221.
Kirschner, Paul A., Sweller, John, & Clark, Richard E. (2006). Why minimal guidance during
instruction does not work: An analysis of the failure of constructivist, discovery, problem-
based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.
When is PBL More E ective? 57
volume 3, no. 1 (Spring 2009)
Leandro G. (2005) Meta-analysis in Medical Research: The Handbook for the Understanding
and Practice of Meta-analysis. Malden, MA: Blackwell.
Maudsley, Gillian (1999). Do we all mean the same thing by Problem-based Learning? A re-
view of the concepts and a formulation of the ground rules. Academic Medicine, 74(2),
178-185.
Newman, Mark (2003). Special Report 2: A pilot systematic review and meta-analysis on the
e ectiveness of Problem Based Learning. ITSN Learning and Teaching Support Network.
Middlesex University, UK.
Norman, G.R. & Schmidt, H.G. (2000). E ectiveness of problem-based learning curricula: theory,
practice and paper darts. Medical Education, 34, 9, 721-728.
Savery, J. R., & Du y, T.M. (1996). Problem based learning: An instructional model and its
constructivist framework. In B. G. Wilson (Ed.) Constructivist learning environments: Cases
studies in instructional design (pp.135-148). Englewood Cli s, NJ: Educational Technology
Publications.
Savery, John R. (2006). Overview of problem-based learning: De nitions and distinctions.
Interdisciplinary Journal of Problem-based Learning, 1(1), 9-20.
Smits, P.B.A., Verbeek, J.H.A.M. & de Buisonjé, C.D. (2002). Problem-based learning in continu-
ing medical education: A review of controlled evaluation studies. British Medical Journal,
324, 153-156.
Spitzer, W.O. (1991) Meta-meta-analysis. (Editorial). Journal of Clinical Epidemiology, 44,
103-107.
United States Medical Licensing Examination (USMLE). Exam descriptions retrieved on March
16, 2007 from http://www.usmle.org/default.asp.
Vernon, D.T.A. & Blake, R.L. (1993). Does problem-based learning work? A meta-analysis of
evaluative research. Academic Medicine. 68, 550-563.
Walsh, D. & Downe, S. (2005) Meta-synthesis method for qualitative research: A literature
review. Journal of Advanced Nursing 50(2), 204–211.
Johannes Strobel is Assistant Professor of Engineering Education and Educational Technology (joint
appointment) at Purdue University.
Angela van Barneveld works as a Program Manager in Global Education at IBM Canada and is pursu-
ing a PhD in Educational Technology.
The Interdisciplinary Journal of Problem-based Learning
58 Johannes Strobel and Angela van Barneveld
Appendix
Correlation Matrix
... One point of criticism against PBL is that there is insufficient structured theoretical knowledge taught in comparison to traditional teaching (see also Albanese & Mitchell, 1993;Carriger, 2016;Dochy et al., 2003;Hattie, 2015). However, several studies have shown that these disadvantages are less important in the long term (Schmidt et al., 2006;Strobel & van Barneveld, 2009). Instead, students who participated in courses with PBL components were often able to longer retain the learning contents than students who were taught by traditional lectures (Dochy et al., 2003;Strobel & van Barneveld, 2009;Yew & Goh, 2016). ...
... However, several studies have shown that these disadvantages are less important in the long term (Schmidt et al., 2006;Strobel & van Barneveld, 2009). Instead, students who participated in courses with PBL components were often able to longer retain the learning contents than students who were taught by traditional lectures (Dochy et al., 2003;Strobel & van Barneveld, 2009;Yew & Goh, 2016). ...
... According to literature, PBL improve in-depth content knowledge, critical thinking and problemsolving skills (Kwan & So, 2008), as well as research, communication and collaboration skills (Hmelo-Silver, 2004;Sekhon et al., 2021;Yeung, 2010;Yew & Goh, 2016). Compared to lecturebased teaching, PBL can improve the retention of concepts and skills in the long-term (Strobel & Barneveld, 2009) and increase students' motivation and confidence (Visser et al., 2002). In PBL, students are exposed to different, even conflicting, views which promotes open-mindedness (Yeung, 2010). ...
Article
Full-text available
Application of problem based learning in agriculture in Sub-Saharan Africa
... According to literature, PBL improve in-depth content knowledge, critical thinking and problemsolving skills (Kwan & So, 2008), as well as research, communication and collaboration skills (Hmelo-Silver, 2004;Sekhon et al., 2021;Yeung, 2010;Yew & Goh, 2016). Compared to lecturebased teaching, PBL can improve the retention of concepts and skills in the long-term (Strobel & Barneveld, 2009) and increase students' motivation and confidence (Visser et al., 2002). In PBL, students are exposed to different, even conflicting, views which promotes open-mindedness (Yeung, 2010). ...
Article
Full-text available
To solve the pressing concerns of climate change, land degradation, and food insecurity in Sub-Saharan Africa (SSA), a thorough understanding about the complex interactions between soils and the climate system is required. Problem-based learning (PBL) approach can enhance students’ trans-disciplinary understanding and application skills on soils and climate, enabling them to become future problem solvers. In this article, we discuss how to apply PBL in higher education in soil and climate change sciences in the context of SSA. First, we discuss the status quo of higher education in SSA and the need for a paradigm shift towards PBL. Secondly, we present the step-by-step implementation of PBL and our expectations on the benefits and hindrances of this approach.
... Sam tok realizacije istraživanja dodatno je fokusirao njihovu pažnju i uticao na konkretan napredak. Istraživanja pokazuju da se projektnim učenjem podstiče dublje razumevanje, odnosno podiže nivo pojmovne analize razumevanja načela i udubljivanja u materiju i kritičkog razmišljanja (Grant, 2002;Marzano, 2007;Strobel & van Barneveld, 2008). ...
Article
Full-text available
The study presented in this paper looks at how the problem of climate change can be adequately presented to students in the early grades of primary school, aiming to examine the relationship between project-based learning and the development of student competencies and achievements in this area. Project-based learning took place over the course of two months, as part of the elective environmental club attended by 30 fourth-grade primary school students. Students' initial and final knowledge about climate change was assessed using a test designed for this purpose. Data on the assessment of students' performance and the monitoring of their engagement were collected using a control list for tracking the development and use of student competencies (research, collaborative and personal), while students' views about the realized project were elicited with a questionnaire. The obtained results suggest an increase in students' achievement in the area of environmental content, while students themselves rate their experiences as positive. Systematic observation revealed the development of a series of abilities in students. Of the three categories of abilities, over the course of the project the greatest progress was observed in the category of research abilities, while in the other two categories progress was less pronounced, suggesting that a longer period is needed for their development as well as a more frequent implementation of projects in instructional practice.
... Sin embargo, las investigaciones en el área (por ejemplo, Strobel and van Barneveld, 2009) han demostrado que el Aprendizaje Basado en la Resolución de Problemas es mejor para la retención a largo plazo de los materiales y el desarrollo de destrezas "replicables". También es mejor para mejorar la actitud de los estudiantes hacia el aprendizaje en general. ...
... This requires students to develop their understanding and knowledge through experience. PBL instruction is effective for long-term retention and performance improvement (Strobel & van Barneveld, 2009). ...
Article
Full-text available
p class="p1">Cooperative problem-based learning (CPBL) has been introduced in engineering education as an effective technique for developing students' professional skills. Moreover, the method is essential for engaging students in learning and developing 21st century skills. However, students are usually "shocked" by such inquiry learning because they face sudden changes in the learning environment. In addition, its implementation in online learning is another challenge. Therefore, this study aims to identify students' perceptions of online CPBL performance in vocational pre-teacher education institutions. A self-reporting questionnaire was designed focusing on 21st century skills and distributed to 71 students after the end of the semester. Quantitatively, the instrument of the survey data was analysed using SPSS software. Quantitative data were analysed using descriptive statistics. The results show that online CPBL improved students’ collaboration, communication, and problem-solving skills. In general, they agreed that online CPBL improved the development of 21st century skills.However, the students faced some difficulties during the online discussion session due to internet connection problems and team member involvement. Details of the items are also discussed.</p
Article
Harnessing a cluster randomized controlled trial, we estimated the impact on students’ advanced placement (AP) examination performance of a project-based learning (PBL) approach to AP compared with a lecture-based AP approach. Through PBL, teachers primarily play a facilitator role, while students work on complex tasks organized around central questions leading to a final product. We estimated positive and significant treatment effects on AP exam performance for the overall sample, within both AP courses studied, and within low- and high-income student groups. Results support teacher-driven adoption of the PBL AP approach within both courses studied, among districts with open-enrollment AP policies and supportive of PBL, for students from low- and high-income households.
Thesis
Full-text available
The world faces crises that find an unprecedented match in contemporary history that create complex challenges and a society that struggles to solve them. Nevertheless, whilst being a time of crisis, it is also a time of opportunities for Design and Design Education to prepare future designers to assume a position of change agent. Our study was designed with three moments of research. During the first moment, we defined the research design, conducted the literature review resulting in the state of the art, and identified two gaps in knowledge addressed in the Thesis. The first gap highlighted the need to identify the specific set of competencies needed for the designer to become an active agent of change. Furthermore, the second gap showed the opportunity to propose a new framework built to support Design Education to build the competencies identified. The second moment of research was guided by our hypoThesis and defined the data collection supported by a mixed methodology approach with a combination of qualitative and quantitative methods using surveys and elements of active research. Throughout this stage, we also collected all the data with the designed instruments. The data collected during the second research moment and with the information from the first research moment were analysed in the final research moment. The rich information gathered and examined culminated in developing the Design Competencies Framework set to be a valuable tool to support Design Education fostering change and train designers to deal with complex problems and become agents of change. At last, we sought validation of our findings and reflected on the results accomplished, allowing us to conclude and envision future developments.
Article
Introduction: In modern medical education, with its transformational changes, teacher-centered learning is transformed into student-centered learning. This shift has escalated so fast with momentum, and its value in teaching and learning has been evaluated in many ways. In the current scenario, problem-based learning (PBL) is a well-recognized effective method of teaching and learning. The credit goes to McMaster University in Hamilton, Ontario, Canada, for the establishment of PBL. In Malaysia, many medical schools have applied this method in their curriculum, one of which is University Malaysia, Sarawak (UNIMAS), which has also adopted PBL in its undergraduate curriculum since 1996. Purpose: The aim of this study is to determine the students’ acceptance of PBL and its positive and negative impact on their learning. Methodology: It was a cross-sectional study conducted to determine the acceptance of the students of PBL. The study population is selected using convenience sampling of 140 out of 148 pre-clinical year-2 students who were exposed to the PBL method, the focus group discussion (FGD) was conducted based on pre-framed questions to know the impact of PBL on the students’ learning. Results: Results showed that the respondents had the satisfaction and accepted PBL. Sixty students from 6 PBL groups were involved in FGD. Feedback from FGD revealed their difficulties with the conduction of 1st session of PBL, for instance, problems in searching resources, new learning environments and peers from different regions. There were positive responses spelled by the subjects that the PBL has improved their communication skills, critical thinking, and self-esteem. Conclusion: Overall the PBL has proved beneficial evidenced a positive impact on the learning process of medical students.
Article
The development of generic skills and competences has become a central component of contemporary engineering education due to increased societal and occupational complexity. Problem- and project-based learning (PBL) has been highlighted as one of the pedagogical approaches fostering generic skills and competences that are transferable between various contexts. However, the arguments linking PBL and generic competences have mostly been theoretically grounded or based on singular cases of teaching experience. The purpose of this literature review is therefore to present a comprehensive overview of the different types of generic competences documented in a PBL environment. The review includes 28 peer-reviewed articles that have documented engineering students’ perceptions of generic skill and competence development. The results reveal either an emphasis in the studies on teamwork, typically combined with a couple of other types of generic competences, or, in a few cases, a narrow focus on problem-solving. The synthesis of generic competences perceived by engineering students in PBL environments furthermore unfolds a landscape of generic competences, which provides a frame of reference to discuss strategies to foster the broad set of generic competences needed for future engineers to deal with the complex societal challenges of our time. Available as open access here: https://www.tandfonline.com/doi/full/10.1080/03043797.2022.2074819?src=
Article
Full-text available
Evidence for the superiority of guided instruction is explained in the context of our knowledge of human cognitive architecture, expert–novice differences, and cognitive load. Although unguided or minimally guided instructional approaches are very popular and intuitively appealing, the point is made that these approaches ignore both the structures that constitute human cognitive architecture and evidence from empirical studies over the past half-century that consistently indicate that minimally guided instruction is less effective and less efficient than instructional approaches that place a strong emphasis on guidance of the student learning process. The advantage of guidance begins to recede only when learners have sufficiently high prior knowledge to provide “internal” guidance. Recent developments in instructional research and instructional design models that support guidance during instruction are briefly described.
Article
Full-text available
This meta-analysis investigated the influence of assessment on the reported effects of problem-based learning (PBL) by applying Sugrue’s (1995) model of cognitive components of problem solving. Three levels of the knowledge structure that can be targeted by assessment of problem solving are used as the main independent variables: (a) understanding of concepts, (b) understanding of the principles that link concepts, and (c) linking of concepts and principles to conditions and procedures for application. PBL had the most positive effects when the focal constructs being assessed were at the level of understanding principles that link concepts. The results suggest that the implications of assessment must be considered in examining the effects of problem-based learning and probably in all comparative education research.
Article
Full-text available
This article is about the quantitative research practices and methodologies that are used in distance education (DE). It begins with an analysis and assessment of a segment of the DE research literature, DE/classroom comparison studies, based on a recently completed meta-analysis of that literature from 1985 to 2002. Overall, the 232 studies reviewed were judged to be of poor methodological quality and severely lacking in critical information about research practices. Studies of synchronous and asynchronous DE are discussed separately and recommendations are made for improving designs and measures within these patterns. Suggestions for future quantitative research areas are provided. In discussing these findings, we recognize that high-quality research is being conducted in the field and that qualitative forms of research contribute greatly to the mosaic of evidence that is the base of available knowledge about DE. Finally, three organizations, the What Works Clearinghouse (USA), EPPI-Centre (UK), and the Campbell Collaboration (international), all devoted to improving the quality of research and research synthesis in education, are described briefly; suggestions are made as to how their philosophies and approaches for judging the worthiness of research evidence can be used to improve DE research.
Article
This book is a joint package of a practical manual on how to undertake meta-analysis in medicine together with an accompanying CD-ROM. This provides individuals with access to meta-analysis software and the instructions and guidance on how to undertake them. The software package contains a computer program 'Metanalysis' which performs statistical analyses for the meta-analysis. It has some unique features currently not available in other meta-analysis software packages: ability to import graphics into Word, PowerPoint etc; Galbraith's plots; cumulative meta-analysis; number needed to treat; publication bias assessment. The graphics generated by the software are in a format compatible with Microsoft PowerPoint. Click here to view sample graphics. Order your copy online today!
Chapter
Summary We, as one of the sports consultants commissioned by the Singapore National Olympic Council, will be presenting our Training Program to plan to nurture Singaporeans in the field of Swimming (100m Freestyle). In this proposal, we will include the physics behind the sports, recommended diet, some forms of physical aids as well as an estimation of the cost of the entire Training Program.
Article
This meta-analytic study 1) examines outcomes from primary research comparing the impact of problem-based learning (PBL) and traditional curricula on medical students‘ National Board of Medical Examiners (NBME) I and II performance and 2) explores the use of Hierarchical Linear Modeling (HLM) in identifying study and PBL implementation characteristics that predict these outcomes. NBME I and II overall scores were used as dependent variables, with study design (randomized/non-randomized), publication year, and PBL experience (number of years since the adoption of PBL as an instructional method at the time of the reported NBME results) as explanatory variables. Initial unconditional HLM (i.e., without inclusion of potential explanatory variables) results indicated PBL curricula had: 1) a positive, albeit not statistically significant, effect on NBME II, with an average effect size of 0.16; and 2) a negative effect on NBME I, with an average effect size of −0.15. Including explanatory variables in HLM analyses explained additional variability in NBME I effect sizes, and identified study design (γ = 0.82, p = 0.01) and PBL experience (γ = 0.07, p = 0.02) as significant predictors of positive PBL effects. Publication year, in contrast, had significant negative effects (γ = −0.06, p = 0.02). This study explained variability in NBME I effect sizes and clarified the impact of PBL discerned in previous reviews. Implications for future research include the need to examine curriculum features that operationally define PBL, as well as extend the consideration of outcomes on which PBL‘s impact can be examined.