Content uploaded by William A. Sandoval
Author content
All content in this area was uploaded by William A. Sandoval on Aug 13, 2014
Content may be subject to copyright.
SANDOVAL AND BELLINTRODUCTION
Design-Based Research Methods for
Studying Learning in Context: Introduction
William A. Sandoval
Graduate School of Education and Information Studies
University of California, Los Angeles
Philip Bell
Cognitive Studies in Education
University of Washington
The field of psychology has a long history of interaction
with education, and educational psychology has had a pro-
found impact on how issues of learning have been framed
and studied in educational contexts. Still, it has never been
simple to translate theoretical insights into educational
practice. Educational psychology has been criticized for not
creating “usable knowledge” (Lagemann, 2002). Currently,
educational researchers generally have been pushed to jus-
tify how their claims are “scientific” and “evidence-based”
(National Research Council, 2002). There is a tension be-
tween the desire for locally usable knowledge on the one
hand and scientifically sound, generalizable knowledge on
the other. Lagemann, for example, argued that the tradi-
tional paradigm of psychology has striven for experimental
control at the expense of fidelity to learning as it actually
occurs. Thus, although such claims might be scientific in
one sense, they do not adequately explain or predict the
phenomena they purport to address. This critique extends
the long-standing debate surrounding the ecological valid-
ity of well-defined psychological tasks and their relation to
psychological phenomena as they come to occur in every-
day settings (Brunswik, 1943; Lewin, 1943). As a field, we
still lack an adequate methodological reconciliation that at-
tends to issues of both experimental control and ecological
validity. At the same time, there is considerable unease with
the perceived “credibility gap” (Levin & O’Donnell, 1999)
of much of educational research because it is not produced
with what are considered to be scientific methods. From
this perspective, the knowledge from educational research
has limited usability because it is not trustworthy.
An educational psychology that is both usable in a practi-
cal sense and scientifically trustworthy cannot proceed with-
out directly studying the phenomena it hopes to explain in its
inherent messiness. A little over a decade ago, Brown (1992)
described her evolving approach to “design experimenta-
tion” as an effort to bridge laboratory studies of learning with
studies of complex instructional interventions based on such
insights. She showed how insights from the laboratory were
inherently limited in their ability to explain or predict learn-
ing in the classroom. The challenge, as she saw it, was to de-
velop a methodology of experimenting with intervention de-
signs in situ to develop theories of learning (and teaching)
that accounted for the multiple interactions of people acting
in a complex social setting. At the same time, Collins (1992)
was putting forth a notion of educational research as a “de-
sign science,” like aerospace engineering, that required a
methodology to systematically test design variants for effec-
tiveness. Achieving such a design science, however, requires
a sufficient understanding of the underlying variables at all
relevant layers of a complex social system (schooling)—an
understanding that we do not yet have (Collins, Joseph, &
Bielaczyc, 2004).
The last 12 years have seen an increasing uptake of the de-
sign experimentation methodology, so much so that a recent
handbook on research in math and science education is re-
plete with examples and formulations of the approach (Kelly
& Lesh, 2000). The general approach has been called by
many names. We have settled on the term design-based re-
search over the other commonly used phrases “design exper-
imentation,” which connotes a specific form of controlled ex-
perimentation that does not capture the breadth of the
approach, or “design research,” which is too easily confused
with research design and other efforts in design fields that
lack in situ research components. The approach to research
described in this issue is design based in that it is theoreti-
EDUCATIONAL PSYCHOLOGIST, 39(4), 199–201
Copyright © 2004, Lawrence Erlbaum Associates, Inc.
Requests for reprints should be sent to William A. Sandoval, Graduate
School of Education and Information Studies, University of California, Box
951521, 2339 Moore Hall, Los Angeles, CA 90095–1521. E-mail:
sandoval@gseis.ucla.edu
cally framed, empirical research of learning and teaching
based on particular designs for instruction. Design-based re-
search simultaneously pursues the goals of developing effec-
tive learning environments and using such environments as
natural laboratories to study learning and teaching. On the re-
search side of the endeavor, design-based researchers draw
from multiple disciplines, including developmental psychol-
ogy, cognitive science, learning sciences, anthropology, and
sociology. On the design side of the work, researchers draw
from the fields of computer science, curriculum theory, in-
structional design, and teacher education.
Since Brown’s and Collins’influential works, the paradigm
has evolved primarily as a means for studying innovative
learning environments, often including new educational tech-
nologies or other complex approaches, in classroom settings.
As this form of interventionist research has spread, questions
have emerged for which there are not yet clear answers. What
exactly counts as design-based research? What kinds of
knowledge can design-based research produce? What stan-
dards do, or should, exist to judge the quality of design-based
research? Such questions are being asked by both practitioners
of the approach and observers of educational research (see
Kelly, 2003). The articles in this special issue of Educational
Psychologist extend recent discussions of these questions,
both to situate design-based research within a broader context
of research on learning and to continue a needed conversation
on the nature of design-based research specifically and useful
forms of educational research generally.
The articles in this set arise from work supported by an Ad-
vanced Studies Institute grant from the Spencer Foundation to
a group of early career researchers trained in design-based re-
search methods. Each article in the set addresses various as-
pects of one central question: How does the effort to design
complex interventions influence research? The effort to de-
sign complex interventions raises a set of methodological and
theoretical issues. One of the most commonly faced method-
ological issues in design-based research is the tension between
making an intervention “work” in a complex setting, which of-
ten necessitates changing the intervention as it unfolds (in a
way that directly mirrors the dynamic, contingent nature of de-
cision making during teaching), with the researchers’ need for
empirical control, which argues against changing the planned
“treatment.” The general issue this raises is one of causal attri-
bution: What makes a particular intervention successful in a
particular place? How can what is learned from a particular
success be generalized? It has been argued that design-based
research can develop different kinds of knowledge, including
better theoretical understanding of the learning phenomena
addressed by an intervention and knowledge of useful and
generalizable design practices (Design-Based Research Col-
lective, 2003; Edelson, 2002). The set of articles in this issue
describe how the design-based research methodology can pro-
duce such knowledge by raising issues that arise from the in-
fluence of design on research and how design-based research-
ers try to address them.
Hoadley begins by framing perhaps the central issue of
any research—the basis on which claims can be warranted.
He discusses how the interplay between designing and then
studying interventions in naturalistic settings can lead to
“methodological alignment.” As initially unpredicted obser-
vations arise among predicted ones, a design-based research
team’s methodological approach changes with developing
theoretical knowledge, leading to intervention designs that
are better fit to their intended setting and to better explana-
tions of how they work. Hoadley frames this notion of meth-
odological alignment in comparison to typical experimental
and quasi-experimental designs of educational psychology to
separate the methodological threats to validity and reliability
faced by any research approach from the issues that specifi-
cally face design-based researchers.
Following this are three articles that each take a different
slice on how design-based research can contribute to theoret-
ical understanding of learning in complex settings. Each of
the articles by Sandoval, Tabak, and Joseph reveal how the
design of complex interventions is an explicitly the-
ory-driven activity. Sandoval introduces the notion of “em-
bodied conjectures” to characterize how instructional de-
signs materially embody theoretical conjectures about how
people learn. They therefore carry expectations about how
designs should function in a setting, and tracing how such ex-
pectations are met or unmet can refine the underlying theo-
retical conjecture. Through a retrospective analysis, he ar-
gues that an important way of increasing the rigor of
design-based research is for researchers to explicitly map the
embodiment of particular conjectures through their design
reification and to then design research studies to specifically
tests the predictions that result. Such predictions pertain to
both outcomes expected from the intervention and ways in
which designed scaffolds are expected to function. The need
to link outcomes to these expected functions across research
iterations is the source of power from this analytic approach.
Tabak considers the theoretical and methodological ten-
sions that arise when complex interventions are introduced
into classroom settings. She describes how intangible aspects
of interventions, such as designs for particular forms of class-
room discussions, blur the boundaries between the interven-
tion and what is typically thought of as “the context.” Rather
than consider such a blurring as only a disadvantage, Tabak ar-
gues that attention to such emergent activity structures is a key
element of design-based research and can contribute to the de-
velopment of theories of contextualization. Her argument is an
explicit bridge between psychological perspectives that
would, under the guise of experimental control, generally ig-
nore aspects of context outside of the perceived intervention
treatment and anthropological views of context that do not at-
tempt to distinguish between“native”anddesigned features.
Joseph describes how the complexity involved in devel-
oping a novel instructional approach and trying to under-
stand its enactment and potential benefits forced choices
about which aspects of the intervention became the focus of
200 SANDOVAL AND BELL
research and which aspects were just “engineered” to work
in the immediate setting. This is a common occurrence dur-
ing design-based research efforts, and such choices arise
somewhat unpredictably from the setting of an intervention.
Joseph describes how such choices can be theoretically
guided and thus contribute to theoretical refinement. In her
case, the effort to design an interest-based curriculum for
elementary students exposed the limitations to current theo-
retical ideas of motivation and interest and led to the devel-
opment of a framework for conceptualizing interest in a
“usable” way.
Bell ends the article set by laying out what he refers to
as the “grammar and epistemology” of design-based re-
search. In surveying design-based research efforts from the
last decade, Bell identifies several variations of the method-
ology that differ in their theoretical perspective and argues
that these differences in underlying theoretical position lead
to epistemological differences in the kinds of claims that
design-based researchers might try to make and the bases
on which they warrant such claims. He argues for the im-
portance of recognizing design-based research as encom-
passing this variety of theoretical and epistemological per-
spectives, to both better understand just what design-based
research is (or might be) and clarify how design efforts can
be better tied to theoretical and methodological consider-
ations. In the end, he argues against a singular definition of
what might constitute “scientific” design-based research
and for a theoretical, and hence methodological, pluralism
to efforts that seek to understand learning and influence ed-
ucational practice.
In her commentary, O’Donnell considers the methodolog-
ical issues and approaches sketched by the article set in terms
of ideas of quasi-experimental and experimental research de-
sign. She critically examines the assumptions about de-
sign-based research behind each article and pushes on meth-
odological issues that remain outstanding and hence a threat
to the promise of design-based research.
As a collection, these articles depict the unique, evolving,
and expanding role of design-based approaches to educa-
tional inquiry. They also highlight the pressing issues and fu-
ture directions currently facing researchers making use of
this methodological orientation. We believe that by bringing
some much-needed clarity around the nature of design-based
research to the broader readership of Educational Psycholo-
gist, it will fuel productive discussions of research methodol-
ogy and modes of inquiry currently appropriate for the study
of learning.
REFERENCES
Brown, A. L. (1992). Design experiments: Theoretical and methodological
challenges in creating complex interventions in classroom settings. Jour-
nal of the Learning Sciences, 2(2), 141–178.
Brunswik, E. (1943). Organismic achievement and environmental probabil-
ity. Psychological Review, 50, 255–272.
Collins, A. (1992). Toward a design science of education. In E. Scanlon & T.
O’Shea (Eds.), New directions in educational technology (pp. 15–22).
New York: Springer-Verlag.
Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoreti-
cal and methodological issues. Journal of the Learning Sciences, 13(1),
15–42.
Design-Based Research Collective. (2003). Design-based research: An emerg-
ing paradigm for educational inquiry. Educational Researcher, 32(1), 5–8.
Edelson, D. C. (2002). Design research: What we learn when we engage in
design. Journal of the Learning Sciences, 11(1), 105–121.
Kelly, A. E. (Ed.). (2003). Special issue on the role of design in educational
research [Special issue]. Educational Researcher, 32(1).
Kelly, A. E., & Lesh, R. A. (Eds.). (2000). Handbook of research design in
mathematics and science education. Mahwah, NJ: Lawrence Erlbaum
Associates, Inc.
Lagemann, E. C. (2002). An elusive science: The troubling history of educa-
tion research. Chicago: University of Chicago Press.
Levin, J. R., & O’Donnell, A. M. (1999). What to do about educational re-
search’s credibility gaps? Issues in Education, 5, 177–229.
Lewin, K. (1943). Defining the “field at a given time.” Psychological Re-
view, 50, 292–310.
National Research Council. (2002). Scientific research in education.Wash
-
ington, DC: National Academy Press.
INTRODUCTION 201
