Design-Based Research Methods in CSCL: Calibrating our Epistemologies and Ontologies

Abstract

Design-based research (DBR) methods are an important cornerstone in the methodological repertoire of the learning sciences, and they play a particularly important role in CSCL research and development. In this chapter, we first lay out some basic definitions of what DBR is and is not, and discuss some history of how this concept came to be part of the CSCL research landscape. We then attempt to describe the state-of-the-art by unpacking the contributions of DBR to both epistemology and ontology of CSCL. We describe a tension between two modes of inquiry-scientific and design-which we view as inherent to DBR, and explain why this has provoked ongoing critique of DBR as a methodology, and debates regarding the type of knowledge DBR should produce. Finally, we present a renewed approach for conducting a more methodologically-coherent DBR, which calibrates between these two modes of inquiry in CSCL research. Definition & Scope DBR is one of a cluster of terms used to describe various intersections between design and research, especially in the realm of academic research in either education or in human-computer interaction. In this section, we attempt to define what we mean by design-based research and contrast it with other definitions. DBR methods were originally defined (Design-Based Research Collective [DBRC], 2003; Hoadley, 2002), like the earlier concept of design experiments (Brown, 1992; Collins, 1990,1992), as a research method or related methodology which used a blended form of design activities and research activities to produce design-relevant, empirically supported knowledge. Designed interventions in DBR are tested iteratively in a context of use, and the iterations become settings to collect data that support or refute inferences about underlying theoretical claims. At the same time, the iterations are used for increasing the fit between the theory, the design, and the enactment or implementation so as to best test the theoretical conjectures. Unlike earlier definitions associated with design experiments (notably Brown's, 1992), DBR methods were claimed to be not merely related to hypothesis generation, but a scientific enterprise in their own right. This approach stemmed from a very practical problem described earlier by Simon (1969) in his seminal book-The Sciences of the Artificial-namely, that

Full text

This is a pre-print of:
Kali, Y. & Hoadley, C. (in press). Design-based research methods in CSCL: Calibrating our epistemologies and ontologies. In U. Cress, C. Rosé,
A. Wise, and J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning. Springer.
1
Design-Based Research Methods in CSCL:
Calibrating our Epistemologies and Ontologies
Yael Kali1, Christopher Hoadley2
1 University of Haifa, Faculty of Education, Haifa, Israel, yael.kali@edtech.haifa.ac.il
2 New York University, Educational Communication and Technology Program, New York, USA,
tophe@nyu.edu
Abstract: Design-based research (DBR) methods are an important cornerstone in the
methodological repertoire of the learning sciences, and they play a particularly important role in
CSCL research and development. In this chapter, we first lay out some basic definitions of what
DBR is and is not, and discuss some history of how this concept came to be part of the CSCL
research landscape. We then attempt to describe the state-of-the-art by unpacking the
contributions of DBR to both epistemology and ontology of CSCL. We describe a tension between
two modes of inquiry—scientific and design—which we view as inherent to DBR, and explain why
this has provoked ongoing critique of DBR as a methodology, and debates regarding the type of
knowledge DBR should produce. Finally, we present a renewed approach for conducting a more
methodologically-coherent DBR, which calibrates between these two modes of inquiry in CSCL
research.
Keywords: Design-based research (DBR), CSCL epistemology, CSCL ontology, methodological
alignment, Design researchers’ transformative learning (DRTL)
Definition & Scope
DBR is one of a cluster of terms used to describe various intersections between design and
research, especially in the realm of academic research in either education or in human-computer
interaction. In this section, we attempt to define what we mean by design-based research and
contrast it with other definitions.
DBR methods were originally defined (Design-Based Research Collective [DBRC], 2003;
Hoadley, 2002), like the earlier concept of design experiments (Brown, 1992; Collins, 1990,1992),
as a research method or related methodology which used a blended form of design activities and
research activities to produce design-relevant, empirically supported knowledge. Designed
interventions in DBR are tested iteratively in a context of use, and the iterations become settings
to collect data that support or refute inferences about underlying theoretical claims. At the same
time, the iterations are used for increasing the fit between the theory, the design, and the enactment
or implementation so as to best test the theoretical conjectures. Unlike earlier definitions associated
with design experiments (notably Brown’s, 1992), DBR methods were claimed to be not merely
related to hypothesis generation, but a scientific enterprise in their own right. This approach
stemmed from a very practical problem described earlier by Simon (1969) in his seminal book—
The Sciences of the Artificial—namely, that

This is a pre-print of:
Kali, Y. & Hoadley, C. (in press). Design-based research methods in CSCL: Calibrating our epistemologies and ontologies. In U. Cress, C. Rosé,
A. Wise, and J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning. Springer.
2
… the genuine problem is to show how empirical propositions can be made at all
about systems that, given different circumstances, might be quite other than they
are (p. XI).
In the case of DBR as a science of the artificial, this genuine problem concerns making empirical
propositions regarding designs of learning environments that are studied while they are being
created.
The notion of DBR as a research methodology contrasts with other points of connection between
design and research. Specifically, Instructional design, User-centered design and other similar
terms from the fields that attempt to create educational interventions—materials, or technologies
might be lumped under the terminology of research-based design (RBD) methods. In such methods
the tools of empirical research are subservient to the goal of ultimately creating a useful designed
product or intervention. The main difference, thus, is that DBR uses design processes to produce
research knowledge, where RBD uses research techniques to produce designs. Evaluation research
of designs is similar to DBR in that at the end there is both a design and research output, but the
difference is that these activities in evaluation research are by necessity separated from each other.
The intervention or tool is complete at the moment in which evaluation is taking place, and the
data used to inform the design is typically distinct from the data used to evaluate that design.
The terms design research or design studies are used variously in different communities, ranging
from the journal Design Studies, which focuses on studies of designers and design processes, to a
notion of design research which labels the learning process a designer must go through in order to
connect a context to a designed solution (e.g., Laurel, 2013). Another more recent term is Design-
Based Implementation Research (Fishman et al., 2013; Kali et al., 2018), which can be
characterized as a subset of DBR with three main distinctive characteristics: joint ownership of the
research agenda by practitioners and designers/researchers; an inherent focus on designs and
research questions related to the issues of scaling interventions systemically (e.g., across a large
school system or a geographic region); and a linkage between micro-level design (design of a
particular intervention, for instance at a classroom level) and macro-level systems change (e.g.,
design of an institution wide framework for adoption) (Law et al., 2016).
We look more specifically at the issue of DBR methods in the particular sense of a research
methodology which yokes the design process and research process to produce knowledge
outcomes (and not just useful, validated designs). Although many have suggested more
generalized definitions of DBR since its introduction (e.g., McKenney & Reeves, 2012, 2018), we
rely on the earlier characterization from the Design-Based Research Collective (DBRC, 2003) as
it encapsulates more directly what critics find challenging about DBR. In this definition, DBR has
five characteristics: (a) overlap between the design and research process (both temporally and
intellectually); (b) iterative cycles of design, enactment in context, analysis, and redesign; (c) a
goal of theory development that is relevant to practice; (d) a commitment to understanding the
designs in authentic settings (as opposed to more reductionist approaches); and (e) a recognition
that the design and the enactment are intertwined in producing the outcomes (that is, that outcomes
are the result of both the use of designed artifacts and the way they are used).

This is a pre-print of:
Kali, Y. & Hoadley, C. (in press). Design-based research methods in CSCL: Calibrating our epistemologies and ontologies. In U. Cress, C. Rosé,
A. Wise, and J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning. Springer.
3
History & Development: DBR in CSCL
We believe the connection between DBR methods and the CSCL research community is not a
coincidence, but rather a natural byproduct of the ways in which almost all CSCL research is
contingent on shifting, culturally and technologically grounded social contexts for learning, and
on theories that help encompass that social context. Various authors (e.g., Kaptelinin & Cole, 2002;
Koschmann, 1999; Paavola et al., 2004; Stahl et al., 2006) have explored how socially
contextualized theories intersect with a technology-enhanced action orientation of research. Such
a design orientation for research is notable, for example, in Kaptelinin and Cole’s (2002) classic
use of activity theory for analyzing the design of a collaborative learning environment. The design
is conceptualized as a perturbation of activity structures, placing the scope less on a particular tool
and more on how the tool, together with the designed collaboration processes support learning. As
Stahl et al. (2006) point out, the intersubjective nature of learning, and the challenges of
intersubjectivity among researchers and analysts of human behavior influences the relationship
between design and research in CSCL:
CSCL research has both analytic and design components. … To design for
improved meaning making, however, requires some means of rigorously studying
praxis. In this way, the relationship between analysis and design is a symbiotic
one—design must be informed by analysis, but analysis also depends on design
in its orientation to the analytic object (Stahl et al., 2006, p. 11).
Challenges such as these have led to discussions and debates about DBR within the context of
CSCL research and development.
In the early 2000s, a blossoming of scholarship on DBR methods yielded a number of special
issues, including those published in Educational Researcher (2003), Journal of the Learning
Sciences (2004), Educational Psychologist (2004), Educational Technology (2005). The articles
included in these special issues helped legitimize the approach, but also proliferated alternative
definitions of what constitutes DBR and how it would fit with other related concepts such as
“design research”, and engaged with critiques of the method and its underlying epistemologies.
Prominent critiques included a failure to contend with lack of appropriate experimental control for
causal inferences (Desforges, 2000), difficulty conveying in adequate detail the relevant aspects
of the design and the data (Reeves, 2005), being susceptible to overinterpreting and/or cherry-
picking interpretations given the breadth of data collected under evolving, rather than fixed,
protocols (Dede, 2004, JLS), and a lack of a clear argumentative grammar (Kelly, 2004).
State of the Art: Argumentative grammars and tensions within DBR
epistemology and ontology
As described above, one way to understand DBR is its dual goal in advancing both learning
theory—explanatory evidence-based arguments on how people learn in various instructional
contexts (especially in those involving CSCL), and learning design (the features and principles for
environments that support such learning). When it comes to theory, we might start with a
positivistic psychological or cognitive framing of what a theory is, but we can also extend the
notion of learning theories much more broadly with interpretivistic socio-cultural conceptions,

This is a pre-print of:
Kali, Y. & Hoadley, C. (in press). Design-based research methods in CSCL: Calibrating our epistemologies and ontologies. In U. Cress, C. Rosé,
A. Wise, and J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning. Springer.
4
situative understandings, humanistic theories, etc. On the other hand, design knowledge might
encompass specific designed artefacts or interventions, ideas about how to instantiate particular
goals through human agency, or ideas about what interventions might be possible. Unlike in
traditional experimental research, design in DBR is not solely a means for the purpose of
conducting research—it is a goal by and of itself, juxtaposed to its twin goal of advancing theory.
Yet, there are important differences in what makes good or useful outcomes in these two arenas—
theory and design. These differences create an inherent tension within DBR, which affects how
we judge the worth of the processes of knowing—DBR’s epistemology, as well as the nature and
types of knowledge produced—what we might term DBR’s knowledge ontology. Following Chi’s
notion of ontological commitments (Chi, 1992; Slotta, 2011), it is worth saying that the types of
knowledge produced in DBR fall in different sorts of categories which are determined in part by
the ontological commitments we hold as designers and researchers. An ontology of DBR in CSCL
should include different categories of knowledge, ranging from design patterns, to presumed
universal laws of psychology. In other words, the tension between theory and design in DBR in
CSCL affects how we know things, and what kinds of knowledge are produced. In this section, we
describe debates within the learning sciences and CSCL communities concerning the value of
DBR, how it can best be conducted and communicated, and what its outcomes should look like.
We then illustrate how these debates are in fact a result of the theory-design inherent tension within
DBR.
DBRs dual epistemic game
People follow rules in deciding what claims are valid in different research contexts. One term for
this is epistemic games. In introducing this term, Perkins (1997) referred to patterns of inquiry,
such as goals, moves and rules, which he described as:
… woven together in a course of inquiry... [and are often] played competitively,
as in the adversarial system of justice of scientific debates (p. 52).
Another term for describing the ways in which researchers progress toward knowledge and
understanding in a field is argumentative grammar. In the world of methodologies, the
argumentative grammar determines the rules for making an argument within the coherent world
of an epistemology or method. Thus, epistemic games can be thought of as the language of claims
and debates in a field, and argumentative grammar as the underlying structure of that language.
Within DBR, a criticism has been that it is not clear on its argumentative grammar (Kelly, 2004):
What, therefore, is the logos of design studies in education? What is the grammar
that cuts across the series of studies as they occur in different fields? Where is the
“separable” structure that justifies collecting certain data and not other data and
under what conditions? What guides the reasoning with these data to make a
plausible argument? Until we can be clear about their argumentative grammar,
design study methods lack a basis for warrant for their claims. (p. 119)
Such criticism objected the pluralism that DBR researchers such as Bell (2004), and later on
McKenney and Reeves, (2012, 2018) or Bakker (2018) ascribed to DBR. Bell, for instance, already
in 2004 maintained that:

This is a pre-print of:
Kali, Y. & Hoadley, C. (in press). Design-based research methods in CSCL: Calibrating our epistemologies and ontologies. In U. Cress, C. Rosé,
A. Wise, and J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning. Springer.
5
At a time when many efforts that are reviewing the status of educational research
seem to be operating under the working assumption that our theoretical and
methodological complexity should be reduced, I argue that rigor and utility can
be actively pursued through pluralism—a coordination of different theoretical
views on learning and education. (Bell, 2004, p. 251)
We claim that this ambiguity within DBR methodologies (even if we refer to methodologies in
plural, and not a single methodology) results not only from the broad range of theoretical views
studied using DBR, but rather—is rooted in the epistemological tension inherently embedded in
the dual goal of DBR. Consequently, the lack of a clear argumentative grammar in DBR is mainly
related to lack of clear linkage between the two languages we speak (advancing theory and
advancing design). That is, we (design researchers) typically play two epistemic games, and
oftentimes—are not clear enough about how we switch between them.
To illustrate what we mean by a dual epistemic game, we turn to philosophical notions of design.
In their seminal book “The design way”, Nelson and Stolterman (2012) characterize the unique
mode of inquiry that designers follow, by contrasting it with the one followed by scientists. While
scientists, in general strive to reason from the concreteness and complexity of the actual world, to
the abstractness and simplicity of principles and laws (yellow arrow, going up the curve in Fig.
1
1
), designers, they say, strive to do the opposite. That is, designers use such abstractions to create
specific designs in the actual world (e.g., a specific product or policy) by making design
judgements (blue arrow, going down the curve in Fig. 1). Therefore, science and design constitute
quite different traditions of inquiry that encompass contrasting rules within their epistemic games.
We claim though, that in DBR we play and intertwine both these traditions, iterating between
abstraction and particularization (green arrow in Fig. 1). A DBR study typically begins by
identifying a gap in educational theory that we (DBR researchers) aim to explore by designing and
enacting an intervention within the so-called “real world”, what Bhaskar (1975) would call the
actual world. To develop an initial design, we take into account generalized abstractions (e.g.,
theories, design principles), and embody them into a specific design (going down the curve). Then,
we collect (messy) data regarding how learners interact with our designs in the actual world, and
analyze this data (using the existing theoretical lenses, but open to refining them) to come up with
new generalized conjectures about learning (going up the curve) and use them to refine the designs
(down the curve), to test these conjectures (up again), and so on with as many iterations as needed
to contribute to both theory and practice.
1
Many thanks to the anonymous reviewer who brought our attention to Bhaskar’s conceptions of philosophy of science making a di stinction
between (a) the ‘real’ world i.e. laws of nature independent of human interpretation, (b) the ‘actual’ world i.e. things that have come to exist
through the action of those laws of nature, and (c) the ‘empirical’ world, i.e. what we, as humans come to observe, measure, describe, or
experience of the actual world. Neilson and Stolterman use the term ‘real’ for the x -axis but we have relabeled it to be the ‘actual’ to align with
Bhaksar’s terminology. We beli eve this is closer to what Neilson and Stolterman meant.

This is a pre-print of:
Kali, Y. & Hoadley, C. (in press). Design-based research methods in CSCL: Calibrating our epistemologies and ontologies. In U. Cress, C. Rosé,
A. Wise, and J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning. Springer.
6
Fig. 1 Contrasting “science” and “design” modes of inquiry (adapted from Nelson and Stolterman, 2012) and the
dual intertwining epistemic game we play in DBR, iterating between abstraction and particularization
It turns out that within this abstraction-particularization tango, we constantly switch epistemic
languages, and therefore it is clear why DBR is missing one agreed upon argumentative grammar.
In doing so, DBR is sometimes used within a positivistic framing to make strong, generalizable
truth claims about a presumably objectively knowable world. But DBR is also sometimes used
within an interpretivist framing to explore aspects of the human experience that are presumed to
be knowable only through individual interpretation and which are inherently not generalizable.
DBR researchers may violate some of the core tenets of either of these core epistemologies, much
to the consternation of researchers hoping to fit it in with their existing epistemological
commitments. Such distress is expressed in the following excerpt from an anonymous reviewer in
his/her comments regarding a manuscript describing a DBR project:
[the manuscript entails] an awkward combination of qualitative and quantitative
research perspectives. Symptomatic of this is the fact that you use both the word
“causal” and the word “holistic” in your title! Show us where you stand.
(Anonymous reviewer)
Thus, DBR sits in tension both with positivism and interpretivism, both “quantitative” and
“qualitative” research, and better adheres to mixed methods (Bell, 2004). Knowledge claims rely
heavily on the designer’s stance and interpretation of not only the data, but also their interpretation
of the design context, circumstances and goals (Tabak, 2004). Therefore, such claims are presumed
to be somewhat generalizable, but—using diSessa’s (1991) terminology—based on local (rather
than global) sciences. Cobb and Gravemeijer (2008) refer to such generalizations as domain
specific instructional theories.

This is a pre-print of:
Kali, Y. & Hoadley, C. (in press). Design-based research methods in CSCL: Calibrating our epistemologies and ontologies. In U. Cress, C. Rosé,
A. Wise, and J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning. Springer.
7
Why we have multiple argumentative grammars, and what is still missing
Recently, Bakker (2018) suggested to address Kelly’s criticism by noting that we do not
necessarily need one argumentative grammar, but rather, multiple grammars. This view is in line
with the pluralistic view of DBR methodology described earlier (e.g., Bell, 2004; McKenney &
Reeves, 2012, 2018). In the chapter “Argumentative grammars used in design research”, Bakker
lays out various solutions that have been developed in the past two decades to serve as underlying
“rules” for making DBR arguments. He presents these “rules” using Toulmins (1958) general
argumentation scheme which clearly distinguishes claims, evidence and reasoning to illustrate the
external structural logic of these grammars. Within these grammars he includes: (a) Proof of
principle that certain learning outcomes are possible (e.g., O’Neill, 2012), which requires advance
setting of criteria for success and failure; (b) Small changes per iteration, which enable
experimental approaches for comparing learning outcomes between iterations (e.g., Kali et al.,
2009); (c) Building on the experience of the DBR community, as in the design principles database
(Kali, 2006, 2008) in which DBR researchers can use, refine, and share their own design principles,
making it possible to abstract generalized explanations based on refinement of insights across
studies; (d) Answering the “how” question, which illustrates the logic of experimental designs that
aim to develop insights regarding how a particular educational approach can support learners
achieve certain educational goals (e.g., Smit et al., 2013); and (e) Conjecture mapping (Sandoval,
2014), which distinguishes between high-level conjectures that are derived from theory and
embodied into the design of learning environments, design conjectures that define the relation
between features in the environments (e.g., tools, activity structures) and the resulting mediating
processes, and theoretical conjectures that focus on the learning outcomes that result from these
processes (Fig 2).
Fig. 2 A generalized conjecture map (adapted from Sandoval, 2014)
We focus specifically on Sandoval’s (2014) conjecture mapping due to its wide acceptance and
use among DBR researchers, but also, because we contend that it nicely illustrates the dual
epistemological game, and the intertwining between the abstracting-generalized-explanations and
the particularization modes of inquiry (Fig. 1).
First, the embodiment of a high-level theoretical conjecture into design features within a learning
environment clearly demonstrates a “down the curve” process of particularization. Then,
characterizing the learning that occurs during enactment in terms of mediating processes represents

This is a pre-print of:
Kali, Y. & Hoadley, C. (in press). Design-based research methods in CSCL: Calibrating our epistemologies and ontologies. In U. Cress, C. Rosé,
A. Wise, and J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning. Springer.
8
beginning stages (typically with interpretive methods) of an “up the curve” process in seek for
abstracted generalized explanations (e.g., patterns of use), which are then further substantiated in
terms of theoretical conjectures (how the mediating processes support learning outcomes). But as
noted by Sandoval (2014), such mapping represents only part of a trajectory of studies (multiple
iterations), that together enable the development of generalized explanations in DBR. That is,
conjecture maps are revised from iteration to iteration, and additional back-and-forth movements
within the abstraction-particularization curve are typically conducted.
Therefore, we believe that although multiple argumentative grammars, as suggested by Bakker
(2018) enable DBR researchers the flexibility in making decisions about what counts as DBR, it
does not solve the dual language issue inherent to DBR, which requires better calibration between
the two epistemic games involved. Moreover, as we explain in the next section, we view the chasm
between the two epistemic games as percolating from DBR epistemology into DBR knowledge
ontology. Due to this chasm, researchers debate not only rigorousness of DBR methods, but also
the value of DBR outcomes.
How the dual epistemic game percolates into design ontology
The debate regarding the value of DBR outcomes was most notably expressed in a series of thee
“reports and reflection” articles in the Journal of the Learning Sciences. Bereiter (2014) argued
that DBR researchers fail to produce outcomes that embed “know why” knowledge within “know
how” artifacts. He labelled such blended knowledge—having the potential to be useful for both
researchers and practitioners in generating innovation—principled practical knowledge (PPK).
Janssen et al. (2015), however, in their response article—Practicality studies: how to move from
what works in principle to what works in practice—maintained that PPK, as specified by Bereiter,
is too abstract to support teachers in implementing innovations developed in DBR research. They
contended that the DBR community underestimates the magnitude of usability issues, and
suggested an additional type of knowledge—fast and frugal heuristics—to complement PPK. This
debate continued with Bereiter’s (2015) response cautioning DBR researchers from being too
specific regarding how to implement the outcomes of their studies. Such specificity, he claims,
may communicate a message of disrespect to teacher professionalism, and hinder teachers from
venturing successfully beyond conventional practices.
This ongoing debate relates back to the dual epistemic game exemplified in Fig. 1. Is DBR trying
to make truth claims within a coherent (interpretive, positivist, or other) epistemology? Sometimes,
DBR produces knowledge that is contingent on context, but more actionable. In other words,
sometimes DBR is more concerned with producing usable knowledge, than with producing truth
claims. This tension in DBR has been referred to in various terminologies such as actionable
knowledge vs. knowledgeable action (Markauskaite & Goodyear, 2017); generalization vs.
generativity (Bakker, 2018); and analytical vs. creative mindsets (McKenney & Reeves, 2012,
2018). Interestingly, all of the researchers who pointed to this tension, note a detrimental bias in
which the research community typically prefers the “scientific” over the “design” mode of inquiry,
as indicated in standards of publication and the like. That is, actionable knowledge, tends to be
valued more than knowledgeable action, generalization more than generativity, and analyticality
more than creativity, in conducting DBR studies.

This is a pre-print of:
Kali, Y. & Hoadley, C. (in press). Design-based research methods in CSCL: Calibrating our epistemologies and ontologies. In U. Cress, C. Rosé,
A. Wise, and J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning. Springer.
9
The Future: Capitalizing on the dual epistemic game in DBR to spur creativity
and innovation in rigorous DBR research
Up to this point we have characterized DBR as being pluralistic, accommodating of a wide range
of methodologies, and have shown how this pluralism has drawn criticism, and interpreted as a
lack in argumentative grammar (e.g., Kelly, 2004). We also illustrated how DBR researchers have
addressed such criticism with various argumentative grammars, as well as with the notion that
having multiple grammars is pertinent (Bakker, 2018).
However, we believe that DBR researchers need to acknowledge the duality in the epistemic game
we play, and that this duality is not a fair target for the criticism of lack of an argumentative
grammar. Rather, we suggest that DBR be examined on the basis of the coherence of arguments
across the dominant argumentative grammars as researchers intertwine the abstraction-
particularization curve (Fig. 1). The next step for DBR is not only to acknowledge, but also to
capitalize on this epistemological and ontological duality while considering the systemic validity
of the activity. That is, it is less important that the epistemic games are narrowly played, and more
important that the outcomes of the research matter and make sense both in the knowledge realm
and to the people involved, leading to actions and decisions that support a consequential validity
of the research. To do so, in this section we draw on two frameworks: (a) methodological alignment
(Hoadley, 2004) and (b) design researchers’ transformative learning (DRTL, Kali, 2016).
Methodological alignment as means for calibrating the theoretical and practical aspects of DBR
The notion of methodological alignment is essential to our understanding of rigor and research
validity. It involves the ways in which researchers connect theories to hypotheses, hypotheses to
interventions, interventions to data gathering, and data gathering to interpretation and application.
Fifteen years ago, Hoadley (2004) argued that we tend to overemphasize certain types of validity
on the expense of others. Specifically, he argued that measurement validity is often regarded as
the sole, or at least main indicator of rigor. That is, the efforts of ensuring that the means of data-
collection accurately align with what is being measured predominates our view of well-designed
research. DBR, he claimed—with its unique research design—affords three other types of validity:
(a) treatment validity—ensuring that the treatments we create accurately align with the theories
we are examining, (b) systemic validity—that the inferences we make to prove our claims are
aligned with these theories, and (c) consequential validity—that these theories are applicable to
decisions based on the research.
We view these three types of validity measures for reaching methodological alignment as
principles for calibrating methodological moves in DBR, aiming at both theoretical and practical
advancements. That is, the multiple iterations in DBR—each involving back-and-forth movements
within the abstraction-particularization curve, between scientific and design modes of inquiry (Fig.
1)—afford DBR researchers with multiple opportunities to reach higher degrees of treatment,
systemic and consequential validity. In this way, methodological alignment principles can serve
DBR in achieving a unique type of rigor, which traditional research methods in education may fail
to afford.
At the same time these calibration principles can address the ontological debate, and assist in
producing PPK. Traditional education research believes that the knowledge (or what Nelson and

This is a pre-print of:
Kali, Y. & Hoadley, C. (in press). Design-based research methods in CSCL: Calibrating our epistemologies and ontologies. In U. Cress, C. Rosé,
A. Wise, and J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning. Springer.
10
Stolterman (2012) refer to as “the true”) lives in the abstracted generalized explanations that are
typically expressed in journal articles. Traditional design believes the knowledge lives in the
designed artifacts—curricula, technology-enhanced learning environments, etc. (what designers
add to “the actual”—according to Nelson and Stolterman (2012)). In DBR, because we have this
different ontological status of knowledge, it lives in neither and both. If we follow the CSCL way
of seeing knowledge as contextualized, distributed, culturally embedded, and constantly
negotiated by real human beings using information communication technologies, we need to
understand that PPK doesn’t live in a research article or a designed learning environment alone. It
lives in humans who must negotiate the ontological tensions we have outlined, and this demands
personal transformation.
Transforming ourselves as a prerequisite for transforming others
In the “Design Researchers’ Transformative Learning” (DRTL) framework, Kali (2016) claimed
that DBR provides an especially fertile ground for transformative learning among those who
conduct it. DRTL builds on Mezirow’s (1996) transformative learning theory, in which such
learning is characterized as “the process of using a prior interpretation to construe a new or revised
interpretation of the meaning of one’s experience in order to guide future action” (p. 162). That is,
transformative learning results not so much in a learners’ recognition of new facts about matters
under study. Rather, these are personal “aha moments” that bring learners to reorganize the ways
of looking at, thinking about, and acting on those matters. Kali (2016) claimed that in DBR, such
personal “aha moments” often expose researchers to flaws in their earlier conceptualization (which
is one of the reasons DBR researchers tend to keep these parts of their research behind the scenes).
But more importantly, the transformative learning enables design researchers not only to develop
new conceptualizations for how to continue their research, but also for how they position
themselves as actors within the situation they are exploring. In this personal positioning aspect,
DRTL differs from the three aspects of learning described by Edelson (2002) in his “what we learn
when we engage in design” article, which are: domain theories, design frameworks, and design
methodologies, which do not include the more personally experienced notion of design knowledge.
We claim that what makes DBR such a potentially fertile ground for DRTL is the methodological
alignment it affords, and the careful, iterative calibration between the pursuit of advancing theory
and design both in terms of DBR epistemology and ontology. Fig. 3 illustrates DRTL as part of
the model we suggest for calibrating DBR epistemologies and ontologies. We claim that DRTL
that results from following the principles of methodological alignment described above, leads to
what McKenney and Reeves (2012, 2018) describe as blending of analytical and creative mindsets,
which is crucial in developing CSCL innovation. It is worth noting, as we exemplify in the case
study below, that such iterative calibration within both our epistemologies and ontologies requires
a somewhat adventurous attitude to research. It also often involves developing unconventional
types of knowledge that may be difficult to judge and share through traditional forms of knowledge
dissemination (e.g., academic publishing, see Kali, 2016) and valuing (e.g., peer review, tenure
processes, etc.).

This is a pre-print of:
Kali, Y. & Hoadley, C. (in press). Design-based research methods in CSCL: Calibrating our epistemologies and ontologies. In U. Cress, C. Rosé,
A. Wise, and J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning. Springer.
11
Fig. 3 Model for calibrating DBR epistemologies and ontologies in DBR
Methodological alignment and DRTL: A CSCL case study
This case focuses on a DBR study conducted in the context a large-scale undergraduate level,
semester-long course in biology. In addition to a quick summary of the story already told
(described in detail in Sagy et al., 2019; Sagy et al., 2018; Tsaushu et al., 2012), the following
sections present the story behind the scenes of this DBR study. Specifically, it illustrates how the
back-and-forth movements within the abstraction-particularization curve enabled the DBR team
to reach higher degrees of methodological alignment, calibrating between the two modes of
inquiry, and how this eventually brought to their transformative learning, and the development of
PPK (Fig. 3).
Story already told – part 1: Redesigning an undergraduate biology course
The motivation for this project came from the course instructors—two biology professors who
have been teaching the course for many years in traditional ways. A DBR team was initiated, which
included the instructors, two science education researchers and two CSCL researchers. The
research was conducted by gradually intervening within the course. In each of the three years of
the study, a more advanced stage of the intervention was enacted with a new cohort of about 300
students. All three stages involved the use of a website that the team designed to go along with the
course, which was used differently at the three stages of the intervention (Sagy et al., 2018). At
the first stage (and first year of the study), the course was taught as it had been taught for years,
through lecturing in a large hall. The only difference was that students could use the course website
to review the contents taught in lectures. At the second stage, the instructors still gave lectures, but
students were required to use the website. At the third stage of the intervention the course website
replaced the lectures. In addition, the instructor served as facilitator in weekly “mini-conference”
meetings, each time on a different topic of the course with a different group of about 30 students.

This is a pre-print of:
Kali, Y. & Hoadley, C. (in press). Design-based research methods in CSCL: Calibrating our epistemologies and ontologies. In U. Cress, C. Rosé,
A. Wise, and J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning. Springer.
12
To prepare for this, students used team-websites designed for this purpose, which included content
resources as well as process scaffolds for developing team knowledge artifacts to share and discuss
in the “mini-conference”.
Untold story: Dilemma in research highlighting the need for methodological alignment
The DBR team’s initial assumption was that within each stage of the intervention they will be able
to find relationships between students’ patterns of use of the course website and their
understanding of the scientific content. They also assumed that they will find improvement in
learning outcomes and attitudes towards biology learning as the stage of the intervention became
more advanced. (This represents a “scientific” mode of inquiry aspect of this DBR endeavor—
going up the abstraction-particularization curve).
However, following design, enactment, and data analysis (representing a “design”, or
particularization mode of inquiry—going down the curve), both assumptions were refuted. That
is, no meaningful or interesting findings were found using what seemed straightforward means of
analysis (e.g., comparing students’ achievements in the course test between iterations, and seeking
relationships between students’ use of the website and their learning outcomes within each
iteration using learning analytics techniques). While interview data seemed to hint at deeper
learning as the intervention advanced, the processes that supported student learning (mediating
processes, in Sandoval’s, 2014 terminology) were not clear, nor were the design features
supporting them. Eventually, further back-and-forth movements within the abstraction-
particularization curve enabled identification of a gap between the values that guided students in
their learning process, and the instructors’ perceptions about these values (Sagy et al., 2019).
Story already told – part 2: The culture of learning continuum as a conceptual lens
This new lens, which the DBR team called “the culture of learning continuum (CLC)” (Sagy et
al., 2018), indicated that students who learned in more advanced versions of the course referred to
course features with higher degrees of what was described in the CLC as internal values.
Specifically, students were more likely to seek personal growth, appreciate the formative nature
of assessment, make efforts to learn (and not only succeed in the test), negotiate meaning with
peers (rather than seek the “right” answer for the test), and take ownership of their own learning
process.
Retrospective analysis of relationships between methodological alignment, DRTL and PPK
Retrospectively, the difficulty to explain the intervention outcomes in terms of mediating processes
at preliminary stages of the project, eventually, improved the teams’ methodological alignment.
Changing what was measured (culture of learning instead of students’ patterns of use of the
website) and how it was measured (measurement validity), transformed the DBR researchers
conception about the intervention. That is, they developed a renewed understanding of what the
intervention represented from a theoretical point of view (treatment validity). As a result, they
developed a renewed view of their role as researchers and designers within the study. They took
on a role that focused more on exploration within the unknown – being open to “build the plane

This is a pre-print of:
Kali, Y. & Hoadley, C. (in press). Design-based research methods in CSCL: Calibrating our epistemologies and ontologies. In U. Cress, C. Rosé,
A. Wise, and J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning. Springer.
13
while flying it”— discover the means of analysis while conducting the research, which required
the blending of analytical and creative mindsets (McKenney & Reeves, 2012, 2018).
But there was also a shift in the ontological work being conducted—as designers, they understood
that their role is to develop PPK in the form of not only the course’s website with its various digital
resources, but also the social activity structures that can support them and the cultural lens for
explaining the rationale behind them (the principled aspect of the practical tools). These turned out
to be crucial for continued implementation after the research was already over (evidence exists
that the instructors continued to implement the advanced versions of the course for many years).
This long-lasting effect was possible due to the transformative learning of the instructors too, who
were part of the research team, who adopted to their professional identity a role of cultivating a
culture of learning (Tsaushu et al., 2012).
Concluding remark
The literal meaning of DBR (design-based research) is that we are nudging both the epistemology
and ontology to follow scientific as well as design modes of inquiry, and knowledge outcomes.
This unusual property of the ontology, as pertained in PPK, calls for an unusual epistemology. At
the same time, the dual epistemic game of advancing theory while advancing design helps holding
the knowledge accountable. The eclecticism of DBR relates to the many ways we can intertwine
scientific and design modes of inquiry, going back-and-forth the abstraction-particularization
curve (Fig. 1). What unifies these activities is moving towards increased coherence, and therefore
systemic validity. Over 15 years ago, Hoadley noted that “the promise of having better alignment
in research—certain and sure links from theories to hypotheses to interventions to data gathering
activities to interpretation and application—should be a strong incentive to continue to pursue the
design-based research approach” (p. 211). The model we suggest for calibrating DBR
epistemologies and ontologies (Fig. 3) can assist in capitalizing on the dual epistemic and ontologic
game inherent to DBR, to spur creativity and innovation in rigorous research.
Thus, we claim that DBR, while accommodating multiple epistemic games, is not simply a laundry
list of ways to make knowledge. Rather, our flexibility in DBR’s epistemic games should be driven
by, and accountable to calibration between these games. In particular, we believe that the
inherently embedded and contextualized nature of CSCL, as well as its design orientation,
demands a set of knowledge activities which seek to use treatment, systemic and consequential
validity of research as the principles for moving between different epistemic framings, and
indeed—different knowledge ontologies. By doing so, we transform not only the types of
knowledge produced but also the knowers themselves, reshaping the role and perspective of
students, teachers, and DBR researchers.

This is a pre-print of:
Kali, Y. & Hoadley, C. (in press). Design-based research methods in CSCL: Calibrating our epistemologies and ontologies. In U. Cress, C. Rosé,
A. Wise, and J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning. Springer.
14
References
Bakker, A. (2018). Design research in education: A practical guide for early career researchers. Routledge.
Bhaskar, R. (1975). A realist theory of science. Leeds Books.
Bell, P. (2004). On the theoretical breadth of design-based research in education. Educational psychologist, 39(4),
243-253.
Bereiter, C. (2014). Principled practical knowledge: Not a bridge but a ladder. Journal of the Learning Sciences, 23(1),
4-17.
Bereiter, C. (2015). The practicality of principled practical knowledge: A response to Janssen, Westbroek, and Doyle.
Journal of the Learning Sciences, 24(1), 187-192.
Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex
interventions in classroom settings. The Journal of Learning Sciences, 2(2), 141–178.
Chi, M. T. H. (1992). Conceptual change within and across ontological categories: Examples from learning and
discovery in science. In R. Giere (Ed.), Cognitive models of science: Minnesota studies in the philosophy of
science (pp. 129–186). University of Minnesota Press.
Cobb, P., & Gravemeijer, K. (2008). Experimenting to support and understand learning processes. In A. E. Kelly, R.
A. Lesh, & J. Y. Baek (Eds.), Handbook of design research methods in education (pp. 68-95). Routledge.
Collins, A. (1990). Toward a design science of education. Center for Technology in Education.
Collins, A. (1992). Toward a design science of education. In New directions in educational technology (pp. 15-22).
Springer.
Dede, C. (2004). If design-based research is the answer, what is the question? A commentary on Collins, Joseph, and
Bielaczyc; diSessa and Cobb; and Fishman, Marx, Blumenthal, Krajcik, and Soloway in the JLS special issue
on design-based research. The Journal of the Learning Sciences, 13(1), 105-114.
Desforges, C. W. (2000). Familiar challenges and new approaches: Necessary advances in theory and methods in
research on teaching and learning. Retrieved February 1, 2019, from
https://web.archive.org/web/20180624013426/http://www.leeds.ac.uk/educol/documents/00001535.htm
Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry.
Educational Researcher, 32(1), 5-8.
diSessa, A. A. (1991). Local sciences: Viewing the design of human-computer systems as cognitive science. In J. M.
Carroll (Ed.), Designing interaction (pp. 162-202). Cambridge University Press.
Edelson, D. C. (2002). Design research: What we learn when we engage in design. The Journal of the Learning
sciences, 11(1), 105-121.
Fishman, B. J., Penuel, W. R., Allen, A. R., Cheng, B. H., & Sabelli, N. O. R. A. (2013). Design-based implementation
research: An emerging model for transforming the relationship of research and practice. National society for
the study of education, 112(2), 136-156.
Hoadley, C. (2002). Creating context: Design-based research in creating and understanding CSCL. In G. Stahl (Ed.),
Computer Support for Collaborative Learning 2002 (pp. 453–462). Erlbaum.
Hoadley, C. (2004). Methodological alignment in design-based research. Educational Psychologist, 39(4), 203-212.
Janssen, F., Westbroek, H., & Doyle, W. (2015). Practicality studies: How to move from what works in principle to
what works in practice. Journal of the Learning Sciences, 24(1), 176-186.
Kali, Y. (2006). Collaborative knowledge-building using the Design Principles Database. International Journal of
Computer Support for Collaborative Learning, 1(2), 187-201.
Kali, Y. (2008). The design principles database as means for promoting design-based research. In A. E. Kelly, R. A.
Lesh, & J. Y. Baek (Eds.), Handbook of design research methods in education: Innovations in science,
technology, engineering, and mathematics learning and teaching (pp. 423-438). Erlbaum.
Kali, Y. (2016). Transformative learning in design research: The story behind the scenes. Keynote presented at the
International Conference of the Learning Sciences, Singapore.
Kali, Y., Eylon, B.-S., McKenney, S., & Kidron, A. (2018). Design-centric research-practice partnerships: Three key
lenses for building productive bridges between theory and practice. In M. Spector, B. Lockee, & M. Childress
(Eds.), Learning, Design, and Technology: An International Compendium of Theory, Research, Practice,
and Policy. Springer. https://doi.org/10.1007/978-3-319-17727-4_122-1
Kali, Y., Levin-Peled, R., & Dori, Y. J. (2009). The role of design-principles in designing courses that promote
collaborative learning in higher-education. Computers in Human Behavior, 25(5), 1067–1078.
Kaptelinin, V., & Cole, M. (2002, March). Individual and collective activities in educational computer game playing.
In CSCL (Vol. 2, pp. 303-316).

This is a pre-print of:
Kali, Y. & Hoadley, C. (in press). Design-based research methods in CSCL: Calibrating our epistemologies and ontologies. In U. Cress, C. Rosé,
A. Wise, and J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning. Springer.
15
Kelly, A. E. (2004). Design research in education: Yes, but is it methodological? Journal of the Learning Sciences,
13(1), 115-128.
Koschmann, T. (1999). Computer Support for Collaboration and Learning, Journal of the Learning Sciences, 8(3-4),
495-497. https://doi.org/10.1080/10508406.1999.9672077
Laurel, B. (2013). Computers as theatre. Addison-Wesley.
Law, N., Niederhauser, D. S., Christensen, R., & Shear, L. (2016). A multilevel system of quality technology-enhanced
learning and teaching indicators. Educational Technology & Society, 19(3), 72–83.
Markauskaite, L., & Goodyear, P. (2017). Epistemic fluency and professional education: innovation, knowledgeable
action and actionable knowledge. Springer.
McKenney, S., & Reeves, T. C. (2012, 2018). Conducting educational design research. Routledge.
Mezirow, J. (1996). Contemporary paradigms of learning. Adult Education Quarterly, 46(3), 158–172.
Nelson, H. G., & Stolterman, E. (2012). The design way: Intentional change in an unpredictable world (2nd ed.). MIT
press.
O’Neill, D. K. (2012). Designs that fly: What the history of aeronautics tells us about the future of design-based
research in education. International Journal of Research and Method in Education, 35(2), 119–140.
doi:10.1080/1743727x.2012.683573
Paavola, S., Lipponen, L., & Hakkarainen, K. (2004). Models of innovative knowledge communities and three
metaphors of learning. Review of Educational Research, 74(4), 557-576.
Perkins, D. N. (1997). Epistemic games. International Journal of Educational Research, 27(1), 49-61.
Reeves, T. C. (2005). Design-based research in educational technology: Progress made, challenges remain.
Educational Technology, 45(1), 48-52.
Sagy, O., Hod, Y., & Kali, Y. (2019). Teaching and learning cultures in higher education: a mismatch in conceptions.
Higher Education Research & Development, 38(4), 849-863.
Sagy, O., Kali, Y., Tsaushu, M., & Tal, T. (2018). The Culture of Learning Continuum: promoting internal values in
higher education. Studies in Higher Education, 43(3), 416-436.
Sandoval, W. (2014). Conjecture mapping: An approach to systematic educational design research. Journal of the
Learning Sciences, 23(1), 18-36.
Simon, H. A. (1969). The sciences of the artificial. MIT press.
Slotta, J. D. (2011). In defense of chi’s ontological incompatibility hypothesis. Journal of the Learning Sciences,
20(1), 151–162.
Smit, J., van Eerde, H. A. A., & Bakker, A. (2013). A conceptualisation of whole-class scaffolding. British
Educational Research Journal, 39(5), 817–834.
Stahl, G., Koschmann, T., & Suthers, D. (2006). Computer-supported collaborative learning: An historical
perspective. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 409-426). Cambridge
University Press.
Tabak, I. (2004). Reconstructing context: Negotiating the tension between exogenous and endogenous educational
design. Educational Psychologist, 39(4), 225-233.
Toulmin, S. E. (1958). The uses of argument. Cambridge University Press.
Tsaushu, M., Tal, T., Sagy, O., Kali, Y., Gepstein, S., & Zilberstein, D. (2012). Peer learning and support of technology
in an undergraduate biology course to enhance deep learning. CBE—Life Sciences Education, 11(4), 402-
412.
Additional Reading
1. McKenney, S., & Reeves, T. C. (2012, 2018). Conducting educational design research. Routledge.
This is book provides a generic model for conducting DBR and explains in detail its main elements:
analysis and exploration; design and construction; evaluation and reflection; and implementation and
spread. The book also offers guidance for proposing, reporting and advancing DBR, and is recommended
especially for graduate students, as well as experienced researchers who are new to this approach.
2. Kelly, A. E. (2004). Design research in education: Yes, but is it methodological? Journal of the Learning
Sciences, 13(1), 115-128.
The critique in this paper, concerning a missing argumentative grammar in DBR, has provoked an ongoing
debate, as well as various approaches for enhancing rigor in DBR. It is a good starting point for researchers

This is a pre-print of:
Kali, Y. & Hoadley, C. (in press). Design-based research methods in CSCL: Calibrating our epistemologies and ontologies. In U. Cress, C. Rosé,
A. Wise, and J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning. Springer.
16
who are already conducting DBR and are required to convince reviewers of the rigor in their work to show
that Yes - it can be methodological!
3. Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational
inquiry. Educational Researcher, 32(1), 5-8.
This paper, published in a special issue of Educational Researcher (the first special issue published on
DBR) is used in the current chapter to characterize DBR, as it encapsulates what critics find challenging
about DBR, which our model for calibrating epistemologies and ontologies addresses.
4. Hoadley, C. (2004). Methodological alignment in design-based research. Educational Psychologist, 39(4),
203-212.
This paper provides a detailed explanation of the notion of methodological alignment, which is one of the
two components (the other being DRTL) in our model for calibrating DBR epistemologies and ontologies.
5. Sagy, O., Kali, Y., Tsaushu, M., & Tal, T. (2018). The Culture of Learning Continuum: promoting internal
values in higher education. Studies in Higher Education, 43(3), 416-436.
This DBR study is the case we use in our chapter to illustrate the “behind the scenes” DRTL processes. The
study also illustrates the use of Sandoval’s (2014) conjecture mapping in DBR. We claim that such
mapping highlights the tension within both epistemic and ontological games within the abstraction-
particularization curve.