Significance of a Study: Revisiting the “So What” Question

Abstract

Every researcher wants their study to matter—to make a positive difference for their professional communities. To ensure your study matters, you can formulate clear hypotheses and choose methods that will test them well, as described in Chaps. 1, 2, 3 and 4. You can go further, however, by considering some of the terms commonly used to describe the importance of studies, terms like significance, contributions, and implications. As you clarify for yourself the meanings of these terms, you learn that whether your study matters depends on how convincingly you can argue for its importance. Perhaps most surprising is that convincing others of its importance rests with the case you make before the data are ever gathered. The importance of your hypotheses should be apparent before you test them. Are your predictions about things the profession cares about? Can you make them with a striking degree of precision? Are the rationales that support them compelling? You are answering the “So what?” question as you formulate hypotheses and design tests of them. This means you can control the answer. You do not need to cross your fingers and hope as you collect data.

Full text

105
Chapter 5
Signicance ofaStudy: Revisiting
the“So What” Question
Part I.Setting theGroundwork
One of the most common questions asked of researchers is “So what?” What differ-
ence does your study make? Why are the ndings important? The “so what” ques-
tion is one of the most basic questions, often perceived by novice researchers as the
most difcult question to answer. Indeed, addressing the “so what” question contin-
ues to challenge even experienced researchers. It is not always easy to articulate a
convincing argument for the importance of your work. It can be especially difcult
to describe its importance without falling into the trap of making claims that reach
beyond the data.
That this issue is a challenge for researchers is illustrated by our analysis of
reviewer comments for JRME. About one-third of the reviews for manuscripts that
were ultimately rejected included concerns about the importance of the study. Said
another way, reviewers felt the “So what?” question had not been answered. To
paraphrase one journal reviewer, “The manuscript left me unsure of what the contri-
bution of this work to the eld’s knowledge is, and therefore I doubt its signi-
cance.” We expect this is a frequent concern of reviewers for all research journals.
Our goal in this chapter is to help you navigate the pressing demands of journal
reviewers, editors, and readers for demonstrating the importance of your work while
staying within the bounds of acceptable claims based on your results. We will begin
by reviewing what we have said about these issues in previous chapters. We will
then clarify one of the confusing aspects of developing appropriate arguments—the
absence of consensus denitions of key terms such as signicance, contributions,
and implications. Based on the denitions we propose, we will examine the critical
role of alignment for realizing the potential signicance of your study. Because the
importance of your study is communicated through your evolving research paper,
we will fold suggestions for writing your paper into the discussion of creating and
executing your study.
© The Author(s) 2023
J. Hiebert et al., Doing Research: A New Researcher’s Guide,
Research in Mathematics Education,
https://doi.org/10.1007/978-3-031-19078-0_5

106
A confusing aspect of developing appropriate arguments is
the absence of consensus definitions of key terms such as
significance, contributions, and implications.
We laid the groundwork in Chap. 1 for what we consider to be important research
in education:
In our view, the ultimate goal of education is to offer all students the best possible learning
opportunities. So, we believe the ultimate purpose of scientic inquiry in education is to
support the improvement of learning opportunities for all students…. If there is no way to
imagine a connection to improving learning opportunities for students, even a distant con-
nection, we recommend you reconsider whether it is an important hypothesis within the
education community.
Of course, you might prefer another “ultimate purpose” for research in education.
That’s ne. The critical point is that the argument for the importance of the hypoth-
eses you are testing should be connected to the value of a long-term goal you can
describe. As long as most of the educational community agrees with this goal, and
you can show how testing your hypotheses will move the eld forward to achieving
this goal, you will have developed a convincing argument for the importance of
your work.
In Chap. 2, we argued the importance of your hypotheses can and should be
established before you collect data. Your theoretical framework should carry the
weight of your argument because it should describe how your hypotheses will
extend what is already known. Your methods should then show that you will test
your hypotheses in an appropriate way—in a way that will allow you to detect how
the results did, and did not, conrm the hypotheses. This will, in turn, allow you to
formulate revised hypotheses. We described establishing the importance of your
study by saying, “The importance can come from the fact that, based on the results,
you will be able to offer revised hypotheses that help the eld better understand an
issue relevant for improving all students’ learning opportunities.”
The ideas from Chaps. 1, 2, and 3 go a long way toward setting the parameters
for what counts as an important study and how its importance can be determined.
Chapter 4 focused on ensuring that the importance of a study can be realized. The
next section lls in the details by proposing denitions for the most common terms
used to claim importance: signicance, contributions, and implications.
You might notice that we do not have a chapter dedicated to discussing the pre-
sentation of the ndings—that is, a “results” chapter. We do not mean to imply that
presenting results is trivial. However, we believe that if you follow our recommen-
dations for writing your evolving research paper, presenting the results will be quite
straightforward. The key is to present your results so they can be most easily com-
pared with your predictions. This means, among other things, organizing your pre-
sentation of results according to your earlier presentation of hypotheses.
5 Signicance ofaStudy: Revisiting the“So What” Question

107
Part II.Clarifying Importance by Revisiting theDenitions
ofKey Terms
What does it mean to say your ndings are signicant? Statistical signicance is
clear. There are widely accepted standards for determining the statistical signi-
cance of ndings. But what about educational signicance? Is this the same as
claiming that your study makes an important contribution? Or, that your study has
important implications? Different researchers might answer these questions in dif-
ferent ways. When key terms like these are overused, their denitions gradually
broaden or shift, and they can lose their meaning. That is unfortunate, because it
creates confusion about how to develop claims for the importance of a study.
By clarifying the denitions, we hope to clarify what is required to claim that a
study is signicant, that it makes a contribution, and that it has important implica-
tions. Not everyone denes the terms as we do. Our denitions are probably a bit
narrower or more targeted than those you may encounter elsewhere. Depending on
where you want to publish your study, you may want to adapt your use of these
terms to match more closely the expectations of a particular journal. But the way we
dene and address these terms is not antithetical to common uses. And we believe
ridding the terms of unnecessary overlap allows us to discriminate among different
key concepts with respect to claims for the importance of research studies. It is not
necessary to dene the terms exactly as we have, but it is critical that the ideas
embedded in our denitions be distinguished and that all of them be taken into
account when examining the importance of a study.
We will use the following denitions:
• Signicance: The importance of the problem, questions, and/or hypotheses for
improving the learning opportunities for all students (you can substitute a differ-
ent long-term goal if its value is widely shared). Signicance can be determined
before data are gathered. Signicance is an attribute of the research problem, not
the research ndings.
• Contributions: The value of the ndings for revising the hypotheses, making
clear what has been learned, what is now better understood.
• Implications: Deductions about what can be concluded from the ndings that are
not already included in “contributions.” The most common deductions in educa-
tional research are for improving educational practice. Deductions for research
practice that are not already dened as contributions are often suggestions about
research methods that are especially useful or methods to avoid.
Signicance
The signicance of a study is built by formulating research questions and hypothe-
ses you connect through a careful argument to a long-term goal of widely shared
value (e.g., improving learning opportunities for all students). Signicance applies
both to the domain in which your study is located and to your individual study. The
Part II.Clarifying Importance by Revisiting theDenitions ofKey Terms

108
signicance of the domain is established by choosing a goal of widely shared value
and then identifying a domain you can show is connected to achieving the goal. For
example, if the goal to which your study contributes is improving the learning
opportunities for all students, your study might aim to understand more fully how
things work in a domain such as teaching for conceptual understanding, or prepar-
ing teachers to attend to all students, or designing curricula to support all learners,
or connecting learning opportunities to particular learning outcomes.
The signicance of your individual study is something you build; it is not prede-
termined or self-evident. Signicance of a study is established by making a case for
it, not by simply choosing hypotheses everyone already thinks are important.
Although you might believe the signicance of your study is obvious, readers will
need to be convinced.
Significance can be determined before data are gathered.
Significance is an attribute of the research problem, not
the research findings.
Signicance is something you develop in your evolving research paper. The the-
oretical framework you present connects your study to what has been investigated
previously. Your argument for signicance of the domain comes from the signi-
cance of the line of research of which your study is a part. The signicance of your
study is developed by showing, through the presentation of your framework, how
your study advances this line of research. This means the lion’s share of your answer
to the “So what?” question will be developed as part of your theoretical framework.
Although dening signicance as located in your paper prior to presenting results
is not a denition universally shared among educational researchers, it is becoming
an increasingly common view. In fact, there is movement toward evaluating the
signicance of a study based only on the rst sections of a research paper—the sec-
tions prior to the results (Makel etal., 2021).
In addition to addressing the “So what?” question, your theoretical framework
can address another common concern often voiced by readers: “What is so interest-
ing? I could have predicted those results.” Predictions do not need to be surprising
to be interesting and signicant. The signicance comes from the rationales that
show how the predictions extend what is currently known. It is irrelevant how many
researchers could have made the predictions. What makes a study signicant is that
the theoretical framework and the predictions make clear how the study will increase
the eld’s understanding toward achieving a goal of shared value.
5 Signicance ofaStudy: Revisiting the“So What” Question

109
What makes a study significant is that the theoretical
framework and the predictions make clear how the study
will increase the field’s understanding toward achieving a
goal of shared value.
An important consequence of interpreting signicance as a carefully developed
argument for the importance of your research study within a larger domain is that it
reveals the advantage of conducting a series of connected studies rather than single,
disconnected studies. Building the signicance of a research study requires time
and effort. Once you have established signicance for a particular study, you can
build on this same argument for related studies. This saves time, allows you to con-
tinue to rene your argument across studies, and increases the likelihood your stud-
ies will contribute to the eld.
Contributions
As we have noted, in elds as complicated as education, it is unlikely that your
predictions will be entirely accurate. If the problem you are investigating is signi-
cant, the hypotheses will be formulated in such a way that they extend a line of
research to understand more deeply phenomena related to students’ learning oppor-
tunities or another goal of shared value. Often, this means investigating the condi-
tions under which phenomena occur. This gets complicated very quickly, so the data
you gather will likely differ from your predictions in a variety of ways. The contri-
butions your study makes will depend on how you interpret these results in light of
the original hypotheses.
A study’s contribution lies in the value of its findings for
revising the hypotheses, making clear what has been
learned, what is now better understood.
Contributions Emerge fromRevisions toyour Hypotheses
We view interpreting results as a process of comparing the data with the predictions
and then examining the way in which hypotheses should be revised to more fully
account for the results. Revising will almost always be warranted because, as we
noted, predictions are unlikely to be entirely accurate. For example, if researchers
expect Outcome A to occur under specied conditions but nd that it does not occur
to the extent predicted or actually does occur but without all the conditions, they
Part II.Clarifying Importance by Revisiting theDenitions ofKey Terms

110
must ask what changes to the hypotheses are needed to predict more accurately the
conditions under which Outcome A occurred. Are there, for example, essential con-
ditions that were not anticipated and that should be included in the revised
hypotheses?
Consider an example from a recently published study (Wang et al., 2021). A
team of researchers investigated the following research question: “How are stu-
dents’ perceptions of their parents’ expectations related to students’ mathematics-
related beliefs and their perceived mathematics achievement?” The researchers
predicted that students’ perceptions of their parents’ expectations would be highly
related to students’ mathematics-related beliefs and their perceived mathematics
achievement. The rationale was based largely on prior research that had consistently
found parents’ general educational expectations to be highly correlated with stu-
dents’ achievement.
The ndings showed that Chinese high school students’ perceptions of their par-
ents’ educational expectations were positively related to these students’ mathematics-
related beliefs. In other words, students who believed their parents expected them to
attain higher levels of education had more desirable mathematics-related beliefs.
However, students’ perceptions of their parents’ expectations about mathematics
achievement were not related to students’ mathematics-related beliefs in the same
way as the more general parental educational expectations. Students who reported
that their parents had no specic expectations possessed more desirable mathematics-
related beliefs than all other subgroups. In addition, these students tended to per-
ceive their mathematics achievement rank in their class to be higher on average than
students who reported that their parents expressed some level of expectation for
mathematics achievement.
Because this nding was not predicted, the researchers revised the original
hypothesis. Their new prediction was that students who believe their parents have
no specic mathematics achievement expectations possess more positive
mathematics- related beliefs and higher perceived mathematics achievement than
students who believe their parents do have specic expectations. They developed a
revised rationale that drew on research on parental pressure and mathematics anxi-
ety, positing that parents’ specic mathematics achievement expectations might
increase their children’s sense of pressure and anxiety, thus fostering less positive
mathematics-related beliefs. The team then conducted a follow-up study. Their nd-
ings aligned more closely with the new predictions and afrmed the better explana-
tory power of the revised rationale. The contributions of the study are found in this
increased explanatory power—in the new understandings of this phenomenon con-
tained in the revisions to the rationale.
Interpreting ndings in order to revise hypotheses is not a straightforward task.
Usually, the rationales blend multiple constructs or variables and predict multiple
outcomes, with different outcomes connected to different research questions and
addressed by different sets of data. Nevertheless, the contributions of your study
depend on specifying the differences between your original hypotheses and your
revised hypotheses. What can you explain now that you could not explain before?
5 Signicance ofaStudy: Revisiting the“So What” Question

111
We believe that revising hypotheses is an optimal response to any question of
contributions because a researcher’s initial hypotheses plus the revisions suggested
by the data are the most productive way to tie a study into the larger chain of research
of which it is a part. Revised hypotheses represent growth in knowledge. Building
on other researchers’ revised hypotheses and revising them further by more explic-
itly and precisely describing the conditions that are expected to inuence the out-
comes in the next study accumulates knowledge in a form that can be recorded,
shared, built upon, and improved.
The signicance of your study is presented in the opening sections of your evolv-
ing research paper whereas the contributions are presented in the nal section, after
the results. In fact, the central focus in this “Discussion” section should be a speci-
cation of the contributions (note, though, that this guidance may not fully align
with the requirements of some journals).
Contributions Answer theQuestion ofGeneralizability
A common and often contentious, confusing issue that can befuddle novice and
experienced researchers alike is the generalizability of results. All researchers pre-
fer to believe the results they report apply to more than the sample of participants in
their study. How important would a study be if the results applied only to, say, two
fourth-grade classrooms in one school, or to the exact same tasks used as measures?
How do you decide to which larger population (of students or tasks) your results
could generalize? How can you state your claims so they are precisely those justi-
ed by the data?
To illustrate the challenge faced by researchers in answering these questions, we
return to the JRME reviewers. We found that 30% of the reviews expressed concerns
about the match between the results and the claims. For manuscripts that ultimately
received a decision of Reject, the majority of reviewers said the authors had over-
reached—the claims were not supported by the data. In other words, authors gener-
alized their claims beyond those that could be justied.
The Connection Between Contributions and Generalizability In our view,
claims about contributions can be examined productively alongside considerations
of generalizability. To make the case for this view, we need to back up a bit. Recall
that the purpose of research is to understand a phenomenon. To understand a phe-
nomenon, you need to determine the conditions under which it occurs. Consequently,
productive hypotheses specify the conditions under which the predictions hold and
explain why and how these conditions make a difference. And the conditions set the
parameters on generalizability. They identify when, where, and for whom the effect
or situation will occur. So, hypotheses describe the extent of expected generaliz-
ability, and revised hypotheses that contain the contributions recalibrate generaliz-
ability and offer new predictions within these parameters.
Part II.Clarifying Importance by Revisiting theDenitions ofKey Terms

112
An Example That Illustrates the Connection In Chap. 4, we introduced an
example with a research question asking whether second graders improve their
understanding of place value after a specially designed instructional intervention.
We suggested asking a few second and third graders to complete your tasks to see if
they generated the expected variation in performance. Suppose you completed this
pilot study and now have satisfactory tasks. What conditions might inuence the
effect of the intervention? After careful study, you developed rationales that sup-
ported three conditions: the entry level of students’ understanding, the way in which
the intervention is implemented, and the classroom norms that set expectations for
students’ participation.
Suppose your original hypotheses predicted the desired effect of the intervention
only if the students possessed an understanding of several concepts on which place
value is built, only if the intervention was implemented with delity to the detailed
instructional guidelines, and only if classroom norms encouraged students to par-
ticipate in small-group work and whole-class discussions. Your claims of generaliz-
ability will apply to second-grade settings with these characteristics.
Now suppose you designed the study so the intervention occurred in ve second-
grade classrooms that agreed to participate. The pre-intervention assessment showed
all students with the minimal level of entry understanding. The same well-trained
teacher was employed to teach the intervention in all ve classrooms, none of which
included her own students. And you learned from prior observations and reports of
the classroom teachers that three of the classrooms operated with the desired class-
room norms, but two did not. Because of these conditions, your study is now
designed to test one of your hypotheses—the desired effect will occur only if class-
room norms encouraged students to participate in small-group work and whole-
class discussions. This is the only condition that will vary; the other two (prior level
of understanding and delity of implementation) are the same across classrooms so
you will not learn how these affect the results.
Suppose the classrooms performed equally well on the post-intervention assess-
ments. You expected lower performance in the two classrooms with less student
participation, so you need to revise your hypotheses. The challenge is to explain the
higher-than-expected performance of these students. Because you were interested
in understanding the effects of this condition, you observed several lessons in all the
classrooms during the intervention. You can now use this information to explain
why the intervention worked equally well in classrooms with different norms.
Your revised hypothesis captures this part of your study’s contribution. You can
now say more about the ways in which the intervention can help students improve
their understanding of place value because you have different information about the
role of classroom norms. This, in turn, allows you to specify more precisely the
nature and extent of the generalizability of your ndings. You now can generalize
your ndings to classrooms with different norms. However, because you did not
learn more about the impact of students’ entry level understandings or of different
kinds of implementation, the generalizability along these dimensions remains as
limited as before.
5 Signicance ofaStudy: Revisiting the“So What” Question

113
This example is simplied. In many studies, the ndings will be more compli-
cated, and more conditions will likely be identied, some of which were anticipated
and some of which emerged while conducting the study and analyzing the data.
Nevertheless, the point is that generalizability should be tied to the conditions that
are expected to affect the results. Further, unanticipated conditions almost always
appear, so generalizations should be conservative and made with caution and humil-
ity. They are likely to change after testing the new predictions.
Contributions Are Assured When Hypotheses Are Signicant andMethods
Are Appropriate andAligned
We have argued that the contributions of your study are produced by the revised
hypotheses you can formulate based on your results. Will these revisions always
represent contributions to the eld? What if the revisions are minor? What if your
results do not inform revisions to your hypotheses?
We will answer these questions briey now and then develop them further in Part
IV of this chapter. The answer to the primary question is “yes,” your revisions will
always be a contribution to the eld if (1) your hypotheses are signicant and (2)
you crafted appropriate methods to test the hypotheses. This is true even if your
revisions are minor or if your data are not as informative as you expected. However,
this is true only if you meet the two conditions in the earlier sentence. The rst con-
dition (signicant hypotheses) can be satised by following the suggestions in the
earlier section on signicance. The second condition (appropriate methods) is
addressed further in Part III in this chapter.
Implications
Before examining the role of methods in connecting signicance with important
contributions, we elaborate briey our denition of “implications.” We reserve
implications for the conclusions you can logically deduce from your ndings that
are not already presented as contributions. This means that, like contributions,
implications are presented in the Discussion section of your research paper.
Many educational researchers present two types of implications: implications for
future research and implications for practice. Although we are aware of this com-
mon usage, we believe our denition of “contributions” cover these implications.
Clarifying why we call these “contributions” will explain why we largely reserve
the word “implications” for recommendations regarding methods.
Part II.Clarifying Importance by Revisiting theDenitions ofKey Terms

114
Implications forFuture Research
Implications for future research often include (1) recommendations for empirical
studies that would extend the ndings of this study, (2) inferences about the useful-
ness of theoretical constructs, and (3) conclusions about the advisability of using
particular kinds of methods. Given our earlier denitions, we prefer to label the rst
two types of implications as contributions.
Consider recommendations for empirical studies. After analyzing the data and
presenting the results, we have suggested you compare the results with those pre-
dicted, revise the rationales for the original predictions to account for the results,
and make new predictions based on the revised rationales. It is precisely these new
predictions that can form the basis for recommending future research. Testing these
new predictions is what would most productively extend this line of research. It can
sometimes sound as if researchers are recommending future studies based on
hunches about what research might yield useful ndings. But researchers can do
better than this. It would be more productive to base recommendations on a careful
analysis of how the predictions of the original study could be sharpened and
improved.
Now consider inferences about the usefulness of theoretical constructs. Our
argument for labeling these inferences as contributions is similar. Rationales for
predictions are where the relevant theoretical constructs are located. Revisions to
these rationales based on the differences between the results and the predictions
reveal the theoretical constructs that were afrmed to support accurate predictions
and those that must be revised. In our view, usefulness is determined through this
revision process.
Implications that do not come under our meaning of contributions are in the third
type of implications, namely the appropriateness of methods for generating rich
contributions. These kinds of implications are produced by your evaluation of your
methods: research design, sampling procedures, tasks, data collection procedures,
and data analyses. Although not always included in the discussion of ndings, we
believe it would be helpful for researchers to identify particular methods that were
useful for conducting their study and those that should be modied or avoided. We
believe these are appropriately called implications.
Implications forPractice
If the purpose of research is to better understand how to improve learning opportu-
nities for all students, then it is appropriate to consider whether implications for
improving educational practice can be drawn from the results of a study. How are
these implications formulated? This is an important question because, in our view,
these claims often come across as an afterthought, “Oh, I need to add some implica-
tions for practice.” But here is the sobering reality facing researchers: By any mea-
sure, the history of educational research shows that identifying these implications
has had little positive effect on practice.
5 Signicance ofaStudy: Revisiting the“So What” Question

115
Perhaps the most challenging task for researchers who attempt to draw implica-
tions for practice is to interpret their ndings for appropriate settings. A researcher
who studied the instructional intervention for second graders on place value and
found that average performance in the intervention classrooms improved more than
in the textbook classrooms might be tempted to draw implications for practice.
What should the researcher say? That second-grade teachers should adopt the inter-
vention? Such an implication would be an overreach because, as we noted earlier,
the ndings cannot be generalized to all second-grade classrooms. Moreover, an
improvement in average performance does not mean the intervention was better for
all students.
The challenge is to identify the conditions under which the intervention would
improve the learning opportunities for all students. Some of these conditions will be
identied as the theoretical framework is built because the predictions need to
account for these conditions. But some will be unforeseen, and some that are identi-
ed will not be informed by the ndings. Recall that, in the study described earlier,
a condition of entry level of understanding was hypothesized but the design of the
study did not allow the researcher to draw any conclusions about its effect.
What can researchers say about implications for practice given the complexities
involved in generalizing ndings to other settings? We offer two recommendations.
First, because it is difcult to specify all the conditions under which a phenomenon
occurs, it is rarely appropriate to prescribe an educational practice. Researchers
cannot anticipate the conditions under which individual teachers operate, conditions
that often require adaptation of a suggested practice rather than implementation of
a practice as prescribed.
Our second recommendation comes from returning to the purpose for educa-
tional research—to understand more fully how to improve learning opportunities
for all students (or to achieve another goal of widely shared value). As we have
described, understanding comes primarily from building and reevaluating rationales
for your predictions. If you reach a new understanding related to improving learning
opportunities, an understanding that could have practical implications, we recom-
mend you share this understanding as an implication for practice.
For example, suppose the researcher who found better average performance of
second graders after the intervention on place value had also studied several condi-
tions under which performance improved. And suppose the researcher found that
most students who did not improve their performance misunderstood a concept that
appeared early in the intervention (e.g., the multiplicative relationship between
positional values of a numeral). An implication for practice the researcher might
share would be to describe the potential importance of understanding this concept
early in the sequence of activities if teachers try out this intervention.
If you use our denitions, these implications for practice would be presented as
contributions because they emerge directly from reevaluating and revising your
rationales. We believe it is appropriate to use “Contributions” as the heading for this
section in the Discussion section of your research paper. However, if editors prefer
“Implications” we recommend following their suggestion.
Part II.Clarifying Importance by Revisiting theDenitions ofKey Terms

116
We want to be clear that the terms you use for the different ways your study is
important is not critical. We chose to dene the terms signicance, contributions,
and implications in very specic and not universally shared ways to distinguish all
the meanings of importance you should consider. Some of these can be established
through your theoretical framework, some by the revisions of your hypotheses, and
some by reecting on the value of particular methods. The important thing, from
our point of view, is that the ideas we dened for each of these terms are distin-
guished and recognized as specic ways of determining the importance of your study.
Part III.TheRole ofMethods inDetermining Contributions
We have argued that every part of the study (and of the evolving research paper)
should be aligned. All parts should be connected through a coherent chain of rea-
soning. In this chapter, we argue that the chain of reasoning is not complete until the
methods are presented and the results are interpreted and discussed. The methods of
the study create a bridge that connects the introductory material (research questions,
theoretical framework, literature review, hypotheses) with the results and
interpretations.
The role that methods play in scientic inquiry is to ensure that your hypotheses
will be tested appropriately so the signicance of your study will yield its potential
contributions. To do this, the methods must do more than follow the standard guide-
lines and be technically correct (see Chap. 4). They must also t with the surround-
ing parts of the study. We call this coherence.
The role that methods play in scientific inquiry is to ensure
that your hypotheses will be tested appropriately so the
significance of your study will yield its potential contribu-
tions.
Coherence Across thePhases ofScientic Inquiry
Coherence means the parts of a whole are fully aligned. When doing scientic
inquiry, the early parts or phases should motivate the later phases. The methods you
use should be motivated or explained by the earlier phases (e.g., research questions,
theoretical framework, hypotheses). Your methods, in turn, should produce results
that can be interpreted by comparing them with your predictions. Methods are
aligned with earlier phases when you can use the rationales contained in your
hypotheses to decide what kinds of data are needed to test your predictions, how
5 Signicance ofaStudy: Revisiting the“So What” Question

117
1. Research Question /
Hypothesis
•Prediction
•Rationale
2. Methods
•Design
•Measures/Data
•Analyses
3. Results/Findings
•CompareData to
Predictions
•CompareData-
basedClaimsto
Rationales
4. Discussion
•Contributions
•Implications
Fig. 5.1 The Chain of Coherence That Runs Through All Parts of a Research Study
best to gather these kinds of data, and what analyses should be performed (see
Chap. 4 and Cai etal., 2019a).
For a visual representation of this coherence, see Fig.5.1. Each box identies an
aspect of scientic inquiry. Hypotheses (shown in Box 1) include the rationales and
predictions. Because the rationales encompass the theoretical framework and the
literature review, Box 1 establishes the signicance of the study. Box 2 represents
the methods, which we dened in Chap. 4 as the entire set of procedures you will
use, including the basic design, measures for collecting data, and analytic
approaches. In Fig.5.1, the hypothesis in Box 1 points you to the methods you will
use. That is, you will choose methods that provide data for analyses that will gener-
ate results or ndings (Box 3) that allow you to make comparisons against your
predictions. Based on those comparisons, you will revise your hypotheses and
derive the contributions and implications of your study (Box 4).
We intend Fig.5.1 to carry several messages. One is that coherence of a study
and the associated research paper require all aspects of the study to ow from one
into the other. Each set of prior entries must motivate and justify the next one. For
example, the data and analyses you intend to gather and use in Box 2 (Methods)
must be those that are motivated and explained by the research question and hypoth-
esis (prediction and rationale) in Box 1.
A second message in the gure is that coherence includes Box 4, “Discussion.”
Aligned with the rst three boxes, the fourth box ows from these boxes but is also
constrained by them. The contributions and implications authors describe in the
Discussion section of the paper cannot go beyond what is allowed by the original
hypotheses and the revisions to these hypotheses indicated by the ndings.
For each hypothesis (and thus each research question) in your study, you
should be able to trace an entire chain of coherence. In a complex study
with multiple hypotheses (and thus multiple research questions), it can be
extremely helpful to diagram these connections (or make a table of them)
so that you can explicitly link each research question to the data collected
for that question, to the analyses that will be conducted on those data to
address that question, to the results obtained for that question, and nally
to the contributions related to that question. A diagram or table of these
links can help you to maintain coherence both while conducting your
study and while writing your research paper.
Part III.TheRole ofMethods inDetermining Contributions

118
Methods Enable Signicance toYield Contributions
We begin this section by identifying a third message conveyed in Fig. 5.1. The
methods of the study, represented by Box 2, provide a bridge that connects the sig-
nicance of the study (Box 1) with the contributions of the study (Box 4). The
results (Box 3) indicate the nature of the contributions by determining the revisions
to the original hypotheses.
In our view, the connecting role played by the methods is often underappreci-
ated. Crafting appropriate methods aligned with the signicance of the study, on one
hand, and the interpretations, on the other, can determine whether a study is judged
to make a contribution.
If the hypotheses are established as signicant, and if appropriate methods are
used to test the predictions, the study will make important contributions even if the
data are not statistically signicant. We can say this another way. When researchers
establish the signicance of the hypotheses (i.e., convince readers they are of inter-
est to the eld) and use methods that provide a sound test of these hypotheses, the
data they present will be of interest regardless of how they turn out. This is why
Makel etal. (2021) endorse a review process for publication that emphasizes the
signicance of the study as presented in the rst sections of a research paper.
Treating the methods as connecting the introductory arguments to the interpreta-
tions of data prevent researchers from making a common mistake: When writing the
research paper, some researchers lose track of the research questions and/or the
predictions. In other words, results are presented but are not interpreted as answers
to the research questions or compared with the predictions. It is as if the introduc-
tory material of the paper begins one story, and the interpretations of results ends a
different story. Lack of alignment makes it impossible to tell one coherent story.
A nal point is that the alignment of a study cannot be evaluated and appreciated
if the methods are not fully described. Methods must be described clearly and com-
pletely in the research paper so readers can see how they ow from the earlier
phases of the study and how they yield the data presented. We suggested in Chap. 4
a rule of thumb for deciding whether the methods have been fully described:
“Readers should be able to replicate the study if they wish.”
Part IV.Special Considerations that Affect aStudy’s
Contributions
We conclude Chap. 5 by addressing two additional issues that can affect how
researchers interpret the results and make claims about the contributions of a study.
Usually, researchers deal with these issues in the Discussion section of their research
paper, but we believe it is useful to consider them as you plan and conduct your
study. The issues can be posed as questions: How should I treat the limitations of my
study? How should I deal with ndings that are completely unexpected?
5 Signicance ofaStudy: Revisiting the“So What” Question

119
Limitations ofaStudy
We can identify two kinds of limitations: (1) limitations that constrain your ability
to interpret your results because of unfortunate choices you made, and (2) limita-
tions that constrain your ability to generalize your results because of missing vari-
ables you could not t into the scope of your study or did not anticipate. We
recommend different ways of dealing with these.
Limitations DuetoUnfortunate Choices
All researchers make unfortunate choices. These are mistakes that could have been
prevented. Often, they are choices in how a study was designed and/or executed.
Maybe the sample did not have the characteristics assumed, or a task did not assess
what was expected, or the intervention was not implemented as planned. Although
many unfortunate choices can be prevented by thinking through the consequences
of every decision or by conducting a well-designed pilot study or two, some will
occur anyway. How should you deal with them?
The consequence of unfortunate choices is that the data do not test the hypothe-
ses as precisely or completely as hoped. When this happens, the data must be inter-
preted with these constraints in mind. Almost always, this limits the researcher to
making fewer or narrower claims than desired about differences and similarities
between the results and the predictions. Usually this means conclusions about the
ways in which the rationales must be revised require extra qualications. In other
words, claims about contributions of the study must be made with extra caution.
Research papers frequently include a subsection in the Discussion called
“Limitations of the Study.” Researchers often use this subsection to identify the
study’s limitations by describing the unfortunate choices, but they do not always
spell out how these limitations should affect the contributions of the paper.
Sometimes, it appears that researchers are simply checking off a requirement to
identify the limitations by saying something like “The results should be interpreted
with caution.” But this does not help readers understand exactly what cautions
should be applied and it does not hold researchers accountable for the limitations.
We recommend something different. We suggest you do the hard work of gur-
ing out how the data should be interpreted in light of the limitations and share these
details with the readers. You might do this when the results are presented or when
you interpret them. Rather than presenting your claims about the contributions of
the study and then saying readers should interpret these with “caution” because of
the study’s limitations, we suggest presenting only those interpretations and claims
of contributions that can be made with the limitations in mind.
Part IV.Special Considerations that Affect aStudy’s Contributions

120
We suggest you do the hard work of figuring out how the
data should be interpreted in light of the limitations and
share these details with the readers. Rather than present-
ing your claims about the contributions of the study and
then saying readers should interpret these with caution,
present only those claims that can be made with the limita-
tions in mind.
One way to think about the constraints you will likely need to impose on your
interpretations is in terms of generalizability. Recall that earlier in this chapter, we
described the close relationship between contributions and generalizability. When
generalizability is restricted, so are contributions.
Limitations DuetoMissing Variables
Because of the complexity of problems, questions, and hypotheses explored in edu-
cational research, researchers are unlikely to anticipate in their studies all the vari-
ables that affect the data and results. In addition, tradeoffs often must be made.
Researchers cannot study everything at once, so decisions must be made about
which variables to study carefully and which to either control or ignore.
In the earlier example of studying whether second graders improve their under-
standing of place value after a specially designed instructional intervention, the
researcher identied three variables that were expected to inuence the effect of the
intervention: students’ entry level of understanding, implementation of the interven-
tion, and norms of the classrooms in which the intervention was implemented. The
researcher decided to control the implementation variable by hiring one experi-
enced teacher to implement the intervention in all the classrooms. This meant the
variable of individual teacher differences was not included in the study and the
researcher could not generalize to classrooms with these differences.
Some researchers might see controlling the implementation of the intervention
as a limitation. We do not. As a factor that is not allowed to vary, it constrains the
generalizations a researcher can make, but we believe these kinds of controlled
variables are better treated as opportunities for future research. Perhaps the research-
er’s observations in the classroom provided information that could be used to make
some predictions about which elements of the intervention are essential and which
are optional—about which aspects of the intervention must be implemented as writ-
ten and which can vary with different teachers. When revising the rationales to show
what was learned in this study, the researcher could include rationales for new,
tentative predictions about the effects of the intervention in classrooms where
implementation differed in specied ways. These predictions create a genuine con-
tribution of the study. If you use our denitions, these new predictions, often
5 Signicance ofaStudy: Revisiting the“So What” Question

121
presented under “implications for future research,” would be presented as
“contributions.”
Notice that if you follow our advice, you would not need to include a separate
section in the Discussion of your paper labeled “Limitations.” We acknowledge,
however, that some journal editors recommend such a subsection. In this case, we
suggest you include this subsection along with treating the two different kinds of
limitations as we recommend. You can do both.
Dealing withUnexpected Findings
Researchers are often faced with unexpected and perhaps surprising results, even
when they have developed a convincing theoretical framework, posed research
questions tightly connected to this framework, presented predictions about expected
outcomes, and selected methods that appropriately test these predictions. Indeed,
the unexpected ndings can be the most interesting and valuable products of the
study. They can range from mildly surprising to “Wow. I didn’t expect that.” How
should researchers treat such ndings? Our answer is based on two principles.
The rst principle is that the value of research does not lie in whether the predic-
tions are completely accurate but in helping the eld learn more about the explana-
tory power of theoretical frameworks. That is, the value lies in the increased
understanding of phenomena generated by examining the ability of theoretical
frameworks (or rationales) to predict outcomes and explain results. The second
principle, a corollary to the rst, is to treat unexpected ndings in a way that is most
educative for the reader.
Based on our arguments to this point, you could guess we will say there will
always be unexpected ndings. Predicted answers to signicant research questions
in education will rarely, if ever, be entirely accurate. So, you can count on dealing
with unexpected ndings.
Consistent with the two principles above, your goal should be to use unexpected
ndings to understand more fully the phenomenon under investigation. We recom-
mend one of three different paths. The choice of which path to take depends on what
you decide after reecting again on the decisions you made at each phase of
the study.
The rst path is appropriate when researchers reexamine their theoretical frame-
work in light of the unexpected ndings and decide that it is still a compelling
framework based on previous work. They reason that readers are likely to have been
convinced by this framework and would likely have made similar predictions. In
this case, we believe that it is educative for researchers to (a) summarize their initial
framework, (b) present the ndings and distinguish those that were aligned with the
predictions from those that were not, and (c) explain why the theoretical framework
was inadequate and propose changes to the framework that would have created
more alignment with the unexpected ndings.
Part IV.Special Considerations that Affect aStudy’s Contributions

122
Revisions to initial hypotheses are especially useful if they include explanations
for why a researcher might have been wrong (and researchers who ask signicant
questions in domains as complex as education are almost always wrong in some
way). Depending on the ways in which the revised framework differs from the origi-
nal, the authors have two options. If the revised framework is an expansion of the
original, it would be appropriate for the authors to propose directions for future
research that would extend this study. Alternatively, if the revised framework is still
largely within the scope of the original study and consists of revisions to the original
hypotheses, the revisions could guide a second study to check the adequacy of the
revisions. This second study could be conducted by the same researchers (perhaps
before the nal manuscript is written and presented as two parts of the same report)
or it could be proposed in the Discussion as a specic study that could be conducted
by other researchers.
The second path is appropriate when researchers reexamine their theoretical
framework in light of the unexpected ndings and recognize serious aws in the
framework. The aws could result from a number of factors, including dening ele-
ments of the framework in too general a way to formulate well-grounded hypothe-
ses, failing to include a variable, or not accounting carefully enough for the previous
work in this domain, both theoretical and empirical. In many of these cases, readers
would not be well served by reading a poorly developed framework and then learn-
ing that the framework, which had not been convincing, did not accurately predict
the results. Before scrapping the study and starting over, we suggest stepping back
and reexamining the framework. Is it possible to develop a more coherent, com-
plete, and convincing framework? Would this framework predict the results more
accurately? If the ndings remain unexpected based on the predictions generated by
this revised, more compelling framework, then the rst path applies.
It is likely that the new framework will better predict the ndings. After all, the
researchers now know the ndings they will report. However, it is unlikely that the
framework will accurately predict all the ndings. This is because the framework is
not built around the ndings of this study of which authors are now aware (but have
not yet been presented). Frameworks are built on research and theory already pub-
lished. This means the redesigned framework is built from exactly the same empiri-
cal ndings and theoretical arguments available before the study was conducted.
The redesigned framework also is constrained by needing to justify exactly those
methods used in the study. The redesigned framework cannot justify different meth-
ods or even slightly altered methods. The task for researchers is to show how the
new theoretical framework necessarily generates, using the same methods, the pre-
dictions they present in the research paper. Just as before, it is unlikely this frame-
work can account for all the ndings. Just as before, after presenting the results the
researchers should explain why they believe particular hypotheses were conrmed
and why others should be revised, even in small ways, based on the ndings reported.
Researchers can now use these ndings to revise the hypotheses presented in the
paper. The point we are making is that we believe it is acceptable to reconstruct
frameworks before writing research reports if doing so would be more educative for
the reader.
5 Signicance ofaStudy: Revisiting the“So What” Question

123
Finally, the third path becomes appropriate when researchers, in reexamining
their theoretical framework, trace the problem to a misalignment between the meth-
ods they used and the theoretical framework or the research questions. Perhaps the
researchers recognize that the tasks they used did not yield data that could test the
predictions and address the research questions. Or perhaps the researchers realize
that the sample they selected would likely have been heavily inuenced by a factor
they failed to take into account. In other words, the researchers decide that the unex-
pected ndings were due to a problem with the methods they used, not with the
framework or the accompanying predictions. In this case, we recommend that the
researchers correct the methodological problems and conduct the study again.
Part V.AFew Suggestions forStructuring Your
Discussion Section
Writing the Discussion section of your research paper can be overwhelming given
all our suggestions about what to include in this section. Here are a few tips that
might help you create a simple template for this section.
We recommend the Discussion begin with a brief summary of the main results,
especially those you will interpret in this section. This summary should not contain
new data or results not previously presented in the paper.
The Discussion could then move to presenting the contributions in the ways we
have described. To do this you could point out the ways in which the results differed
from the predictions and suggest revisions to your rationales that would have better
predicted the results. Doing this will show how the contributions of your study
extend what is known beyond the research you drew on to build your original ratio-
nale. You can then propose how to extend your contributions to research by propos-
ing future research studies that would test your new predictions. If you believe the
revisions you make to your rationales produce new insights or understandings that
could be helpful for educational practitioners, you can identify these contributions
to practice as well. This comprises the bulk of the Discussion section.
If you have embedded the limitations in earlier sections of the paper, you will
have presented your results and interpreted your ndings constrained by these limi-
tations. If you choose (or are asked) to describe limitations in the Discussion, you
could identify the limitations and then point to the ways they affected your interpre-
tations of the ndings. Finally, the Discussion could conclude with the implications
of the study for methodological choices that could improve research in the domain
in which your study is located or how future studies could overcome the limitations
you identied.
Because we are providing guidance on writing your research paper for publica-
tion, we will reiterate here that you should investigate the expectations and conven-
tions of the journal to which you will submit your paper. Usually, it will be acceptable
to use the terms “signicance,” “contributions,” and “implications” as we have
Part V.AFew Suggestions forStructuring Your Discussion Section

124
dened them. However, if the editors expect you to use the terms differently, follow
the editors’ expectations. Our denitions in this chapter are meant to help you think
clearly about the different ways you can make a case for the importance of your
research. What matters is that you have carefully built and described a coherent
chain of scientic inquiry that allows your study to translate the signicance of your
research problem into contributions to the eld.
We began the chapter with the “So what?” question. The question looks simple
and straightforward but is challenging and complicated. Its simple appearance can
lead researchers to believe it should have a simple answer. But it almost never does.
In this chapter, we tried to address the many complications that arise when answer-
ing the question. We hope you now have some new insights and new tools for
answering the question in your next study.
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing,
adaptation, distribution and reproduction in any medium or format, as long as you give appropriate
credit to the original author(s) and the source, provide a link to the Creative Commons license and
indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative
Commons license, unless indicated otherwise in a credit line to the material. If material is not
included in the chapter's Creative Commons license and your intended use is not permitted by
statutory regulation or exceeds the permitted use, you will need to obtain permission directly from
the copyright holder.
5 Signicance ofaStudy: Revisiting the“So What” Question