No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Communicating the Significance of Research Questions: Insights from Peer Review at a Flagship Journal
Abstract
Significant research in science and mathematics education should advance the field’s knowledge and understanding of the teaching and learning of science and mathematics. How, then, should the significance of a research question in science and mathematics education be assessed? And, when disseminating the findings of research, how should the significance of the research questions be communicated? In this study, we analyzed peer reviews to answer these questions. Our analysis revealed the main issues peer reviewers identify about research questions and the ways they are communicated during the dissemination of research. The findings provide insights for new and experienced researchers about communicating the significance of research questions, and they also illustrate how reviewer comments in peer-reviewed journals can provide a window into the field’s frontiers.
... It is clear that not all researchable questions are worth the effort to investigate. According to Cai et al. (2020), of all researchable questions in mathematics and science education, priority is given to those that are significant. Research questions are significant if they can advance the fields' knowledge and understanding about the teaching and learning of science and mathematics. ...
... Research questions are significant if they can advance the fields' knowledge and understanding about the teaching and learning of science and mathematics. Through an analysis of peer reviews for a research journal, Cai et al. (2020) provide a window into the field's frontiers related to significant researchable questions. In an earlier article, Cai et al. (2019a) argued that ...
... (p. 118) In their analysis, Cai et al. (2020) provide evidence that many reviews that highlighted issues with the research questions in rejected manuscripts specifically called for authors to make an argument to motivate the research questions, whereas none of the manuscripts that were ultimately accepted (pending revisions) received this kind of comment. Cai et al. (2020) provide a framework not only for analyzing peer reviews about research questions but also for how to communicate researchable questions in journal manuscript preparations. ...
The original version of this article unfortunately contains correction.
... It is clear that not all researchable questions are worth the effort to investigate. According to Cai et al. (2020), of all researchable questions in mathematics and science education, priority is given to those that are significant. Research questions are significant if they can advance the fields' knowledge and understanding about the teaching and learning of science and mathematics. ...
... Research questions are significant if they can advance the fields' knowledge and understanding about the teaching and learning of science and mathematics. Through an analysis of peer reviews for a research journal, Cai et al. (2020) provide a window into the field's frontiers related to significant researchable questions. In an earlier article, Cai et al. (2019a) argued that ...
... (p. 118) In their analysis, Cai et al. (2020) provide evidence that many reviews that highlighted issues with the research questions in rejected manuscripts specifically called for authors to make an argument to motivate the research questions, whereas none of the manuscripts that were ultimately accepted (pending revisions) received this kind of comment. Cai et al. (2020) provide a framework not only for analyzing peer reviews about research questions but also for how to communicate researchable questions in journal manuscript preparations. ...
... Recognizing the common concerns around the rigor of CS education research (Al-Zubidy et al., 2016;Heckman et al., 2022), which Lishinski et al. (2016) claim is caused by CS research being undertaken by teachers that lack formal training in research methods and theory, the methodological credibility of each article was also assessed. Relating to research question three, this evaluation ensured that the methods that each researcher claimed to be using were justified and applied appropriately (Nkwake, 2015) and highlighted areas where researchers can raise the thoroughness, trustworthiness, replicability and validity of research within this field (Cai et al., 2020;Dodgson, 2020;Lishinski et al., 2016). Any methodological issues and discrepancies in the articles are highlighted within the results section. ...
For over a decade, we have seen a global shift toward preparing students for the predicted labor market and society through the introduction of digital technology curricula. This scoping review uncovered 26 studies published between 2012 and 2022 that examined empirical research on K–12 or pre-service teachers’ experiences with Digital Technologies curricula. Most of the research involved primary school teachers (n = 20), focused on teachers’ experiences with Professional Learning and Development (n = 13), and adopted a mixed methods methodological approach (62%) utilizing a questionnaire data collection instrument (n = 18). Methodological trends and areas for improvement were uncovered and analyzed to aid future research that can be used to support teachers’ implementation of Digital Technologies curricula.
Research journals play significant roles in the advancement of academic fields of inquiry. This chapter starts with a brief description of the Journal for Research in Mathematics Education. Most importantly, this chapter provides practical guides to promoting and disseminating significant research in mathematics education. The guides provided in this chapter will be helpful and insightful for those who are interested in publishing in the Journal for Research in Mathematics Education.KeywordsJournal for Research in Mathematics EducationMathematics educationSignificant researchResearch dissemination
Peer review has been defined as a process of subjecting an author's scholarly work, research or ideas to the scrutiny of others who are experts in the same field. It functions to encourage authors to meet the accepted high standards of their discipline and to control the dissemination of research data to ensure that unwarranted claims, unacceptable interpretations or personal views are not published without prior expert review. Despite its wide-spread use by most journals, the peer review process has also been widely criticised due to the slowness of the process to publish new findings and due to perceived bias by the editors and/or reviewers. Within the scientific community, peer review has become an essential component of the academic writing process. It helps ensure that papers published in scientific journals answer meaningful research questions and draw accurate conclusions based on professionally executed experimentation. Submission of low quality manuscripts has become increasingly prevalent, and peer review acts as a filter to prevent this work from reaching the scientific community. The major advantage of a peer review process is that peer-reviewed articles provide a trusted form of scientific communication. Since scientific knowledge is cumulative and builds on itself, this trust is particularly important. Despite the positive impacts of peer review, critics argue that the peer review process stifles innovation in experimentation, and acts as a poor screen against plagiarism. Despite its downfalls, there has not yet been a foolproof system developed to take the place of peer review, however, researchers have been looking into electronic means of improving the peer review process. Unfortunately, the recent explosion in online only/electronic journals has led to mass publication of a large number of scientific articles with little or no peer review. This poses significant risk to advances in scientific knowledge and its future potential. The current article summarizes the peer review process, highlights the pros and cons associated with different types of peer review, and describes new methods for improving peer review.
Recently we repaired a very old, wooden rowboat, patching leaks in the hull, replacing the seat, applying a fresh coat of paint, and wondering all the while how many times these tasks had been repeated over the decades that the boat has been in use. Our work also reminded us of the ancient Greek story of the ship of Theseus — a story of personal identity that has challenged philosophers throughout the ages (Nozick 1981).
The authors argue for a reconceptualization of rigor that requires sustained, direct, and systematic documentation of what takes place inside programs to document how students and teachers change and adapt interventions in interactions with each other in relation to their dynamic local contexts. Building on promising new programs at the Institute of Education Sciences, they call for the formulation of collaborative research standards that must require researchers to provide evidence that they have engaged in a process to surface and negotiate the focus of their joint work, and to document the ways participation in this process was structured to include district and school leaders, teachers, parents, community stakeholders, and, wherever possible, children and youth. They close by describing how this new criterion—“relevance to practice”—can ensure the longevity and efficacy of educational research.
In our report for DG-Research of the European Commission entitled “Study on the economic and technical evolution of the scientific publication markets in Europe”, published almost two years ago, we showed that, on average, journals published ...
A National Research Council report, Scientific Research in Education,
has elicited considerable criticism from the education research community,
but this criticism has not focused on a key assumption of
the report—its Humean, regularity conception of causality. It is argued
that this conception, which also underlies other arguments for
“scientifically-based research,” is narrow and philosophically outdated,
and leads to a misrepresentation of the nature and value of
qualitative research for causal explanation. An alternative, realist approach
to causality is presented that supports the scientific legitimacy
of using qualitative research for causal investigation, reframes
the arguments for experimental methods in educational research,
and can support a more productive collaboration between qualitative
and quantitative researchers.
This paper presents the first meta-analysis for the inter-rater reliability (IRR) of journal peer reviews. IRR is defined as the extent to which two or more independent reviews of the same scientific document agree.
Altogether, 70 reliability coefficients (Cohen's Kappa, intra-class correlation [ICC], and Pearson product-moment correlation [r]) from 48 studies were taken into account in the meta-analysis. The studies were based on a total of 19,443 manuscripts; on average, each study had a sample size of 311 manuscripts (minimum: 28, maximum: 1983). The results of the meta-analysis confirmed the findings of the narrative literature reviews published to date: The level of IRR (mean ICC/r2=.34, mean Cohen's Kappa=.17) was low. To explain the study-to-study variation of the IRR coefficients, meta-regression analyses were calculated using seven covariates. Two covariates that emerged in the meta-regression analyses as statistically significant to gain an approximate homogeneity of the intra-class correlations indicated that, firstly, the more manuscripts that a study is based on, the smaller the reported IRR coefficients are. Secondly, if the information of the rating system for reviewers was reported in a study, then this was associated with a smaller IRR coefficient than if the information was not conveyed.
Studies that report a high level of IRR are to be considered less credible than those with a low level of IRR. According to our meta-analysis the IRR of peer assessments is quite limited and needs improvement (e.g., reader system).
Although often asked tactfully, a frequent question posed to authors by JRME reviewers is “So what?” Through this simple and well-known question, reviewers are asking: What difference do your findings make? How do your results advance the field? “So what?” is the most basic of questions, often perceived by novice researchers as the most difficult question to answer. Indeed, addressing the “so what” question continues to challenge even experienced researchers. All researchers wrestle with articulating a convincing argument about the importance of their own work. When we try to shape this argument, it can be easy to fall into the trap of making claims about the implications of our findings that reach beyond the data.
In our recent editorials (Cai et al., 2019a, 2019b), we discussed the important roles that research questions and theoretical frameworks play in conceptualizing, carrying out, and reporting mathematics education research. In this editorial, we discuss the methodological choices that arise when one has articulated research questions and constructed at least a rudimentary theoretical framework. Just as the researcher must justify the significance of research questions and the appropriateness of the theoretical framework, we argue that the researcher must thoroughly describe and justify the selection of methods. Indeed, the research questions and the theoretical framework should drive the choice of methods (and not the reverse). In other words, a sufficiently well-specified set of research questions and theoretical framework establish the parameters within which the most productive methods will be selected and developed.
Taking as its point of departure the discussion about the disconnection between research and practice, this article presents learning study as a research approach to overcoming this gap. Learning study has commonalities with design research and lesson study, but is a teacher‐researcher collaboration where both have a common object of research. Thus, it is research with teachers, rather than on teachers and focuses on constructing knowledge concerning objects of learning as well as teaching‐learning relationships. The focus of the research collaboration is professional problems related to the object of learning that teachers encounter in their everyday practice. The process is guided by a theory of learning and pedagogy—the variation theory. The knowledge product of learning study is a theoretical description of what must be learned in order to develop a specific capability. Examples of knowledge contributions from learning study are given and it is suggested that such knowledge can be considered to be public knowledge that can be shared, used and developed by other teachers in other contexts. Furthermore, it is suggested that there are specific features of learning study that make it a research approach that may strengthen connections between research and practice.
In our March editorial (Cai et al., 2019), we discussed the nature of significant research questions in mathematics education. We asserted that the choice of a suitable theoretical framework is critical to establishing the significance of a research question. In this editorial, we continue our series on high-quality research in mathematics education by elaborating on how a well-constructed theoretical framework strengthens a research study and the reporting of research for publication. In particular, we describe how the theoretical framework provides a connecting thread that ties together all of the parts of a research report into a coherent whole. Specifically, the theoretical framework should help (a) make the case for the purpose of a study and shape the literature review; (b) justify the study design and methods; and (c) focus and guide the reporting, interpretation, and discussion of results and their implications.
In 2002, the National Research Council (NRC) released Scientific Research in Education , a report that proposed six principles to serve as guidelines for all scientific inquiry in education. The first of these principles was to “pose significant questions that can be investigated empirically” (p. 3). The report argued that the significance of a question could be established on a foundation of existing theoretical, methodological, and empirical work. However, it is not always clear what counts as a significant question in educational research or where such questions come from. Moreover, our analysis of the reviews for manuscripts submitted to JRME ¹ suggests that some practical, specific guidance could help researchers develop a significant question or make the case for the significance of a research question when preparing reports of research for publication.
The authors discussed 2 paths that the mathematics education community should consider with regard to citation-based metrics of journal quality: either working within the system to enhance positioning or resisting or modifying the system itself.
We present the results of 2 studies, a citation-based study and an opinion-based study, that ranked the relative quality of 20 English-language journals that exclusively or extensively publish mathematics education research. We further disaggregate the opinion-based data to provide insights into variations in judgment of journal quality based on geographic location, journal affiliations and publishing records, and experience in the field. We also report factors that survey respondents indicated were important indicators of journal quality. Finally, we compare our results to previous related rankings and conclude by discussing how our results might inform authors, editors, and evaluators in their efforts to publish and recognize quality research in mathematics education.
In this article, Graham Nuthall critiques four major types of research on teaching effectiveness: studies of best teachers, correlational and experimental studies of teaching-learning relationships, design studies, and teacher action and narrative research. He gathers evidence about the kind of research that is most likely to bridge the teaching-research gap, arguing that such research must provide continuous, detailed data on the experience of individual students, in-depth analyses of the changes that take place in the students' knowledge, beliefs, and skills, and ways of identifying the real-time interactive relationships between these two different kinds of data. Based on his exploration of the literature and his research on teaching effectiveness, Nuthall proposes an explanatory theory for research on teaching that can be directly and transparently linked to classroom realities.
Quality control is an important and integrated part of the scientific system. However, developments in science and society are changing quality control into quality monitoring. New, virtual, and fluid organizational forms are emerging. Common boundaries are seen as being broken down as, for example, in the "triple helix" and the "mode 2" concepts. The stakeholders in science are showing an interest in being more involved in science. They want their evaluation criteria to be used, and they want evaluations to be done on a regular basis because they do not trust the new scientific institutions to be left on their own. Quality monitoring changes the requirements for conducting evaluations as part of quality control. Assessing the societal value of research becomes increasingly important. Finally, quality monitoring emphasizes organizational learning rather than controlling quality in scientific organizations.
The credibility of the publication system in science is determined in large part by the precision of the manuscript review process. Studies on the precision of the review process in scientific journals have reported conflicting results. This paper reviews those studies and re-examines the data reported. The findings indicate that highly selective decision-making with imprecise reviewers results in outcomes that are only slightly better than chance.
The reliability of peer review of scientific documents and the evaluative criteria scientists use to judge the work of their peers are critically reexamined with special attention to the consistently low levels of reliability that have been reported. Referees of grant proposals agree much more about what is unworthy of support than about what does have scientific value. In the case of manuscript submissions this seems to depend on whether a discipline (or subfield) is general and diffuse (e.g., cross-disciplinary physics, general fields of medicine, cultural anthropology, social psychology) or specific and focused (e.g., nuclear physics, medical specialty areas, physical anthropology, and behavioral neuroscience). In the former there is also much more agreement on rejection than acceptance, but in the latter both the wide differential in manuscript rejection rates and the high correlation between referee recommendations and editorial decisions suggests that reviewers and editors agree more on acceptance than on rejection. Several suggestions are made for improving the reliability and quality of peer review. Further research is needed, especially in the physical sciences.
Meaningful vs. mechanical learning: A study in Grade III subtraction
- W A Brownell
- H E Moser
Research: To inform, deform, or reform?
- J Confrey
Mathematics educational neuroscience: Promises and challenges
- A Norton
- M A Bell
The application of a model for investigating classroom processes
- W W Cooley
- G Leinhardt
Publishing in academic journals in education
- A Holbrook
- S Bourke
- H Fairbairn
- G Preston
- R Cantwell
- J Scevak
Scientific research in education
Relevance to practice as a criterion for rigor. Educational Researcher
- K D Gutiérrez
- W R Penuel