No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Performant Peer Review for Design Science Manuscripts: A Pilot Study on Dedicated Highlighters
Abstract
Peer review is under pressure. Demand for reviews is outstripping supply where reviewers tend to be busy people who contribute voluntarily. Authors highly value reviews, yet complain about the time it takes to get feedback to the point of putting research timeliness at stake. Though part of the review process has been moved to the Web, the review itself is still often conducted with the only help of a yellow highlighter, physical or digital. This work looks for more performant highlighters that account for the review specifics. Peer review does not stop at spotting the manuscript (de)merits, it also strives for manuscript improvement and gatekeeping. These functions are conducted within an often tacit research-quality framework, and frequently in a discontinuous way. Unfortunately, when it comes to support review practices, current facilities fall short. This work introduces a set of requirements for review-dedicated highlighters. These requirements are instantiated and evaluated through Review&Go, a color-coding highlighter that generates a review draft out of the reviewer’s highlighting activities. The aim is to offer representational guidance to enhance context/cognitive awareness so that reviewers can exert less effort while offering valuable and timely reviews.
ResearchGate has not been able to resolve any citations for this publication.
Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.
Reading literature is important, but problematic. In Quora and other PhD forums, students moan about their frustrating reading and literature review experiences. Strategic reading might help. This term is coined to conceive of reading as a process of constructing meaning by interacting with text in a targeted way. The fact that strategic reading is purpose-driven suggests that the purpose might qualify the reading. If this purpose is Design Science Research (DSR), what would be the strategy for reading? Traditionally, students are encouraged to annotate while reading. Digital annotations are expected to be useful for supporting comprehension and interpretation. Our belief is that strategic reading can be more effective if annotation is conducted in direct relationship to a main DS activity: root-cause analysis (RCA). RCA can provide the questions whose answers should be sought in the literature. Unfortunately, this process is not supported by current tools. When reading papers, researchers might not be all aware of the issues being raised during RCA. And the other way around, when it comes to RCA, evidences found in the literature might not be promptly accessible. This paper reports on research to develop a technical solution to this problem: a plug-in for Google Chrome that provides seamless integration between the RCA platform (i.e. MindMeister) and the reading platforms (i.e. Mendeley). The aim: improving RCA awareness while reading so that annotations can be traced back to the RCA issues. First evaluations are positive as for improving reading focus and facilitating reference recoverability.
Design science research (DSR) has staked its rightful ground as an important and legitimate Information Systems (IS) research paradigm. We contend that DSR has yet to attain its full potential impact on the development and use of information systems due to gaps in the understanding and application of DSR concepts and methods. This essay aims to help researchers (1) appreciate the levels of artifact abstractions that may be DSR contributions, (2) identify appropriate ways of consuming and producing knowledge when they are preparing journal articles or other scholarly works, (3) understand and position the knowledge contributions of their research projects, and (4) structure a DSR article so that it emphasizes significant contributions to the knowledge base. Our focal contribution is the DSR knowledge contribution framework with two dimensions based on the existing state of knowledge in both the problem and solution domains for the research opportunity under study. In addition, we propose a DSR communication schema with similarities to more conventional publication patterns, but which substitutes the description of the DSR artifact in place of a traditional results section. We evaluate the DSR contribution framework and the DSR communication schema via examinations of DSR exemplar publications.
A review and meta-analysis was performed of seventy-five articles concerned with innovation characteristics and their relationship to innovation adoption and implementation. One part of the analysis consisted of constructing a methodological profile of the existing studies, and contrasting this with a hypothetical optimal approach. A second part of the study employed meta-analytic statistical techniques to assess the generality and consistency of existing empirical findings. Three innovation characteristics (compatibility, relative advantage, and complexity) had the most consistent significant relationship to innovation adoption. Suggestions for future research in the area were made.
Student surveys across the world have highlighted that students are dissatisfied with the feedback they receive on their assignments and many institutions have been putting plans in place to address this issue. Much of this work has focused on improving the quality of written comments. This paper takes a different perspective. It argues that the many diverse expressions of dissatisfaction with written feedback, both from students and teachers, are all symptoms of impoverished dialogue. Mass higher education is squeezing out dialogue with the result that written feedback, which is essentially a one‐way communication, often has to carry almost all the burden of teacher–student interaction. The paper suggests ways in which the nature and quality of feedback dialogue can be enhanced when student numbers are large without necessarily increasing demands on academic staff. It concludes with a conceptual discussion of the merits of taking a dialogical approach when designing feedback.
The importance of peer review in the furthering of science cannot be overstated. However, most doctoral students and early career professionals receive little formal or informal training in conducting peer reviews.
In recognition of this deficit in peer reviewer training, the present article was developed to provide an overview of the peer-review process at Annals of Behavioral Medicine and describe the general and specific elements that should be included in a high-quality review for the journal.
We conclude by offering exemplar reviews of a manuscript that was ultimately accepted for publication in the journal and provide commentary on specific aspects of these reviews.
AGU journals will incorporate open source software to facilitate dialog among reviewers, editors and authors during peer review.
It has been more than ten years since the publication of Hevner et al [1] and five years since Venable [2] surveyed editors and DSR researchers on standards and criteria for judging the quality and suitability of DSR submissions for publication. Since then, there has been much further discussion about evaluation, design theory, and standards for DSR publication. This paper attempts to answer the question of how standards for judging the quality (e.g., rigour and relevance) of DSR research publications have changed since 2010 and to develop a snapshot of the relative importance of different extant DSR publication criteria. To do so, the author surveyed editors of IS Scholars’ “basket-of-eight” journals, DESRIST conference program committee members, and DESRIST (co-)authors. This paper compares the quantitative findings of the current survey to the 2010 survey.
Contents
Executive summary ● Scholarly communication ● The research cycle ● Types of scholarly communication ● Changes in scholarly communication system ● The journal ● What is a journal? ● The journals publishing cycle ● Sales channels and models ● Journal economics and market size ● Journal and articles numbers and trends ● Global trends in scientific output ● Authors and readers ● Publishers ● Peer review. ● Reading patterns ● Disciplinary differences ● Citations and the Impact Factor ● Costs of journal publishing ● Authors’ behaviour, perceptions and attitudes ● Publishing ethics ● Copyright and licensing ● Long term preservation ● TRANSFER code ● Researchers’ access to journals ● Open access ● Drivers of open access ● Open access business models ● Types of open access journal ● Delayed open access ● Open access via self-archiving ("Green" OA) ● Other open access variants ● SCOAP3 ● Open access to scholarly books ● Public access ● System-wide and economic perspectives ● Other developments in open access ● Transition and sustainability issues ● Effect of self-archiving on journals. ● Open access impacts on use ● New developments in scholarly communication ● “Science 2.0” or "Open Science" ● FORCE11 and “Science in Transition” ● Publishing platforms and APIs ● Social media ● Mobile access and apps ● Research data ● Semantic web and semantic enrichment ● New article formats and features. ● Text and data mining ● Reproducibility ● Big data & analytics ● Identity and disambiguation ● Research management and analytics ● FundRef ● Library publishing ● Open Annotation ● Learned societies ● Author services and tools ● Collaborative writing and sharing tools ● Open notebook science ● Conclusions ● Information sources ● Publisher organisations ● Global statistics and trends ● Open access ● Publishing industry research and analysis ● References
180pp
Pre-publication peer review has long been recognised as a cornerstone of scholarly publishing. Despite various criticisms and a number of shortcomings, this scrutiny and critical assessment by experts is still considered essential by many. This chapter describes the realistic expectations of peer review and what constitutes good practice, emphasising the important role of the editor. It also outlines the many ways traditional peer review is being adapted, the new models that are appearing, and the increasing emphasis on openness and transparency. Various problems are addressed, including difficulties in finding reviewers, the imbalance between publication output and participation in peer review worldwide, the 'wastage' of reviews that accompanies the often repeated submission of manuscripts to journal after journal after rejection, and the increasing pressure on researchers not only to publish but to publish in high-impact journals. All these are impacting the peer-review processes of journals and editorial workload. The recent innovation to concentrate only on the assessment of soundness of research methodology and reporting pre-publication is receiving acceptance from researchers keen to publish their work without undue delay, and is being adopted by an increasing number of publishers. In this model, the evaluation of interest, importance and potential impact is left for after publication. The possibilities for post-publication review and evaluation in an online world are many and varied, but there are also many challenges. It is clear that, nearly three and a half centuries on from the appearance of the first journals, the opportunities for innovation and experimentation in peer review are greater than ever before.
This book is an introductory text on design science, intended to support both graduate students and researchers in structuring, undertaking and presenting design science work. It builds on established design science methods as well as recent work on presenting design science studies and ethical principles for design science, and also offers novel instruments for visualizing the results, both in the form of process diagrams and through a canvas format. While the book does not presume any prior knowledge of design science, it provides readers with a thorough understanding of the subject and enables them to delve into much deeper detail, thanks to extensive sections on further reading. Design science in information systems and technology aims to create novel artifacts in the form of models, methods, and systems that support people in developing, using and maintaining IT solutions. This work focuses on design science as applied to information systems and technology, but it also includes examples from, and perspectives of, other fields of human practice. Chapter 1 provides an overview of design science and outlines its ties with empirical research. Chapter 2 discusses the various types and forms of knowledge that can be used and produced by design science research, while Chapter 3 presents a brief overview of common empirical research strategies and methods. Chapter 4 introduces a methodological framework for supporting researchers in doing design science research as well as in presenting their results. This framework includes five core activities, which are described in detail in Chapters 5 to 9. Chapter 10 discusses how to communicate design science results, while Chapter 11 compares the proposed methodological framework with methods for systems development and shows how they can be combined. Chapter 12 discusses how design science relates to research paradigms, in particular to positivism and interpretivism. Lastly, Chapter 13 discusses ethical issues and principles for design science research. © Springer International Publishing Switzerland 2014. All rights are reserved.
How do some students extract complex information from advanced reading materials while others struggle to find the meaning of even simple passages? What characteristics do strategic readers exhibit that others don't possess? Unlocking the cognitive strategies of exceptional readers means finding ways to teach every student to read strategically, and help them be successful in the classroom and beyond. This essential reading instruction teaching tool offers hard evidence to show how effective readers use specific strategies to extract and comprehend information.This book melds scientific research with hands-on experience to offer a comprehensive look at reading instruction. Instructors will discover that they can use the seven strategies no matter what their teaching style, and students will discover how to read to learn. McEwan has developed easily reproducible tools geared toward helping teachers get the most out of their readers: (1) Teachers Think-Alouds to facilitate modeling the seven strategies for your students; (2) Templates to speed planning time; (3) Activities to encourage use of the seven strategies; and (4) Forms, posters, props, and prompts to assist both students and teachers. While theories come and go, research has shown that using these seven strategies will prove successful in the classroom-and beyond. This book contains the following six chapters: (1) The Power of Strategic Reading Instruction; (2) Becoming a Strategic Teacher; (3) Understanding the Seven Strategies; (4) Instructional Activities to Engage Your Students; (5) Organizing for Strategic Reading Instruction (SRI) in Your Classroom and (6) Implementing Schoolwide Strategic Reading Instruction. A Conclusion and an Index are included.
What are the challenges and opportunities facing peer review in a networked world? We review the state of journal peer review at the end of 2010, beset by ever-increasing volumes of journal articles demanding to be reviewed in ever-shorter times and by a range of criticisms of its effectiveness and value. We consider the opportunities for improvement and review alternative approaches including open and post-publication review and the possible use of article-level metrics as a replacement. We conclude that far from being in crisis, peer review has remains widely supported and diversely innovative.
We report on a research study commnissioned by ALPSP into the current status Of online submission and peer-review systems, the perceptions of these by authors, referees and editors, and the impact of their introduction on journals.
‘If peer review was a drug it would never be allowed onto the market,’ says Drummond Rennie, deputy editor of the Journal Of the American Medical Association and intellec tual father of the international congresses of peer review that have been held every four years since 1989. Peer review would not get onto the market because we have no convincing evidence of its benefi ts but a lot of evidence of its fl aws. Yet, to my continuing surprise, almost no scientists know anything about the evidence on peer review. It is a process that is central to science - deciding which grant proposals will be funded, which papers will be published, who will be promoted, and who will receive a Nobel prize. We might thus expect that scientists, people who are trained to believe nothing until presented with evidence, would want to know all the evidence available on this important process. Yet not only do scientists know little about the evidence on peer review but most continue to believe in peer review, thinking it essential for the progress of science. Ironically, a faith based rather than an evidence based process lies at the heart of science.
![Figure][1]
The primary function of a medical journal is to provide important new research information to its readers. Since the new data may alter clinical care or research protocols, the goal to convey the information in as timely a manner as possible is implicit. Readers want to have new
Reducing the peer-reviewer’s burden
- M Clarke
Clarke, M.: Reducing the peer-reviewer's burden (2010), http://blogs.nature.com/peer-topeer/2010/05/reducing_the_peerreviewers_bur_1.html (accessed 22 Mar. 2019).
Good designs for Written Feedback to Students. McKeachie’s Teaching Tips: Strategies, Research, and Theory for College and University Teachers
- D Nicol
Nicol, D.: Good designs for written feedback to students. McKeachie's Teaching Tips:
Strategies, Research, and Theory for College and University Teachers. pp. 108-124. Cengage Learning (2010).
- VK Vaishnavi
Familiarity: A UX consideration
- M Lawrence
Scientific Peer Reviewing: Practical Hints and Best Practices
- P Spyns
- M E Vidal
Spyns, P., Vidal, M.E.: Scientific Peer Reviewing: Practical Hints and Best Practices.
Springer (2015).
Five and ten years on: have DSR standards changed?
- John R Venable
- B Donnellan
- M Helfert
- J Kenneally
- D VanderMeer
- M Rothenberger
- R Winter