Figure 3 - uploaded by Bastian Jaeger
Content may be subject to copyright.
The Role of Routine Researcher Variability in Replication Results.

The Role of Routine Researcher Variability in Replication Results.

Source publication
Preprint
Full-text available
The paper reports findings from a crowdsourced replication. Eighty-four replicator teams attempted to verify results reported in an original study by running the same models with the same data. The replication involved an experimental condition. A “transparent” group received the original study and code, and an “opaque” group received the same unde...

Contexts in source publication

Context 1
... had to remove 4 teams from the calculation of the curated group correlations because these teams had major mistakes but no possible counterfactual, thus they had un-curated non-routine variability in addition to routine variability and were unfit for comparison. Figure 3 decomposes the role of variability by experimental group. ...
Context 2
... in more complex or decision-rich research forms the researcher degrees of freedom grow exponentially and the reliability rapidly decreases. This is evidenced by routine researcher variability causing more deviance among the opaque group (see Figure 3). The opaque group is much closer to 'just doing research' than doing a verification as they faced many small decisions to make without simply looking at the original code. ...

Citations

Book
Full-text available
Diese Publikation ist eine Einführung in Open Access und das wissenschaftliche Publizieren zur Unterstützung der Wissensvermittlung durch Multiplikator:innen im Sinne des Formats Train-the-Trainer. Entsprechend richtet sich dieses Train-the-Trainer-Konzept insbesondere an Personen, die im Rahmen ihrer Tätigkeiten andere Personen hinsichtlich Informations- und Beratungskompetenzen im Bereich Open Access und dem wissenschaftlichen Publizieren aus- und weiterbilden und dazu entsprechende Schulungsangebote erstellen wollen. Das Train-the-Trainer-Konzept umfasst dazu neben den fachlichen Inhalten auch Informationen zu didaktischen Grundlagen und Methoden sowie zur Organisation von Veranstaltungen. Der modulare Aufbau der Inhalte, fertige Ablaufpläne (Lehrdrehbücher) und Übungsvorschläge zu jeder Lehreinheit unterstützen eine flexible, zielgruppengerechte Konzeption von Veranstaltungen im Präsenz- sowie Online-Format.
Article
Full-text available
Purpose Strongly framed research questions are clear as to the population (P), the exposures or interventions (E/I), comparison groups (C), outcomes (O), time when relevant (T), and what the investigator wants to know. A solid framework sets up the measurement model, analysis, and anticipated results. The purpose of this study was to estimate the extent to which research questions in journals that focused on patient-reported outcome measures (PROM) and quality of life (QOL) are clear. Methods All 440 research articles published in four PROM journals in 2020. excluding reviews, psychometric, and qualitative papers, were reviewed. Research questions were classified as: (i) adequately framed (ii) poorly framed; or (iii) unframed based on clarity criteria. Examples from each journal were presented and reframed to match results in the article. Results Of 440 articles, 195 (44.3%) were classified as adequately framed; 230 (52.2%) as poorly framed; and 15 (3.4%) as unframed. There was heterogeneity across journals (Chi-square: 20.8; 6 df; p = 0.002). Only 29% were framed according to what the investigators wanted to know; 72% were framed like a “to do” list; and 6% were framed as a research agenda. Conclusion Almost half of the questions were poorly framed or unframed a practice that could contribute to research wastage. Even “adequately framed” questions rarely stated what they wanted to know a priori, increasing the risk of biased reporting. Researchers, reviewers, and editors should encourage the use established frameworks for research questions.