Fiona Fidler

• PhD
• Professor (Associate) at University of Melbourne

Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studi...

Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce rep...

Aim Previous research has shown patients and the public in Australia generally support medical researchers in making de‐identified research data available to other scientists. However, this research has focussed on certain types of data and recipients. We surveyed Australians affected by cancer to characterize their attitudes toward the sharing of...

Many journals in ecology and evolutionary biology encourage or require authors to make their data and code available alongside articles. In this study we investigated how often this data and code could be used together, when both were available, to computationally reproduce results published in articles. We surveyed the data and code sharing practi...

Background: Despite wide recognition of the benefits of sharing research data, public availability rates have not increased substantially in oncology or medicine more broadly over the last decade. Methods: We surveyed 285 cancer researchers to determine their prior experience with sharing data and views on known drivers and inhibitors. Results:...

Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studi...

Objectives To synthesise research investigating data and code sharing in medicine and health to establish an accurate representation of the prevalence of sharing, how this frequency has changed over time, and what factors influence availability. Design Systematic review with meta-analysis of individual participant data. Data sources Ovid Medline,...

The rates at which journal articles in ecology and evolutionary biology make data and code available have been studied previously. This study examines how often this data and code, when available, can be used to computationally reproduce results published in articles. This study surveys the data and code sharing practices of 177 meta-analyses publi...

This paper explores judgements about the replicability of social and behavioural sciences research and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited from groups using a structured approach called the IDEA protocol (‘investigate’, ‘discuss’, ‘estimate’ and ‘aggregate’). Five grou...

Objectives: To characterise the attitudes of Australians affected by cancer towards the sharing of de-identified research data with third parties, including the public. Design, setting, participants: Anonymous online survey between October 2022 and February 2023 of adult Australians previously diagnosed with cancer. Main outcome measures: Self-repo...

Objectives Many meta-research studies have investigated rates and predictors of data and code sharing in medicine. However, most of these studies have been narrow in scope and modest in size. We aimed to synthesise the findings of this body of research to provide an accurate picture of how common data and code sharing is, how this frequency has cha...

Replication is an important “credibility control” mechanism for clarifying the reliability of published findings. However, replication is costly, and it is infeasible to replicate everything. Accurate, fast, lower cost alternatives such as eliciting predictions from experts or novices could accelerate credibility assessment and improve allocation o...

As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce the repliCATS (Collaborative Assessments for Trustworthy Science) process, a new method for eliciting expert predictions about the replicability of research. This process is a structured expert elicitation approach ba...

Journal peer review regulates the flow of ideas through an academic discipline and thus has the power to shape what a research community knows, actively investigates, and recommends to policymakers and the wider public. We might assume that editors can identify the ‘best’ experts and rely on them for peer review. But decades of research on both exp...

Background Various stakeholders are calling for increased availability of data and code from cancer research. However, it is unclear how commonly these products are shared, and what factors are associated with sharing. Our objective was to evaluate how frequently oncology researchers make data and code available and explore factors associated with...

Conservation science practitioners seek to preempt irreversible impacts on species, ecosystems, and social-ecological systems, requiring efficient and timely action even when data and understanding are unavailable, incomplete, dated, or biased. These challenges are exacerbated by the scientific community's capacity to consistently distinguish betwe...

Background Various stakeholders are calling for increased availability of data and code from cancer research. However, it is unclear how commonly these products are shared, and what factors are associated with sharing. Our objective was to evaluate how frequently oncology researchers make data and code available, and explore factors associated with...

Replication—an important, uncommon, and misunderstood practice—is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledg...

Unreliable research programmes waste funds, time, and even the lives of the organisms we seek to help and understand. Reducing this waste and increasing the value of scientific evidence require changing the actions of both individual researchers and the institutions they depend on for employment and promotion. While ecologists and evolutionary biol...

Numerous studies have demonstrated low but increasing rates of data and code sharing within medical and health research disciplines. However, it remains unclear how commonly data and code are shared across all fields of medical and health research, as well as whether sharing rates are positively associated with implementation of progressive policie...

Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to agg...

Solutions to the crisis in confidence in the psychological literature have been proposed in many recent articles, including increased publication of replication studies, a solution that requires engagement by the psychology research community. We surveyed Australian and Italian academic research psychologists about the meaning and role of replicati...

Numerous studies have demonstrated low but increasing rates of data and code sharing within medical and health research disciplines. However it remains unclear how commonly data and code are shared across all fields of medical and health research, as well as whether sharing rates are positively associated with implementation of progressive policies...

This paper explores judgements about the replicability of social and behavioural sciences research, and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited using a structured iterative approach for eliciting judgements from groups, called the IDEA protocol (‘Investigate’, ‘Discuss’, ‘...

Assessing the credibility of research claims is a central, continuous, and laborious part of the scientific process. Credibility assessment strategies range from expert judgment to aggregating existing evidence to systematic replication efforts. Such assessments can require substantial time and effort. Research progress could be accelerated if ther...

Experts are often asked to represent their uncertainty as a subjective probability. Structured protocols offer a transparent and systematic way to elicit and combine probability judgements from multiple experts. As part of this process, experts are asked to individually estimate a probability (e.g., of a future event) which needs to be combined/agg...

Replication is a hallmark of scientific research. As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce a new technique to evaluating replicability, the repliCATS (Collaborative Assessments for Trustworthy Science) process, a structured expert elicitation approach based...

Replication, an important, uncommon, and misunderstood practice, is making a comeback in psychology. Achieving replicability is a necessary but not sufficient condition for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and...

Peer review practices differ substantially between journals and disciplines. This study presents the results of a survey of 322 editors of journals in ecology, economics, medicine, physics and psychology. We found that 49% of the journals surveyed checked all manuscripts for plagiarism, that 61% allowed authors to recommend both for and against spe...

Peer review practices differ substantially between journals and disciplines. This study presents the results of a survey of 322 journal editors of high-impact journals in ecology, economics, medicine, physics and psychology. Editors were asked for details about peer review policies and practices at their journals, as well as their views on five pub...

Many recent articles have proposed solutions to the crisis in confidence in the psychology literature, including increased publication of replication studies, a solution that requires engagement by the psychology research community. We surveyed Australian and Italian academic research psychologists about the meaning and role of replication in psych...

Communication and advocacy approaches that influence attitudes and behaviors are key to addressing conservation problems, and the way an issue is framed can affect how people view, judge, and respond to an issue. Responses to conservation interventions can also be influenced by subtle wording changes in statements that may appeal to different value...

Beef production is a major driver of biodiversity loss and greenhouse gas emissions globally, and multiple studies recommend reducing beef production and consumption. Although there have been significant efforts from the biodiversity conservation sector toward reducing beef‐production impacts, there has been comparatively much less engagement in re...

The Community of Open Scholarship Grassroots Networks (COSGN), includes 120 grassroots networks, representing virtually every region of the world and every research discipline. These networks communicate and coordinate on topics of common interest. We propose, using an NSF 19-501 Full-Scale implementation grant, to formalize governance and coordina...

Recent large-scale projects in other disciplines have shown that results often fail to replicate when studies are repeated. The conditions contributing to this problem are also present in ecology but there have not been any equivalent replication projects. Here we examine ecologists’ understanding of and opinions about replication studies. When ask...

Changing human behavior and attitudes are key to conserving global biodiversity. Despite evidence from other disciplines that strategic messaging can influence behavior and attitudes, it remains unclear how to best design messages to benefit biodiversity. We conducted a systematic literature review to investigate the status of conservation messagin...

Current global enthusiasm for urban greening and bringing nature back into cities is unprecedented. Evidence of the socioecological benefits of large, permanent greenspaces is mounting, but the collective potential for pop‐up parks (PUPs) – small, temporary greenspaces – to augment urban ecosystem services is unknown. To showcase the potential of P...

People interpret verbal expressions of probabilities (e.g. ‘very likely’) in different ways, yet words are commonly preferred to numbers when communicating uncertainty. Simply providing numerical translations alongside reports or text containing verbal probabilities should encourage consistency, but these guidelines are often ignored. In an online...

Background: The events of 9/11 and the October 2002 National Intelligence Estimate on Iraq’s Continuing Programs for Weapons of Mass Destruction precipitated fundamental changes within the United States Intelligence Community. As part of the reform, analytic tradecraft standards were revised and codified into a policy document – Intelligence Commun...

This report outlines: a) a need for objective, transparent and usable criteria for judging the decision-readiness of published research evidence and b) the many, important research challenges associated with producing such criteria and ensuring their uptake in the scientific community and beyond. It was produced by Focus Group 2 at TECSS.

We surveyed 807 researchers (494 ecologists and 313 evolutionary biologists) about their use of Questionable Research Practices (QRPs), including cherry picking statistically significant results, p hacking, and hypothesising after the results are known (HARKing). We also asked them to estimate the proportion of their colleagues that use each of the...

Questionable research practice survey. (PDF)

Qualitative data analysis. (DOCX)

Disambiguating QRP 1 cherry-picking vs the file drawer. (DOCX)

Introduction Natural resource management uses expert judgement to estimate facts that inform important decisions. Unfortunately, expert judgement is often derived by informal and largely untested protocols, despite evidence that the quality of judgements can be improved with structured approaches. We attribute the lack of uptake of structured proto...

Defining scoring rules. (PDF)

Elicitation documents. (PDF)

Demographic data and analysis. (PDF)

Peer review is widely considered fundamental to maintaining the rigour of science, but it often fails to ensure transparency and reduce bias in published papers, and this systematically weakens the quality of published inferences. In part, this is because many reviewers are unaware of important questions to ask with respect to the soundness of the...

Peer review is widely considered fundamental to maintaining the rigor of science, but it is an imperfect process. In that context, it is noteworthy that formal standards or guidelines for peer reviews themselves are rarely discussed in many disciplines, including ecology and evolutionary biology. Some may argue that a dearth of explicit guidelines...

Peer review is widely considered fundamental to maintaining the rigor of science, but it is an imperfect process. In that context, it is noteworthy that formal standards or guidelines for peer reviews themselves are rarely discussed in many disciplines, including ecology and evolutionary biology. Some may argue that a dearth of explicit guidelines...

en Article impact statement: Continued development of conservation psychology is essential to addressing the challenges of biodiversity conservation. This article is protected by copyright. All rights reserved

We surveyed 807 researchers (494 ecologists and 313 evolutionary biologists) about their use of Questionable Research Practices (QRPs), including cherry picking statistically significant results, p hacking, and hypothesising after the results are known (HARKing). We also asked them to estimate the proportion of their colleagues that use each of the...

We surveyed 807 researchers (494 ecologists and 313 evolutionary biologists) about their use of Questionable Research Practices (QRPs), including cherry picking statistically significant results, p hacking, and hypothesising after the results are known (HARKing). We also asked them to estimate the proportion of their colleagues that use each of the...

Market-based instruments along with conceptualizing the environment as a collection of ‘ecosystem services’ has become increasingly common within environmental and conservation policy. This kind of thinking is also increasingly prominent in the public discourse surrounding environment and conservation policy, particularly in the context of communic...

We propose to change the default P-value threshold for statistical significance for claims of new discoveries from 0.05 to 0.005.

"We propose to change the default P-value threshold forstatistical significance for claims of new discoveries from 0.05 to 0.005."

Making decisions about the management and conservation of nature is necessarily complex, with many competing pressures on natural systems, opportunities and benefits for different groups of people and a varying, uncertain social and ecological environment. An approach which is narrowly focused on either human development or environmental protection...

Recent replication projects in other disciplines have uncovered disturbingly low levels of reproducibility, suggesting that those research literatures may contain unverifiable claims. The conditions contributing to irreproducibility in other disciplines are also present in ecology. These include a large discrepancy between the proportion of “positi...

We welcome the comment from Clark et al. [1xSee all References][1] and are gratified that they found value in our article. In response, we would like to state that our paper was devoted to exploring relatively well-developed empirical evidence of insufficient transparency in ecology and evolution. We did not review fraud because there is an absence...

To make progress scientists need to know what other researchers have found and how they found it. However, transparency is often insufficient across much of ecology and evolution. Researchers often fail to report results and methods in detail sufficient to permit interpretation and meta-analysis, and many results go entirely unreported. Further, th...

This study presents the results of an approach to the prediction of the outcomes of geopolitical events, which we term the IDEA protocol. The participants investigate the background and causal factors behind a question, predict the outcome, and discuss their thinking with others. They then make a second, private and anonymous judgement of the proba...

Towards the end of the last century, statistical reporting in medical research underwent substantial reform, with null hypothesis significance testing replaced with an estimation approach. Interestingly, this reform may have been largely motivated by the social costs of error within medical research, rather than simply scientific error per se. This...

The topic of advocacy by scientists has been debated for decades, yet there is little agreement about whether scientists can or should be advocates. The fear of crossing a line into advocacy continues to hold many scientists back from contributing to public discourse, impoverishing public debate about important issues. We believe that progress in t...

1. In field surveys, ecological researchers and practitioners routinely make quantitative judgements that are known to vary in quality. Feedback about judgement accuracy is crucial for improving estimation performance, yet is not usually afforded to fieldworkers. One reason it is rare lies in the difficulty of obtaining ‘true values’ (e.g. percenta...

In 2010, the US Intelligence Advanced Research Projects Activity (IARPA) announced a 4-year forecasting "tournament". Five collaborative research teams are attempting to outperform a baseline opinion pool in predicting hundreds of geopolitical, economic and military events. We are contributing to one of these teams by eliciting forecasts from Delph...

We describe a six-step estimation framework for research that starts with the formulation of research goals in terms of “How much?” questions. Such questions are best answered by effect size (ES) estimates and confidence intervals (CIs) calculated from data, where the ESs estimates and CIs are point and interval estimates of population parameters....

The Tiwi people of northern Australia have managed natural resources continuously for 6000-8000 years. Tiwi management objectives and outcomes may reflect how they gather information about the environment. We qualitatively analyzed Tiwi documents and management techniques to examine the relation between the social and physical environment of decisi...

How the brain is lateralised for emotion processing remains a key question in contemporary neuropsychological research. The right hemisphere hypothesis asserts that the right hemisphere dominates emotion processing, whereas the valence hypothesis holds that positive emotion is processed in the left hemisphere and negative emotion is controlled by t...

Aim Expert knowledge routinely informs ecological research and decision‐making. Its reliability is often questioned, but is rarely subject to empirical testing and validation. We investigate the ability of experts to make quantitative predictions of variables for which the answers are known. Location Global. Methods Experts in four ecological subfi...

Replicate samples are important, but they cannot be used to properly test the validity of a scientific hypothesis.

Expert knowledge is used widely in the science and practice of conservation because of the complexity of problems, relative lack of data, and the imminent nature of many conservation decisions. Expert knowledge is substantive information on a particular topic that is not widely known by others. An expert is someone who holds this knowledge and who...

We present the results of an initial experiment that indicates that people are less overconfident and better calibrated when they assign confidence levels to someone else's interval judgements (evaluator confidences) compared to assigning confidence levels to their own interval judgements (judge confidences). We studied what impact this had on a nu...

Suppose you obtain p = .02 in an experiment, then replicate the experiment with new samples. What p value might you obtain, and what interval has an 80% chance of including that replication p? Under conservative assumptions the answer is, perhaps surprisingly (.0003, .30). The authors report three email surveys that asked authors of articles publis...

The information outlines two example questions that represent the style of the test questions used in the six workshops; one example question is for weed ecologists and the other for health epidemiologists. (DOC)

Expert judgements are essential when time and resources are stretched or we face novel dilemmas requiring fast solutions. Good advice can save lives and large sums of money. Typically, experts are defined by their qualifications, track record and experience [1,2]. The social expectation hypothesis argues that more highly regarded and more experienc...

Estimation based on effect sizes, confidence intervals, and meta-analysis usually provides a more informative analysis of empirical results than does statistical significance testing, which has long been the conventional choice in psychology. The sixth edition of the American Psychological Association Publication Manual now recommends that psycholo...

Our research in statistical cognition uses both qualitative and quantitative methods. A mixed method approach makes our research more comprehensive, and provides us with new directions, unexpected insights, and alternative explanations for previously established concepts. In this paper, we review four statistical cognition studies that used mixed m...

A statistically significant result, and a non-significant result may differ little, although significance status may tempt an interpretation of difference. Two studies are reported that compared interpretation of such results presented using null hypothesis significance testing (NHST), or confidence intervals (CIs). Authors of articles published in...

Effect sizes (ESs) and confidence intervals (CIs) are widely advocated in psychology and other disciplines. To date most expository articles have focused only on univariate analyses, despite there being similarly good reasons for reporting and interpreting ESs and CIs following multivariate analyses. We surveyed articles published in leading psycho...

It is a common misconception that statistical significance indicates a large and/or important effect. In fact the three concepts—statistical significance, effect size, and practical importance—are distinct from one another and a favorable result on one dimension does not guarantee the same on any other. In this article, we explain these concepts an...

Elicitation of expert opinion is important for risk analysis when only limited data are available. Expert opinion is often elicited in the form of subjective confidence intervals; however, these are prone to substantial overconfidence. We investigated the influence of elicitation question format, in particular the number of steps in the elicitation...

Null-hypothesis significance tests (NHSTs) have received much criticism, especially during the last two decades. Yet many behavioral and social scientists are unaware that NHSTs have drawn increasing criticism, so this essay summarizes key criticisms. The essay also recommends alternative ways of assessing research findings. Although these recommen...

We demonstrate the application of a novel four-step elicitation technique for generating expert predictions of the concentration of pharmaceutical and over-the-counter drugs expected to be present in recycled water used for supplementing drinking water over the next twenty years (indirect potable reuse or IPR). This work is part of a larger project...

Most questions across science call for quantitative answers, ideally, a single best estimate plus information about the precision of that estimate. A confidence interval (CI) expresses both efficiently. Early experimental psychologists sought quantitative answers, but for the last half century psychology has been dominated by the nonquantitative, d...

Null-hypothesis significance testing (NHST) is the primary means by which data are analyzed and conclusions made, partic- ularly in the social sciences, but in other sciences as well (notably ecology and economics). Despite this supremacy however, numerous problems exist with NHST as a means of interpreting and understanding data. These problems ha...

This report documents the development of a protocol for teaching plain English, based on Richard Lanham's book Revising Prose. Experiments during preliminary writing workshops showed that use of plain English resulted in shorter read times, improved ease of reading, improved logical structure of prose, and slightly improved comprehension by readers...

Practitioners and teachers should be able to justify their chosen techniques by taking into account research results: This is evidence-based practice (EBP). We argue that, specifically, statistical practice and statistics education should be guided by evidence, and we propose statistical cognition (SC) as an integration of theory, research, and app...

The authors examined statistical practices in 193 randomized controlled trials (RCTs) of psychological therapies published in prominent psychology and psychiatry journals during 1999-2003. Statistical significance tests were used in 99% of RCTs, 84% discussed clinical significance, but only 46% considered-even minimally-statistical power, 31% inter...

Many common misinterpretations of Null Hypothesis Significance Testing (NHST) are related to the inverse probability fallacy. The inverse probability fallacy is the mistaken belief that the probability of the data given the null hypothesis, P(D|H0), is equivalent to the probability of the null hypothesis given the data, P(H0|D). We contrasted the e...