PreprintPDF Available

The Waters We Swim In: Replicability and the Evolution of Scientific Norms

Authors:
Preprints and early-stage research may not have been peer reviewed yet.

Abstract

In recent years, a series of high-profile retractions and fraud cases have arisen in physics, sparking a conversation about research integrity and replicability. Here, we discuss how the practice of science is shaped by the social and political context in which it operates. Reflection on our norms and values could provide a route to create community-driven safeguards that respond to the changing demands in which our research occurs. We propose that collaborations between physicists, philosophers, social scientists, and historians of science could facilitate these reflections and provide new ideas for social science and humanities colleagues.
arXiv:2501.05788v1 [physics.soc-ph] 10 Jan 2025
The Waters We Swim In: Replicability and the
Evolution of Scientific Norms
Hope Bretscher1and uria Mu˜noz Gargant´e2
1Max Planck Institute for the Structure and Dynamics of Matter,
Hamburg Germany
2Max Planck Institute for the History of Science, Berlin, Germany
January 2025
0.1 Introduction
In recent years, a series of high-profile retractions and fraud cases in academia
have captured public attention, raising concerns about the integrity of sci-
ence [8, 11]. These scandals spotlight individual labs as exemplars of bad re-
search practice. However, they sit atop a decade of discussions of the so-called
‘replication-crisis’, often centred around fields like psychology and medical re-
search [4]. Staggering statistics in these fields continue to highlight that the
vast majority of research results cannot be reproduced by other researchers.
The breadth of the replication crisis suggests that this problem is not just the
result of a few bad apples intentionally manipulating data, but rather a reflec-
tion of broader systemic issues in research culture [14].
Some physicists argue that the replication crisis is less severe in their do-
mains. They claim that hard-science fields are more likely to be self-correcting
due to their cumulative knowledge building, distributed lab-bench experiments
and often close proximity to commercialisation, where practical applications
provide a tangible test of success or failure [2]. High-profile retractions of con-
troversial claims such as those involving high-TCsuperconductors [3], Ma-
jorana fermions in nanowires [17], or topological superconductivity [10] are
frequently cited as evidence that physics can self-regulate.
However, how often would a physicist with the relevant expertise and tools
who strictly follows published research methods reach the same conclusions as
a given paper? What if they try to re-analyze the raw data published alongside
a paper, following the analysis methods? These are questions of replicability.
In contrast to p-hacking and HARKing (hypothesising after the results are
known) oft discussed in psychology, the breakdown of replicability in physics
may take other forms. These could be the result of pressure to rapidly publish
bold claims in high-impact journals that comes from funding bodies, universities,
1
and peers, interwoven with other norms and practices. Such norms may include
basing studies on a small number of experimental samples which require highly
customised setups to synthesise or measure; embargoed, or lack of transparent
sharing of raw data, code, or processed data; or the necessity for scientists
to employ judgements in selecting and processing relevant datasets. What is
important to note is that these norms are not misconduct—they are standard
practice.
Leaving physics to ‘self-correct’ through technological applications may be
slow, costly, and risky, and require significant financial and research capital.
What happens to public trust in science when breakthroughs are later debunked
or fail to materialise? As funding and hype in emerging fields like quantum
technologies ramp up, the risk of scientific scandals increases. If our research
becomes plagued by replication crises and we lose the public’s trust in the ro-
bustness of science, then we fear that our physics futures will be put in peril.
Avoiding these risks requires developing safeguards. These safeguards must
promote rigorous and lively research while addressing the specific practices and
incentives that shape science today. In a sense, to make progress and ensure
a healthy ecosystem, we believe scientists must dive deeper into the pressures
and norms, going beyond those listed above, and become aware of the water
we swim in. As this introspection can be challenging, collaboration with the
humanities can catalyse this awareness.
0.2 Watch the Water: Understanding the Contexts that
Shape Scientific Knowledge
John Ziman, a condensed matter theorist-turned sociologist of science, remarked
that “Scientists know philosophy and sociology as fish know water. They un-
derstand instinctively how to live in it without being aware that they are doing
so.” [18, p. 751]. Ziman’s point was not that scientists are naturally skilled in
philosophy or sociology, but rather that they unconsciously follow norms and
practices without considering how these influence their research. While these
norms are often viewed as immutable, we want to emphasise that they are his-
torically contingent, shaped by social, economic, and political factors.
Honouring his background in condensed matter physics, Ziman argued that
to truly understand the “water” in which science operates, “this medium needs
to be broken down into its component parts, perhaps to be resynthesized in
new and more up-to-date forms.” [18, p. 751] Recognising and analysing
these components—whether they are methodologies, theoretical and philosoph-
ical frameworks, or social norms—is one of the central tasks of science studies,
historians and philosophers of science. A closer attention and collaboration with
these fields could support scientists to gain a better meta-perspective on their
own practices and evolving medium.
Much like in physics, this medium where science operates exhibits emergent
properties. The concept of emergence, introduced to condensed matter physics
by Philip Anderson—a contemporary and colleague of Ziman—has become a
foundational philosophical principle in this field.
2
In his essay “More Is Different” (1972), Anderson argued that although
all systems in nature are governed by universal laws, certain properties, such
as those resulting from spontaneous symmetry breaking, are emergent. These
emergent properties cannot be fully predicted based on fundamental laws at
smaller scales but arise instead from the collective behaviour of a system’s com-
ponents. In such cases, he noted, “the whole becomes not only more than but
very different from the sum of its parts.” [1, p. 395]. Similarly, the norms and
practices of science are emergent, shaped by the collective actions and interac-
tions of scientific communities, and evolve over time.
With his expertise in both condensed matter physics and sociology of sci-
ence, Ziman extended the concept of emergence to scientific knowledge, suggest-
ing that what counts as valid, reproducible knowledge in one science emerges
from the set of norms and practices associated with that science [18]. Ziman
warned against idealising scientific norms as static and universal. Instead, he
argued that understanding how these norms are negotiated in response to ex-
ternal pressures is key to addressing issues like the replication crisis. This view
resonates with developments in the history and philosophy of science, where it is
acknowledged that scientific norms, including the very notion of “objectivity,”
are subject to historical change.
0.3 Objectivity and Replicability as Evolving Norms
The concept of objectivity itself has evolved significantly over time, as docu-
mented by historians of science such as Lorraine Daston and Peter Galison in
their book Objectivity. They trace how the understanding of what it means to
be an “ob jective” scientist has shifted through various historical eras, affect-
ing both the training of scientists and the perception of what constitutes good
science [5]. Initially, in the 18th century, the ideal was truth-to-nature, where
scientists were expected to refine observations to represent the ideal forms of
phenomena. By the mid-19th century, this gave way to mechanical objectiv-
ity, which sought to eliminate personal interpretation entirely, relying instead
on instruments and unbiased recording methods like photography. Finally, the
20th century brought the ideal of trained judgement, blending objectivity with
expert discernment to interpret complex data. These shifts in the meaning of
objectivity show that what counts as rigorous science is historically contingent
and culturally shaped.
These historical transformations in objectivity resonate strongly with the
replication crisis in modern science. The replication crisis underscores the fact
that replicability, often considered a hallmark of objective science, is itself a
concept that varies according to which version of objectivity is in play. When
mechanical objectivity dominated, replication meant exact duplication of a re-
sult under identical conditions, ensuring that findings were free from subjective
interpretation. This notion, however, is often unrealistic in complex fields such
as biology and psychology, where exact replication is difficult due to variations
in experimental conditions, human subjects, and statistical methods.
The failure of many scientific studies to replicate today suggests that the sci-
3
entific community’s understanding of replicability may need to evolve again, just
as it did for objectivity. Trained judgement might offer a more nuanced perspec-
tive, acknowledging that some level of expert interpretation and context-specific
adjustment is often necessary to achieve meaningful reproducibility. However,
trained judgement, if not transparently communicated, can also open the door
to confirmation bias and selective reporting, which are among the very issues
driving the current crisis.
Feminist philosopher of science Helen Longino provides an additional layer
of analysis, arguing for a social model of objectivity that shifts focus from in-
dividual scientists to the community as a whole [13]. Longino emphasises that
objectivity is not achieved by eliminating personal bias through detachment but
by promoting critical interaction and diverse perspectives within scientific com-
munities. This approach positions the replication crisis not simply as a technical
problem of flawed methodology but as a sociological issuea failure of the scien-
tific community to maintain sufficient diversity of viewpoints and openness to
scrutiny.
This perspective aligns with historian of science Naomi Oreskes, who has
argued that scientific knowledge is inherently social, shaped by collective prac-
tices and continually refined as new perspectives enter the field. For Oreskes,
scientific knowledge is always provisional, built through processes of debate, cri-
tique, and consensus-building [15]. Thus, a crisis of replication is also a crisis of
scientific communication and community norms. When the community norms
discourage dissent, prioritise novel findings over careful replication, or impose
rigid publication requirements, even the best scientific methods can produce
unreliable results.
This understanding reframes the replication crisis as a crisis of scientific
culture rather than merely a technical problem to be solved by better statistics
or experimental protocols. If objectivity itself is an emergent property of the
interactions within a scientific community, then addressing the replication crisis
requires more than enforcing stricter methodologies. It involves cultivating a
culture where replication, critique, and transparency are valued as much as
discovery and innovation.
0.4 Where Do We Go from Here?
Ultimately, the replication crisis can be seen as a reflection of deeper tensions
within science’s transforming ideals of objectivity and how they intersect with
the pressures of modern research environments. By engaging with these broader
discussions about the philosophy and sociology of science, we can better un-
derstand the challenges of ensuring replicability in a complex, ever-changing
research landscape.
A step forward in this discussion requires accepting that that the knowl-
edge we produce is molded by the incentives, pressures, values, and technologies
of the day. The strengthening of ties between commercialisation, geopolitical
motivations, and basic academic research which was true in Ziman’s day, is
perhaps even more pronounced today. National funding priorities and inter-
4
national trends shape what types of research is pursued and what questions
are asked. Easily quantifiable metrics such as paper published, citations, and
impact-factors, now critical for career advancement, influence the depth and
scope of research pursued in papers. The rapid communication made possible
by the internet has begun to shift the norms of research, affecting expectations
about sample sizes, methodological rigour, and transparency of raw data.
We challenge scientists to reflect on the pressures shaping our current medium,
define what replication and self-correction mean in this context, and re-imagine
how our norms and scientific practices could respond to these pressures. For ex-
ample, the traditional scientific paper format has remained largely unchanged
despite the rise of the internet and social media, which allow for rapid com-
munication and greater data sharing. How could these technologies facilitate
more rigorous evaluation and dissemination of data, code, and scientific results?
Could papers become a more ‘living’ documents or discussions with transparent
histories, to allow conclusions to be updated and revised as new information
comes to light? Additionally, how could replication work be published or recog-
nised? As early career researchers, we ourselves are inspired by the work of
many before us, and by the many conversations and creative solutions that are
being explored by scientists and that can seed new changes [2, 9, 16].
Second, we wish to encourage multidisciplinary collaborations and conver-
sations between scientists, historians, sociologists and philosophers of science
[6, 7, 12]. These exchanges not only will allow scientists to swim more inten-
tionally through the complex social landscape of research, but also provide new
ideas and inspiration for the social sciences and humanities colleagues.
At times, it may seem impossible to intervene in systems that, being the
result of global financial, economic, and social trends, feel too large and en-
trenched to change. But as scientists, we each still play our own small role in
creating and reproducing the norms and practices of our field, just as we create
and reproduce the small ‘bits’ of research, that together, will emerge as scientific
knowledge in the future.
5
References
[1] Anderson, P. W. More is different: Broken symmetry and the nature of
the hierarchical structure of science. Science 177, 4047 (1972), 393–396.
[2] Buck, S. When does science self-correct? lessons from a replication crisis
in early 20th century chemistry, Nov 2023.
[3] Castelvecchi, D. Nature retracts controversial superconductivity paper
by embattled physicist. Nature (2023).
[4] Collaboration, O. S. Estimating the reproducibility of psychological
science. Science 349, 6251 (2015), aac4716.
[5] Daston, L., and Galison, P. L. Objectivity. Princeton University Press,
2021.
[6] Daston, L., and Harrison, P. Science and history cannot afford to be
indifferent to each other: Aeon essays, Jan 2024.
[7] de Swart, J., Thresher, A. C., and Arg¨
uelles, C. A. The human-
ities can help make physics greener. Nature Reviews Physics 6, 7 (2024),
404–405.
[8] Dyer, O. Departing stanford president retracts two science papers after
investigation, 2023.
[9] Frolov, S., Akrap, A., Guchhait, S., Seradjeh, B., and
Skinner, B. International conference on reproducibility in con-
densed matter physics. url=https://www.pqi.org/international-conference-
reproducibility-condensed-matter-physics, 2024.
[10] Garisto, D. Retraction imminent for controversial physicist. Nature 620
(2023), 3.
[11] Kincaid, E. Superconductor researcher loses fifth paper, Jun 2024.
[12] Le’vy, R., Mody, C., Labbe’, C., and Halffman, W. Nanobubbles.
urlhttps://nanobubbles.hypotheses.org/, 2021.
[13] Longino, H. E. Science as social knowledge: Values and objectivity in
scientific inquiry. Princeton University Press, 1990.
[14] of Sciences, N. A., Policy, Affairs, G., on Research Data, B.,
Information, on Engineering, D., Sciences, P., on Applied, C.,
Statistics, T., on Mathematical Sciences, B., et al. Reproducibility
and replicability in science. National Academies Press, 2019.
[15] Oreskes, N. The scientific consensus on climate change. Science 306,
5702 (2004), 1686–1686.
6
[16] Skinner, B., and Frolov, S. workshop - virtual science forum.
https://virtualscienceforum.org/reproducibility_in_cmp/, 2023.
[17] Zhang, H., Liu, C.-X., Gazibegovic, S., Xu, D., Logan, J. A.,
Wang, G., Van Loo, N., Bommer, J. D., De Moor, M. W., Car,
D., et al. Retracted article: Quantized majorana conductance. Nature
556, 7699 (2018), 74–79.
[18] Ziman, J. Is science losing its objectivity? Nature 382, 6594 (1996),
751–754.
7
ResearchGate has not been able to resolve any citations for this publication.
Article
Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
Article
The philosophy of science is not independent of the way research is organized. Can scientists produce objective knowledge in a world where their research is increasingly directed towards making money or meeting social needs?
When does science self-correct? lessons from a replication crisis in early 20th century chemistry
  • S Buck
Buck, S. When does science self-correct? lessons from a replication crisis in early 20th century chemistry, Nov 2023.