No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Replication crisis = trust crisis? The effect of successful vs failed replications on laypeople's trust in researchers and research
Abstract
In methodological and practical debates about replications in science, it is (often implicitly) assumed that replications will affect public trust in science. In this preregistered experiment (N = 484), we varied (a) whether a replication attempt was successful or not and (b) whether the replication was authored by the same, or another lab. Results showed that ratings of study credibility (e.g. evidence strength, ηP2 = .15) and researcher trustworthiness (e.g. expertise, ηP2 = .15) were rated higher upon learning of replication success, and lower in case of replication failure. The replication’s author did not make a meaningful difference. Prior beliefs acted as covariate for ratings of credibility, but not trustworthiness, while epistemic beliefs regarding the certainty of knowledge were a covariate to both. Hence, laypeople seem to notice that successfully replicated results entail higher epistemic significance, while possibly not taking into account that replications should be conducted by other labs.
... Although open science does not guarantee "good science" (see Markowitz et al., 2021), the public nevertheless may perceive a scientific study adopting open science practices to have taken extra time, effort, and contain additional rigor for quality controls-therefore evaluating such work to be of a higher quality than their counterparts. Scientists and their research are frequently trusted by the public as a reliable source of fair and accurate information (Hendriks et al., 2020;Pew Research Center, 2019), and in this light, the public's improved perception of research quality for open science (vs. non-open science) may mainly stem from the perceived expertise of the researcher (see Hendriks et al., 2016Hendriks et al., , 2020, but not necessarily from the perception of integrity and the lack of self-interest presupposed by the logic of open science. ...
... Scientists and their research are frequently trusted by the public as a reliable source of fair and accurate information (Hendriks et al., 2020;Pew Research Center, 2019), and in this light, the public's improved perception of research quality for open science (vs. non-open science) may mainly stem from the perceived expertise of the researcher (see Hendriks et al., 2016Hendriks et al., , 2020, but not necessarily from the perception of integrity and the lack of self-interest presupposed by the logic of open science. ...
... The current paper aimed to address the assumed, yet rarely substantiated, promises and benefits of open science practices-the argument that more transparent research standards adhering to open science may rescue eroding public trust in science (e.g., Besanc¸on et al., 2021;Dienlin et al., 2021). Previous research addressing similar questions, so far, reported mixed results (e.g., Anvari & Lakens, 2018;Hendriks et al., 2020;Wingen et al., 2020). While recent observational evidence suggests open science-consistent research-especially, registered reports-is evaluated more positively during the peer review process than non-openscience-consistent research (Soderberg et al., 2021), there was a lack of comprehensive, causal evidence of whether open science practices could increase trust toward scientists and their work. ...
Researchers often focus on the benefits of adopting open science, yet questions remain whether the general public, as well as academics, value and trust studies consistent with open science compared to studies without open science. In three preregistered experiments (total N = 2,691), we find that the general public perceived open science research and researchers as more credible and trustworthy than non-open science counterparts (Studies 1 and 2). We also explored if open science practices compensated for negative perceptions of privately-funded research versus publicly-funded research (Study 2), although the evidence did not support this hypothesis. Finally, Study 3 examined how communication scholars perceive researchers and their work as a function of open science adoption, along with publication outlet (e.g., high-prestige vs. low-prestige journals). We observed open science research was perceived more favorably than non-open science research by academics. We discuss implications for the open science movement and public trust in science.
... Relative to the control condition with no information about the results' replicability, perceptions of the researcher's ability and ethics, as well as perceived truthfulness of the results, increased when another researcher replicated the original findings, but significantly decreased when the findings failed to replicate. Similarly, in a primarily German sample, Hendriks et al. (2020) manipulated whether a study replication attempt was successful or unsuccessful, and found that failure to replicate decreased ratings of study credibility and researcher trustworthiness. ...
... The goal of the present work was to address this important limitation and to expand upon the nascent knowledge base regarding perceptions of failed replications in psychology. First, we wished to examine a broad range of reactions to a researcher's (ostensible) replication failure versus success, replicating and extending past experiments documenting some narrower reputational costs of failed replications (Ebersole et al., 2016;Hendriks et al., 2020). We explored both attitudes and behavioral intentions toward interacting with the researcher, as well as perceptions of their research in general, and we predicted an overall replication-failure bias (such that researchers and their work broadly would be perceived substantially more negatively in the replication failure vs. success condition). ...
... Study 1 conceptually replicated past research and provided further evidence that replication failures lead to more negative perceptions of researchers and their research. Expanding upon earlier findings (Ebersole et al., 2016;Hendriks et al., 2020), we demonstrated in a sample of adults not only did the public have more negative perceptions of a researcher's competence and scientific integrity and of their research when their findings failed to replicate than when they replicated successfully, but they also liked the researcher less and reported weaker behavioral intentions to interact with them in the future. Of importance, the observed effect sizes were very robust, suggesting that the consequences of a failed replication are quite serious and arguably greater than is justified, given the nature of what a failed replication can(not) tell us (Maxwell et al., 2015;Schmidt and Oh, 2016). ...
The reproducibility movement in psychology has resulted in numerous highly publicized instances of replication failures. The goal of the present work was to investigate people’s reactions to a psychology replication failure vs. success, and to test whether a failure elicits harsher reactions when the researcher is a woman vs. a man. We examined these questions in a pre-registered experiment with a working adult sample, a conceptual replication of that experiment with a student sample, and an analysis of data compiled and posted by a psychology researcher on their public weblog with the stated goal to improve research replicability by rank-ordering psychology researchers by their “estimated false discovery risk.” Participants in the experiments were randomly assigned to read a news article describing a successful vs. failed replication attempt of original work from a male vs. female psychological scientist, and then completed measures of researcher competence, likability, integrity, perceptions of the research, and behavioral intentions for future interactions with the researcher. In both working adult and student samples, analyses consistently yielded large main effects of replication outcome, but no interaction with researcher gender. Likewise, the coding of weblog data posted in July 2021 indicated that 66.3% of the researchers scrutinized were men and 33.8% were women, and their rank-ordering was not correlated with researcher gender. The lack of support for our pre-registered gender-replication hypothesis is, at first glance, encouraging for women researchers’ careers; however, the substantial effect sizes we observed for replication outcome underscore the tremendous negative impact the reproducibility movement can have on psychologists’ careers. We discuss the implications of such negative perceptions and the possible downstream consequences for women in the field that are essential for future study.
... In the ERE, students learn the importance of reproducible results, and the impact of the replication crisis on the natural sciences more broadly [21,22]. Through participation in the ERE, students learn how to reproduce, confirm, and update previous results, all while simultaneously advancing current scientific knowledge. ...
... The lesson teaches students what a transit is, how to observe a transiting exoplanet, how a transit light curve is generated, and how conditions can affect the appearance of a light curve. After completing the transits lesson, students move on to a background literature review on the topics of transit photometry [24,28,29,34] and the importance of reproducibility in scientific research [21,22]. After reading the suggested articles, the students complete a discussion board post summarizing, listing any questions they have, and engaging with at least one post made by their peers. ...
Course-Based Undergraduate Research Experiences (CUREs) have been shown to provide students with a variety of learning benefits including better conceptual understanding, improved critical thinking and data literacy skills, and increased interest in pursuing scientific careers. Additionally, CUREs provide students with opportunities to participate in authentic research experiences that have a broader impact outside of the classroom. Despite the numerous benefits, the field of astronomy has lagged behind disciplines like biology and chemistry when it comes to including CUREs in the curriculum. Not limited to astronomy, however, is the lack of research opportunities and courses offered to students enrolled in undergraduate degree programs online. In the Fall of 2020, Arizona State University (ASU) introduced the nation’s first online bachelor’s degree program in astronomy and planetary sciences (APS). To make research accessible to a more diverse population of learners, it is imperative that students in this program have access to the same opportunities to participate in authentic research as those in the parallel in-person program. In this work, we describe the development, implementation, and assessment of a fully online CURE for astronomy majors as part of the APS program. We conducted a mixed methods analysis consisting of a Likert style survey administered pre- and postcourse as well as student interviews at the conclusion of the semester. Survey results from the course’s first two offerings (N=24) indicated that students’ research self-efficacy and science identity both improved. An exoplanet-specific multiple-choice assessment (N=26) showed statistically significant improvements in conceptual understanding postcourse. Additionally, student interview (N=11) responses relayed that students felt a stronger sense of belonging to both ASU and the larger astronomy community after participation in the course. The results from this study are encouraging and suggest that student participation in this online CURE led to similar improvements across a variety of outcomes previously identified in studies of in-person CUREs spanning multiple disciplines.
... With software having such a driving importance in our digital lives and increasingly in academic research [24], the preservation of software is a contributor to the issue of academic reproducibility, commonly referenced as a "replication crisis" [16,23,39]. Open source software and data are important components in cultivating a more open methodology of science to alleviate this crisis [8,9,22,38], though software is useless if it cannot be found or accessed [28]. ...
Software is often developed using versioned controlled software, such as Git, and hosted on centralized Web hosts, such as GitHub and GitLab. These Web hosted software repositories are made available to users in the form of traditional HTML Web pages for each source file and directory, as well as a presentational home page and various descriptive pages. We examined more than 12,000 Web hosted Git repository project home pages, primarily from GitHub, to measure how well their presentational components are preserved in the Internet Archive, as well as the source trees of the collected GitHub repositories to assess the extent to which their source code has been preserved. We found that more than 31% of the archived repository home pages examined exhibited some form of minor page damage and 1.6% exhibited major page damage. We also found that of the source trees analyzed, less than 5% of their source files were archived, on average, with the majority of repositories not having source files saved in the Internet Archive at all. The highest concentration of archived source files available were those linked directly from repositories' home pages at a rate of 14.89% across all available repositories and sharply dropping off at deeper levels of a repository's directory tree.
... Second, neoliberalism might have increased pressures to publish ('publish or perish' culture 59,60 ), fuelling questionable research practices associated with the replication crisis 61,62 . Although the replication crisis in both natural and social sciences [63][64][65] is fairly recent (compared with, for example, when the lower trust among conservatives was first observed 2 ), it can indeed lead to public's decreased trust in science [66][67][68] . In short, the internal dynamics of science-embedded within a wider societal system-is one of the crucial aspects to account for when it comes to public (dis)trust in science. ...
Trust in scientists is a key predictor of compliance with science-based solutions to societal challenges. Although liberals in the US generally trust scientists more than conservatives do, it is not clear how these ideological differences vary across different scientific occupations and whether they can be mitigated. The present registered report (N = 7,800, US participants) demonstrated that even though the strength of the relationship between political ideology and trust varies across scientific occupations, liberals (compared to conservatives) show higher trust in most scientists. Moreover, following motivational accounts of scientist distrust, the study tested five theoretically grounded intervention strategies to improve conservatives’ trust in scientists. None of the interventions were successful, suggesting that trust in scientists reflects relatively stable attitudes that require more elaborate and time-intensive interventions.
... Second, neoliberalism might have increased pressures to publish ('publish or perish' culture 59,60 ), fuelling questionable research practices associated with the replication crisis 61,62 . Although the replication crisis in both natural and social sciences [63][64][65] is fairly recent (compared with, for example, when the lower trust among conservatives was first observed 2 ), it can indeed lead to public's decreased trust in science [66][67][68] . In short, the internal dynamics of science-embedded within a wider societal system-is one of the crucial aspects to account for when it comes to public (dis)trust in science. ...
Trust in scientists is a key predictor of compliance with science-based solutions to societal challenges. Although liberals in the USA generally trust scientists more than conservatives do, it is not clear how these ideological differences vary across different scientific occupations and whether they can be mitigated. Here, in this Registered Report (including 7,800 US participants), we demonstrate that, even though the strength of the relationship between political ideology and trust varies across scientific occupations, liberals (compared with conservatives) show higher trust in most scientists. Moreover, following motivational accounts of scientist distrust, the study tested five theoretically grounded intervention strategies to improve conservatives’ trust in scientists. None of the interventions were successful, suggesting that trust in scientists reflects relatively stable attitudes that require more elaborate and time-intensive interventions.
... For example, science's fraught historical relationship with racism, its role in perpetuating racialized forms of knowledge production, sustaining racial paradigms 29 and disregarding ethical canons by experimenting on non-white human subjects 30 , has reduced research participation in some populations 31 . Furthermore, the epistemic authority of science and scientists has been challenged by misinformation and disinformation 32,33 , a "reproducibility crisis" 34 , conspiracy theories 35,36 and science-related populist attitudes 37,38 . Science-related populism has been conceptualized as a perceived antagonism between 'the ordinary people' and common sense on one side and academic elites and scientific expertise on the other 37 . ...
This 68-country survey (n = 71,922) examines how people encounter information about science and communicate about it with others, identifies cross-country differences, and tests the extent to which economic and sociopolitical conditions predict such differences. We find that social media are the most used sources of science information in most countries, except those with democratic-corporatist media systems where news media tend to be used more widely. People in collectivist societies are less outspoken about science in daily life, whereas low education is associated with higher outspokenness. Limited access to digital media is correlated with participation in public protests on science matters.
... In this regard, we believe that outcome-wide approaches (VanderWeele, 2017), in particular, hold exceptional promise, given the inherent complexity and heterogeneity of potential effects in science communication. For example, uncertainty communication may be generally advisable Dries et al., 2024) but still have negative effects on trust in certain contexts, such as uncertainty as a result of low replicability (Anvari & Lakens, 2018;Hendriks, Kienhues et al., 2020;Wingen et al., 2020). Similarly, a high degree of simplification of scientific information may lead to better subjective understanding but also to unwanted side effects in terms of a subsequent devaluation of expert advice (easiness-effect; Scharrer et al., 2012) or an inflated sense of one's own knowledge (e.g., lower intellectual humility; Vaupotič et al., 2024). ...
... For example, science's fraught historical relationship with racism, its role in perpetuating racialized forms of knowledge production, sustaining racial paradigms 29 and disregarding ethical canons by experimenting on non-white human subjects 30 , has reduced research participation in some populations 31 . Furthermore, the epistemic authority of science and scientists has been challenged by misinformation and disinformation 32,33 , a "reproducibility crisis" 34 , conspiracy theories 35,36 and science-related populist attitudes 37,38 . Science-related populism has been conceptualized as a perceived antagonism between 'the ordinary people' and common sense on one side and academic elites and scientific expertise on the other 37 . ...
Science is crucial for evidence-based decision-making. Public trust in scientists can help decision makers act on the basis of the best available evidence, especially during crises. However, in recent years the epistemic authority of science has been challenged, causing concerns about low public trust in scientists. We interrogated these concerns with a preregistered 68-country survey of 71,922 respondents and found that in most countries, most people trust scientists and agree that scientists should engage more in society and policymaking. We found variations between and within countries, which we explain with individual- and country-level variables, including political orientation. While there is no widespread lack of trust in scientists, we cannot discount the concern that lack of trust in scientists by even a small minority may affect considerations of scientific evidence in policymaking. These findings have implications for scientists and policymakers seeking to maintain and increase trust in scientists.
... These dimensions and measures show similarities to the dimensions of competence, trustworthiness, and goodwill/ caring identified by McCroskey and Teven (1999), rooted in theoretical foundations reaching back to Aristotle. The conceptualization of the Muenster Epistemic Trustworthiness Inventory (METI) has proven useful in numerous studies (e.g., Hendriks et al., 2020Hendriks et al., , 2023Reif et al., 2020) and has been developed in the context of digital media use. It also aligns with the twodimensional concept of Fiske and Dupree (2014), where expertise is referred to as competence, and warmth combines integrity and benevolence (similar to the basic dimensions of source credibility by Hovland et al., 1953). ...
We developed and validated the Public Trust in Science (PuTruS) Scale to recognize the multidimensional nature of the concept. Drawing on an epistemic understanding of trust in science as a perception, we integrate prior research on different levels of trust objects, emphasizing the importance of trust in scientists. In addition, we include transparency and dialogue orientation to reflect increased public engagement expectations. Data from two German online panel surveys (n W1 = 3,439; n W2 = 1,030) points toward a five-dimensional structure (expertise, integrity, benevolence, transparency, dialogue orientation). For external validation, we used deference to scientific authority, conspiracy beliefs, and science populism.
... Answering this question can provide insights into what sources of information potentially affect public trust in science (positively or negatively) via their use of trust cues. Studies to date have either focused on the effects of the frequency of media use on public trust in science [e.g., Huber et al., 2019;Wintterlin et al., 2022], often pointing to small or no effects, or to effects of very specific contents [e.g., Hendriks et al., 2020]. We argue that there are two main reasons why science-related information use has not emerged as a (significant or strong) predictor of trust in science: firstly, researchers did not establish the connection to specific content, and secondly aggregated analyses were performed for overall samples. ...
A potential decline in public trust in science has often been linked to digital media environments, which serve as intermediaries of trust by providing cues for why (not) to trust science. This study examines whether exposure to trust cues in content affects public trust in science (across population groups). The study employs a mixed-method design, combining content analysis (n = 906) and panel survey data (n = 1,030) in Germany. The findings reveal that exposure to trust cues in certain media predicts public trust in science. Variations across trust groups indicate a nuanced nature of trust-assessing processes in digital media environments.
... This term draws an analogy to human psychology, which also deals with opaque structures -human minds -by analyzing observable behaviors and responses (Taylor & Taylor, 2021). However, psychology has faced a replication crisis, caused by issues such as small sample sizes, poorly designed experiments, publication bias, lack of transparency, low statistical power, selective reporting, preferences for novelty, or the general complexity of psychological phenomena (Hendriks et al., 2020;Lilienfeld & Strother, 2020). Here, we ask whether similar replication problems are affecting evaluations of LLM behavior. ...
In an era where large language models (LLMs) are increasingly integrated into a wide range of everyday applications, research into these models' behavior has surged. However, due to the novelty of the field, clear methodological guidelines are lacking. This raises concerns about the replicability and generalizability of insights gained from research on LLM behavior. In this study, we discuss the potential risk of a replication crisis and support our concerns with a series of replication experiments focused on prompt engineering techniques purported to influence reasoning abilities in LLMs. We tested GPT-3.5, GPT-4o, Gemini 1.5 Pro, Claude 3 Opus, Llama 3-8B, and Llama 3-70B, on the chain-of-thought, EmotionPrompting, ExpertPrompting, Sandbagging, as well as Re-Reading prompt engineering techniques, using manually double-checked subsets of reasoning benchmarks including CommonsenseQA, CRT, NumGLUE, ScienceQA, and StrategyQA. Our findings reveal a general lack of statistically significant differences across nearly all techniques tested, highlighting, among others, several methodological weaknesses in previous research. We propose a forward-looking approach that includes developing robust methodologies for evaluating LLMs, establishing sound benchmarks, and designing rigorous experimental frameworks to ensure accurate and reliable assessments of model outputs.
... For instance, learners adopt norms for justifying arguments from their teachers and each other (Ryu & Sandoval, 2012) and can engage in meta-talk about the quality of their arguments (Kuhn et al., 2013). They also evaluate the knowledge production of other social groups, such as the knowledge produced by economists or scientists (Hendriks et al., 2020). Sometimes these social epistemic processes can be inequitable and unjust, such as when students are deprived of opportunities to engage in inquiry and higher-order thinking (Zohar et al., 2001) or when people's testimony is accorded lower trustworthiness due to their gender or ethnicity (Fricker, 2007). ...
The nurturing of learners’ ways of knowing is vital for supporting their intellectual growth and their participation in democratic knowledge societies. This paper traces the development of two interrelated theoretical frameworks that describe the nature of learners’ epistemic thinking and performance and how education can support epistemic growth: the AIR and Apt-AIR frameworks. After briefly reviewing these frameworks, we discuss seven reflections on educational theory development that stem from our experiences working on the frameworks. First, we describe how our frameworks were motivated by the goal of addressing meaningful educational challenges. Subsequently, we explain why and how we infused philosophical insights into our frameworks, and we also discuss the steps we took to increase the coherence of the frameworks with ideas from other educational psychology theories. Next, we reflect on the important role of the design of instruction and learning environments in testing and elaborating the frameworks. Equally important, we describe how our frameworks have been supported by empirical evidence and have provided an organizing structure for understanding epistemic performance exhibited in studies across diverse contexts. Finally, we discuss how the development of the frameworks has been spurred by dialogue within the research community and by the need to address emerging and pressing real-world challenges. To conclude, we highlight several important directions for future research. A common thread running through our work is the commitment to creating robust and dynamic theoretical frameworks that support the growth of learners’ epistemic performance in diverse educational contexts.
... More trust in science is, for instance, positively related to pro-environmental behaviour (Cologna and Siegrist, 2020). Previous research has dealt with how communication about the nature of science, such as scientific uncertainty (Van der Bles et al., 2020), two-sided scientists' messages (Mayweg-Paus and Jucks, 2018), communication focused on replication (Hendriks et al., 2020) and self-correction (Altenmüller et al., 2021) affects perceived trustworthiness of scientists. While some empirical studies have shown that communicating epistemic uncertainty has no negative effect on perceived trustworthiness. ...
In the context of science communication, complexity is often reduced. This study employs a 2 × 2 experimental design (N = 432) to investigate how two factors, namely the communication of complexity (reduced vs not reduced) and the provision of suggestions for concrete action (suggested vs not suggested), influence individuals' productive engagement with the socio-scientific topic of sustainable energy. Measured variables include topic-specific intellectual humility, judgements of source trustworthiness, willingness to act, anxiety, and hope. As expected, communication of complexity led to higher topic-specific intellectual humility, higher epistemic trustworthiness and higher anxiety. When a concrete action was communicated, participants reported lower topic-specific intellectual humility. Participants' willingness to act was not significantly affected by the experimental manipulation. The results of the study imply that the communication of complexity does not hinder people's productive engagement with science.
... Based on this three-fold concept, the reasons why scientific experts are perceived as trustworthy are high levels of expertise, integrity, and benevolence. This conceptualization of the Muenster Epistemic Trustworthiness Inventory (METI) has proven useful in numerous studies (e.g., Hendriks et al., 2022;Hendriks et al., 2020;Reif et al., 2020) and has been developed in the context of digital media use. However, it is also similar to the two-dimensional concept of Fiske and Dupree (2014), in which expertise is referred to as competence. ...
Public trust in science is pivotal, yet its measurement often lacks complexity. We developed, tested, and validated the theory-based multidimensional Public Trust in Science (PuTS) Scale. Data from two German online panel surveys (nW1 = 3,439; nW2 = 1,030) confirmed the superiority of five dimensions (expertise, integrity, benevolence, transparency, dialogue) over one, two, or four. We also introduced a concise five-item version (PuTS-5). External validation employed conspiracy beliefs and science populism. Our instrument, rooted in theory, provides an empirically manageable approach to assess public trust in science and may improve future research endeavors on the science–public trust relationship.
... 29 As a consequence, limited reproducibility raises doubts in the quality of scientific results and contributes to a loss of trust in science. 30 The reproducibility crisis in the biomedical sciences 31 and catalytic sciences 32 also exists in the field of biocatalysis. More and more researchers perceive an information overload and feel like drowning in a data tsunami. ...
... While science is generally held in high esteem, its epistemic and cultural authority has been challenged by mis-and disinformation 11,12 , historical failings of science 13 , an alleged "reproducibility crisis" 14 , conspiracy theories 15,16 , and science-related populist attitudes 17,18 . ...
Scientific information is crucial for evidence-based decision-making. Public trust in science can help decision-makers act based on the best available evidence, especially during crises such as climate change or the COVID-19 pandemic 1,2. However, in recent years the epistemic authority of science has been challenged, causing concerns about low public trust in scientists 3. Here we interrogated these concerns with a pre-registered 67-country survey of 71,417 respondents on all inhabited continents and find that in most countries, a majority of the public trust scientists and think that scientists should be more engaged in policymaking. We further show that there is a discrepancy between the public's perceived and desired priorities of scientific research. Moreover, we find variations between and within countries, which we explain with individual-and country-level variables, including political orientation. While these results do not show widespread lack of trust in scientists, we cannot discount the concern that lack of trust in scientists by even a small minority may affect considerations of scientific evidence in policymaking. These findings have implications for scientists and policymakers seeking to maintain and increase trust in scientists.
... While science is generally held in high esteem, its epistemic and cultural authority has been challenged by mis-and disinformation 11,12 , historical failings of science 13 , an alleged "reproducibility crisis" 14 , conspiracy theories 15,16 , and science-related populist attitudes 17,18 . ...
Scientific information is crucial for evidence-based decision-making. Public trust in science can help decision-makers act based on the best available evidence, especially during crises such as climate change or the COVID-19 pandemic. However, in recent years the epistemic authority of science has been challenged, causing concerns about low public trust in scientists. Here we interrogated these concerns with a pre-registered 67-country survey of 71,417 respondents on all inhabited continents and find that in most countries, a majority of the public trust scientists and think that scientists should be more engaged in policymaking. We further show that there is a discrepancy between the public's perceived and desired priorities of scientific research. Moreover, we find variations between and within countries, which we explain with individual-and country-level variables, including political orientation. While these results do not show widespread lack of trust in scientists, we cannot discount the concern that lack of trust in scientists by even a small minority may affect considerations of scientific evidence in policymaking. These findings have implications for scientists and policymakers seeking to maintain and increase trust in scientists.
... While science is generally held in high esteem, its epistemic and cultural authority has been challenged by mis-and disinformation 11,12 , historical failings of science 13 , an alleged "reproducibility crisis" 14 , conspiracy theories 15,16 , and science-related populist attitudes 17,18 . ...
Scientific information is crucial for evidence-based decision-making. Public trust in science can help decision-makers act based on the best available evidence, especially during crises such as climate change or the COVID-19 pandemic. However, in recent years the epistemic authority of science has been challenged, causing concerns about low public trust in scientists. Here we interrogated these concerns with a pre-registered 67-country survey of 71,417 respondents on all inhabited continents and find that in most countries, a majority of the public trust scientists and think that scientists should be more engaged in policymaking. We further show that there is a discrepancy between the public’s perceived and desired priorities of scientific research. Moreover, we find variations between and within countries, which we explain with individual- and country-level variables, including political orientation. While these results do not show widespread lack of trust in scientists, we cannot discount the concern that lack of trust in scientists by even a small minority may affect considerations of scientific evidence in policymaking. These findings have implications for scientists and policymakers seeking to maintain and increase trust in scientists.
... H. Kennedy & Muzzerall, 2022;Rapp, 2016)-this means in perceptions of scientists' integrity and benevolence rather than their competence. Thus, information about scientific control processes (e.g., peer review, preregistrations, open materials) which make it less likely for scientists to bias their results in a desired (e.g., liberal) direction may increase trust (especially morality-based trust; Altenmüller et al., 2021;Hendriks et al., 2020;Rosman et al., 2022) among conservatives (i.e., reduce political polarization, Van Bavel et al., 2020). ...
Trust in science is polarized along political lines—but why? We show across a series of highly controlled studies (total N = 2,859) and a large-scale Twitter analysis ( N = 3,977,868) that people across the political spectrum hold stereotypes about scientists’ political orientation (e.g., “scientists are liberal”) and that these stereotypes decisively affect the link between their own political orientation and their trust in scientists. Critically, this effect shaped participants’ perceptions of the value of science, protective behavior intentions during a pandemic, policy support, and information-seeking behavior. Therefore, these insights have important implications for effective science communication.
... As Paul Gertler and colleagues report in Nature, in economics and other social sciences, replication creates antagonism, "acrimonious debate," and often "adversarial relationships" between original and replication researchers (Gertler, Galiani, and Romero 2018: 418). At the same time, to the degree that cases of replication failure are publicized more than cases of success, replication contributes to decreased public trust in scientists and their research (Hendricks, Kienhues, and Bromme 2020). Truth-verifying replication is especially antagonistic in its posture and, therefore, particularly prone to generating conflict (Peterson and Panofsky 2021). ...
This article examines how and to what effect the scientific ideal and practice of replication is adopted by a Mexican federal government agency charged with measuring poverty. The commitment to replication among state poverty experts is traced to their self-conception as democratic reformers working against cultures of state opacity associated with an authoritarian past. These experts deploy the ideal of replication as a bureaucratic ethos and the practice of replication as a public-facing and legitimating strategy. Replication successfully performs transparency and generates trust by appealing to, and strengthening ties with, elite academic and policy actors. Ultimately, the article shows how scientific ideals and practices adopted by state actors are cast as democracy enhancing even as they produce elite closure and limit public participation.
... This combination of heterogeneity and homogeneity gives rise to the concern of replicability of the existing experimental findings (Nosek et al., 2022). Notably, replicability is the foundation of building public trust in scientific evidence and scientists (Hendriks et al., 2020). With this concern, we call for more replications of the effect of exposure to climate change conspiracy theories. ...
Research on conspiracy theories and climate change is emerging. Existing studies came from a range of scholarly disciplines, asked different research questions, and used various research methods. Given the heterogeneity in these studies, there is a pressing need to synthesize the current state of knowledge on the topic and identify the priority directions for further research. We address this need with a preregistered systematic review. Our review covered 43 studies from 38 articles. We organized our review of these studies by the following methodological categories: survey studies, experiments, interview studies, media studies, ethnographic studies, and mathematically modelling. For each category, we offered a summary of evidence, highlighted the key insights, and identify knowledge gaps. We concluded with a proposed integrative framework for research on the topic. As conspiracy theories can be an obstacle to climate change mitigation and adaptation, the present review would provide invaluable insights for not only scientists but also climate-related practitioners (e.g., environmental organizations, educators) and decision makers (e.g., policymakers) to consider.
... As Fetterman and Sassenberg (2015) contend, the replication crisis is bound to have negative reputational effects on science. Recently Hendriks et al. (2020) showed that study credibility and researcher trustworthiness increase significantly if a study was successfully replicated and decreases otherwise (cf. also Mede et al., 2020). ...
The social sciences are facing numerous crises including those related to replication, theory, and applicability. We highlight that these crises imply epistemic malfunctions and affect science communication negatively. Several potential solutions have already been proposed, ranging from statistical improvements to changes in norms of scientific conduct. In this paper, we propose a structural solution: the elimination of the discussion section from social science research papers. We point out that discussion sections allow for an inappropriate narrativization of research that disguises actual results and enables the misstatement of true limitations. We go on to claim that removing this section and outsourcing it to other publications provides several epistemic advantages such as a division of academic labour, adversarial modes of progress, and a better alignment of the personal aims of scientists with the aims of science. After responding to several objections, we conclude that the potential benefits of moving away from the traditional model of academic papers outweigh the costs and have the potential to play a part in addressing the crises in the social sciences alongside other reforms. As such, we take our paper as proffering a further potential solution that should be applied complimentarily with other reform movements such as Open Science and hope that our paper can start a debate on this or similar proposals.
... While estimations of replicability may vary, they nevertheless appear to be suboptimal-an issue that is not exclusive to psychology and found across many other disciplines (e.g., animal behaviour 8-10 ; cancer biology 11 ; economics 12 ), and symptomatic of persistent issues within the research environment 13,14 . The 'replication crisis' has introduced a number of considerable challenges, including compromising the public's trust in science 15 and undermining the role of science and scientists as reliable sources to inform evidencebased policy and practice 16 . At the same time, the crisis has provided a unique opportunity for scientific development and reform. ...
The emergence of large-scale replication projects yielding successful rates substantially lower than expected caused the behavioural, cognitive, and social sciences to experience a so-called ‘replication crisis’. In this Perspective, we reframe this ‘crisis’ through the lens of a credibility revolution, focusing on positive structural, procedural and community-driven changes. Second, we outline a path to expand ongoing advances and improvements. The credibility revolution has been an impetus to several substantive changes which will have a positive, long-term impact on our research environment.
... This experimental study indicates that the risk of strengthening, rather than weakening, belief in misinformation due to repeating myths in refutation attempts is low, at least as long as an explanation for why the claim is false is included (Swire- . Instead of focusing on the text structure and headline format, writers may need to pay attention to the text being comprehensive, trustworthy and persuasive to the reader for maximum effectiveness, for example, by avoiding aggressive language (König & Jucks, 2019) and highlighting scientific consensus and replication success if applicable (Bode et al., 2021;Hendriks et al., 2020). ...
Objectives:
Misinformation is a crucial problem, particularly online, and the success of debunking messages has so far been limited. In this study, we experimentally test how debunking text structure (truth sandwich vs. bottom-heavy) and headline format (statement vs. questions) affect the belief in misinformation across topics of the safety of COVID vaccines and GMO foods.
Design:
Experimental online study.
Methods:
A representative German sample of 4906 participants were randomly assigned to reading one of eight debunking messages in the experimentally varied formats and subsequently rated the acceptance of this message and the agreement to misinformation statements about the mentioned topics and an unrefuted control myth.
Results:
While the debunking messages specifically decreased the belief in the targeted myth, these beliefs and the acceptance of the debunking message were unaffected by the text structures and headline formats. Yet, they were less successful when addressing individuals with strong pre-existing, incongruent attitudes and distrust in science.
Conclusions:
The risk of backfire effects in debunking misinformation is low. Text structure and headline format are of relatively little importance for the effectiveness of debunking messages. Instead, writers may need to pay attention to the text being comprehensive, trustworthy and persuasive to maximize effectiveness.
... Special attention is paid to the issues of trust in the results and the choice of research topics [21,22,23]. In a number of case-study works by Russian authors the aspects of improving the quality of training of mining engineers related to their interaction with industrial organizations through scientific research are also considered [24,25,26]. ...
The article considers approaches to the development of criteria for assessing the quality of scientific research in St. Petersburg Mining University, Russia (Mining University), the functioning of which contains the main features of research university as well as of a scientific organization. The research methods include expert assessments, analysis and synthesis and other desk research methods. Based on the analysis of the results of the scientific activity of the organization in recent years, a set of basic quantitative indicators has been developed, as well as a set of additional qualitative indicators of the effectiveness of the scientific departments of the university. Qualitative indicators, including the rating assessment of the research carried out by the Scientific and Technical Council of the university, are designed to complement the widely used scientometric data. Thus, scientometric, financial indicators and a comprehensive (multifactorial) qualitative assessment can serve as criteria for the quality assessment for the scientific activity of the university. The information obtained is relevant for the formation of a quality management system for scientific research at Mining University, as well as for other research universities with a similar organizational structure. However, the conclusions obtained cannot be automatically extended to the procedures for the formation of quality management systems for scientific research in organizations of a different profile.
... Second, our results highlight the relevance of morality-related aspects (e.g., perceived bias) for polarized trust: Here, the effects were stronger and more robust on morality-based trust than on expertise-based trust (studies 1, 4a, and 4b), which hints at polarization of trust in science being particularly grounded in aspects of morality (26,27), in this context, this means in scientists' perceived integrity and benevolence rather than their competence. In this regard, scientific control processes (e.g., peer review, preregistrations, open materials) make it less likely for scientists to bias their results in a desired (e.g., liberal) direction, and information about these processes may thus increase trust, especially morality-based trust (28)(29)(30), among conservatives (i.e., reduce political polarization, 31). Third, it could be promising to directly reduce people's reliance on stereotypes when judging scientists' trustworthiness by providing individuating information (21) or even engaging in trust-building interactions with scientists (32). ...
Public trust in science is polarized along political lines. Conservatives trust scientists and their findings less than do liberals – but why? Here, we show that people across the political spectrum hold stereotypes about scientists' political orientation (e.g., "scientists are liberal"), compare that to their own political orientation, and only trust scientists to the degree that they perceive them to be ideologically similar. Thus, we predict that the link between political orientation and trust in scientists is affected by political stereotypes about scientists. We tested this hypothesis in five studies in Germany and the US and combined a series of highly controlled experimental and correlational studies (total N = 2,859) with a large-scale analysis of Twitter data (N = 3,977,868). We experimentally manipulated scientists' perceived political orientation (study 1) or used naturally occurring variations in political stereotypes about scientific disciplines (e.g., "sociologists are liberal", studies 2 to 5). Across all studies, we consistently observed that stereotypes about scientists explain the link between political orientation and trust in scientists. Results showed that conservatives’ distrust in scientists is substantially reduced for stereotypically moderate disciplines (e.g., economists) and even reversed for scientists perceived as conservative. Confirming the critical consequences of this finding, the effect shaped participants' perceptions of the value of science, their protective behavior intentions during a pandemic, policy support, and information-seeking behavior.
... The current status quo tends to create in high school graduates: (a) a susceptibility to misinformation concerning the nature and social aspects of science (Duschl, 2022), (b) a lack of knowing how to figure out who and whose data to trust (Hendriks et al., 2020;Suldovsky et al., 2019), and (c) an incompetence to challenge scienceirrational and misinformed people who they may encounter daily. ...
This article establishes a rational, feasible, and necessary conclusion to reform high school science content into an equitable experience for its wide diversity of students' self‐identities. Research indicates that 85% of graduates would not normally have enrolled in any science course unless required. Their values are more aligned with their everyday world and/or the world of the humanities, to varying degrees. The 15% had already fulfilled their science prerequisite for postsecondary science‐related programs, to varying degrees. The article's conclusion rests mainly on historical and economic evidence, respectively: (1) The Sputnik crisis that instilled public fear and anxiety about the perceived technological gap between the United States and Soviet Union. This led to reforming high school science and implementing National Aeronautics and Space Administration. (2) The on‐growing climate‐change crisis for which the smart international money is increasingly investing in sustainable businesses and industries, which catalyze a shift in public values from the current “profit society” to a “sustainable society.” The article's rationale connects the two historical events. Over the past 30 years, the nature of normal science has evolved into post‐normal science. Today the public square also includes: (a) an international assessment project that receives a negative validity audit in this article; (b) a vocal small minority within the 85%, proud of their antiscience self‐identities and their leaders' hostile behavior (a problem to ameliorate by a reformed sustainable science education); and (c) instances of small‐scale, suitable reform examples developed over the last 70 years, often referred to as humanistic school science.
... Additionally, a confirmatory replication study would be necessary, given that our p-values are close to .05 which increases the danger of false-positive significant findings in our work (Benjamin et al., 2017). Thus, a successful replication study could increase the trustworthiness of our findings (Anvari & Lakens, 2018;Hendriks, Kienhues, & Bromme, 2020;Wingen, Berkessel, & Englich, 2020). Moreover, different samples and methods would be needed before generalizing our findings and drawing strong conclusions or advising policy. ...
How do media reports about the Covid-19 pandemic influence our mood? Building on the social comparison theory, we predicted that reading negative news affecting a similar group would result in an impaired mood. In contrast, reading negative news about a dissimilar group should lead to an improved mood. To test this, 150 undergraduate students read positive or negative news about the well-being of a similar or dissimilar group during the pandemic. As predicted, a mood assimilation effect occurred for similar groups, whilst a contrast effect occurred for a dissimilar group. The findings suggest that media reports can have a strong impact on mood. The direction of these effects, however, seems to depend strongly on social comparison processes.
... The more that participants viewed the issue to be uncertain, the less they perceived the source of information as trustworthy (Research Question 2b), which could be attributed to their more differentiated understanding of scientific knowledge (Hendriks et al., 2020). The nature of a moderating effect of epistemic beliefs on source evaluation could be further explored in the future. ...
Public and private decision-making on health problems relies on scientific evidence. However, scientific knowledge includes uncertainty, as does knowledge about COVID-19. In an experimental study, we tested how the trustworthiness (on the three dimensions expertise, integrity, and benevolence) of a source of information (either a scientist or a politician), was affected when messages were either two-sided (including arguments pro and contra the effectiveness of mask-wearing) or one-sided (only pro arguments). Results showed that scientists were ascribed more expertise and integrity compared to politicians, and both sources were ascribed more expertise when they gave two-sided (instead of one-sided) information. Moreover, trustworthiness ratings on all three dimensions were affected by participants’ prior topic attitudes and epistemic certainty beliefs. These findings underline that when a source provides two-sided information, this may increase people’s willingness to trust that source. To use this strategy most effectively in health communication, more research should be done on how many and what types of counterarguments to include.
... These effects appear to be relatively persistent: Neither explanations of irreproducibility reasons and recent transparency efforts (Wingen et al., 2019) nor information on causes, consequences, and resolutions of the "replication crisis" (Chopik et al., 2018) could stop people from losing trust in science. Further research suggests, however, that admitting the wrongness of an irreproducible study (Fetterman and Sassenberg, 2015), responding to a failed replication by conducting a follow-up study (Ebersole et al., 2016), or successful replication (Hendriks et al., 2020) may compensate or alleviate reputational damage. Moreover, learning about replication failure does not appear to affect individuals' trust in future research (Anvari and Lakens, 2018). ...
Populist and anti-intellectual sentiments pose a considerable challenge to science and science communication in many countries worldwide. One proliferating variant of such sentiments can be conceived as science-related populism. Science-related populism criticizes that scientists, scholars, and experts supposedly determine how society produces ‘true knowledge’ and communicates about it, because they are seen as members of an academic elite which allegedly applies unreliable methods, is ideologically biased – and ignores that the common sense of ordinary people ought to be superior to scientific knowledge. Accordingly, science-related populism assumes that the ordinary people, and not academic elites, should be in charge for the production and communication of ‘true knowledge’. Scholarly and journalistic accounts suggested that science-related populism can have negative implications for the legitimacy of scientific expertise in society and societal discourse about science. However, there has been neither a conceptual framework nor empirical methods and evidence to evaluate these accounts. This cumulative dissertation addresses this deficit: It includes five articles that present a conceptualization of science-related populism (Article I), a survey scale to measure science-related populist attitudes (Article II), empirical findings on these attitudes and related perceptions (Article II, Article III, and Article IV), and a discussion of populist demands toward science communication (Article V). The synopsis scrutinizes the arguments and results published in these articles in three ways: First, it discusses further theoretical considerations on science-related populism, advantages and challenges of its measurement, and broader contexts of empirical evidence on it. Second, it describes implications of science-related populism for communication and discourse about science, and proposes ways in which these implications can be addressed in science communication practice. Third, it considers how scholarship of science-related populism can advance social-scientific research on populism and anti-scientific resentments and could develop in the future.
... Другие пишут о растущем скептицизме по отношению к ученым, производящим научные результаты и рекомендации, об общественном непонимании и отрицании научных разработок (например, вакцин или ГМО) и проблем (таких как изменение климата) [Kabat, 2017]. О проблеме доверия также говорят в контексте восприятия исследовательских практик и внутренних кризисов науки, связанных, например, с академическим мошенничеством [Hendriks et al., 2016] или проблемой воспроизводимости научных результатов [Hendriks, Kienhues, Bromme, 2020], что может приводить к сомнению относительно достоверности и убедительности научных результатов для широкой общественности. ...
Категория доверия имеет долгую историю в области анализа науки. При этом внимание к эмпирическому исследованию общественного доверия науке стало особенно возрастать в последние годы. Эта популярность связана как с объявленным учеными кризисом доверия, так и с ролью доверия в жизни людей. Используя систематический отбор литературы, автор предлагает обзор существующих исследований факторов общественного доверия науке. В качестве ключевых вопросов рассматриваются используемые в работах концепции доверия науке, способы его измерения и основные категории выявляемых факторов. В результате показывается вариативность как стратегий исследования общественного доверия, так и формирующих его факторов. Они могут быть использованы для повышения эффективности научной коммуникации и для разработки исследовательских перспектив.
... It is, however, important to note that trust in science is probably not purely affective and relates also to cognitive beliefs about scientists' competence (Altenmüller et al., 2021;Hendriks et al., 2015). The cognitive component of attitudes towards science may further include beliefs about the value of science for solving problems (Broomell & Kane, 2017;Hilgard & Jamieson, 2017), evaluations of scientific uncertainty and complexity (Rabinovich & Morton, 2012;van der Bles et al., 2019), or even views about rather specific scientific methods and controversies, such as replicability or publication modes (Hendriks et al., 2020;Wingen et al., 2022). Finally, the behavioural component of attitudes towards science may include seeking information about new developments in science and research (Scharrer et al., 2014(Scharrer et al., , 2017Sweeny et al., 2010). ...
A better understanding of the public attitude towards science could be crucial to tackle the spread of mis- and disinformation related to the Covid-19 pandemic and beyond. We here contribute to this understanding by conceptualising and analysing the attitude toward science as a psychological network. For this analysis, we utilised cross-sectional data from a German probability sample (N = 1,009), the “Science Barometer”, collected during the first wave of the Covid-19 pandemic. Overall, our network analysis revealed that especially the perceived value of science for curbing the pandemic is central to the attitude towards science. Beliefs about this value are related to trust in science and trust in scientific information and to positive and negative evaluations of scientific controversy and complexity. Further, valuing common sense over science was related to seeking less scientific information on official websites, suggesting that this belief, in particular, may drive mis- and disinformation and could be a promising target for interventions. Finally, we found no evidence that seeking scientific information on social media had detrimental consequences for the attitude towards science. Implications for health communication and science communication, limitations, and future directions are discussed.
... This, in turn, reduces the chances of obtaining false-positive results (Wicherts et al., 2016), thus increasing replicability rates (Munafò, 2016;Munafò et al., 2017). Given that replications significantly contribute to trust in psychological theories (van den Akker et al., 2018) and that low replicability has been shown to impair public trust in science (Anvari and Lakens, 2018;Chopik et al., 2018;Hendriks et al., 2020;Wingen et al., 2020), increased trust is thus a likely result of OSPs. It should, however, be noted that some of these assumptions cannot be tested empirically, and that at least one study found that low replicability does not have much of a detrimental effect on public perceptions of science (Mede et al., 2021). ...
In two studies, we examined whether open science practices, such as making materials, data, and code of a study openly accessible, positively affect public trust in science. Furthermore, we investigated whether the potential trust-damaging effects of research being funded privately (e.g. by a commercial enterprise) may be buffered by such practices. After preregistering six hypotheses, we conducted a survey study (Study 1; N = 504) and an experimental study (Study 2; N = 588) in two German general population samples. In both studies, we found evidence for the positive effects of open science practices on trust, though it should be noted that in Study 2, results were more inconsistent. We did not however find evidence for the aforementioned buffering effect. We conclude that while open science practices may contribute to increasing trust in science, the importance of making use of open science practices visible should not be underestimated.
A psychology article’s p values say a lot about how its studies were conducted and whether its results are likely to replicate. Examining p values across the entire literature can, in turn, shed light on the state of psychology overall and how it has changed since the start of the replication crisis. In the present research, I investigate strong ( p < .01) and weak (.01 ≤ p < .05) p values reported across 240,355 empirical psychology articles from 2004 to 2024. Over this period and across every subdiscipline, the typical study has begun reporting markedly stronger p values. Nowadays, articles reporting strong p values are also more often published in top journals and receive more citations. Yet it also appears that robust research is still not correspondingly linked to career success given that researchers at the highest ranked universities tend to publish articles with the weakest p values. Investigating language usage suggests that two-thirds of this association can be explained by highly ranked universities preferring laborious, expensive, and subtle research topics even though these generally produce weaker results. Altogether, these findings point to the strength of most contemporary psychological research and suggest academic incentives have begun to promote such research. However, there remain key questions about the extent to which robustness is truly valued compared with other research aspects.
This chapter will introduce readers to the field of industrial/organizational psychology (IO) with a focus on understanding human behaviors within an organizational context. It will examine the intersection between IO psychology and applied psychology and how they impact our understanding of human and organizational behavior. IO psychology applies psychological principles to workplace settings, focusing on five key areas: assessment and selection, research on technologies, talent development, motivation, and process facilitation. Research methods like surveys, interviews, and observation help address workplace performance challenges. Contemporary issues include ethics, confidentiality, conflicts of interest, workplace violence, and socially responsible practices. Becoming an IO practitioner involves education, licensure, and ongoing development. Each area matters as it enhances workplace effectiveness, employee well-being, and organizational success.
In a time where new research methods are constantly being developed and science is evolving, researchers must continually educate themselves on cutting-edge methods and best practices related to their field. The second of three volumes, this Handbook provides comprehensive and up-to-date coverage of a variety of issues important in developing, designing, and collecting data to produce high-quality research efforts. First, leading scholars from around the world provide an in depth explanation of various advanced methodological techniques. In section two, chapters cover general important methodological considerations across all types of data collection. In the third section, the chapters cover self-report and behavioral measures and their considerations for use. In the fourth section, various psychological measures are covered. The final section of the handbook covers issues that directly concern qualitative data collection approaches. Throughout the book, examples and real-world research efforts from dozens of different disciplines are discussed.
Academics have traditionally played a vital role in both the generation and dissemination of knowledge, ideas and narratives. Social media, relative to traditional media, provides for new and more direct ways of science communication. Yet, since not all academics may engage with social media, the sample that does so may have an outsize influence on shaping public perceptions of academia more broadly through at least two channels: the set topics they engage with and through the particular style and tone of communication. This paper describes patterns in academics' expression online found in a newly constructed global dataset covering over 100,000 scholars linking their social media content to academic record. We document large and systematic variation in politically salient academic expression concerning climate action, cultural, and economic concepts. We show that these appear to often diverge from general public opinion in both topic focus and style.
People often learn of new scientific findings from brief news reports, and may discount or ignore prior research, potentially contributing to misunderstanding of findings. In this preregistered study, we investigated how people interpret a brief news report on a new drug for weight loss. Participants read an article that either highlighted the importance of prior research when judging the drug’s effectiveness, or made no mention of this issue. For articles describing no prior research, mean confidence in the drug was 62%. For articles that noted prior research was conducted, confidence increased as the proportion of studies with positive findings increased. When prior research was highlighted, confidence decreased by a small amount, even when it should have increased (i.e., even when most of the evidence supported the drug’s effectiveness). Thus, people’s judgements were more skeptical, but not necessarily more accurate. Judgments were not affected by education level, statistics experience, or personal relevance of the research topic.
Communicating research findings to the public in a clear but engaging manner is challenging, yet central for maximizing their societal impact. This systematic review aimed to derive evidence-based strategies for science communication from experimental studies. Three databases were searched in December 2022. Experimental studies published in English or German were included if they tested the effect of providing written information about science to adults aged 16+ years by assessing the impact on at least one of four domains of science communication aims (understanding and knowledge, attitudes and trust, intention and behavior, engagement). A total of 171 publications were included. Derived strategies include avoiding jargon, carefully structuring texts, including citations and expert sources, being mindful about how and when to indicate conflict or uncertainty in science, using neutral language, and highlighting Open Science principles and replicability. They can be used to communicate science effectively to lay audiences, benefitting the society.
Prior research indicates that income relates to trust in science. However, no prior studies exclusively focus on this relationship, leaving questions on the characteristics and universality of the relationship unanswered. This study enriches our understanding of the relationship between individual-level income and trust in science on 3 fronts. First, this study explicates income into the dimensions of relative income (objective economic status) and subjective income (perceived economic hardship and satisfaction). Second, it provides a global overview by assessing the aforementioned relationship across 145 countries, investigating whether the relationship is universal or contingent on country-level characteristics. Third, the study investigates moderators at country and individual-level. Results indicate that subjective income is more strongly related to trust in science than relative income and that it is strongest in previously studied populations. The relationship is moderated by institutional quality (which increases the relationship) and economic climate (GDP per capita increasing the relationship) at the country-level, and science efficacy (increasing the relationship) and trust in government (decreasing the relationship) at the individual-level.
Die Coronakrise hat ein Schlaglicht auf das Verhältnis von Politik und Wissenschaft geworfen. Besonders die Praxis der wissenscha lichen Politikberatung ist dabei in den Blick der Öffentlichkeit gerückt und hat kritische Fragen aufgeworfen. Welche Rolle sollten Wissenschaler:innen bei einer akuten Gefährdung der öffentlichen Gesundheit spielen? Welche wissenschaftlichen Disziplinen müssen in die Politikberatung eingebunden werden? Was kann und soll das häufig geäußerte Mantra "Follow the Science" eigentlich bedeuten? Inwieweit können politische Entscheidungen durch wissenschaftliches Wissen legitimiert werden? Fragen wie diese sind nicht zuletzt mit Blick auf radikale Einschränkungen des öffentlichen Lebens und der individuellen Lebensgestaltung in 2020/2021 virulent geworden. Vor diesem Hintergrund wurde von uns im Sommer 2021 ein Workshop mit dem Titel "Wissenschaft und Politik in der Pandemie: Lektionen der COVID-19 Krise" am Institut für Medizingeschichte und Wissenschaftsforschung der Universität zu Lübeck veranstaltet. Das vorliegende Logbuch vermittelt in Form von Kurzzusammenfassungen ausgewählter Beiträge und bereichert durch das Graphic Recording der Veranstaltung einen Eindruck von den Diskussionen des Workshops und soll dadurch zur Weiterbeschäftigung insbesondere mit Themen einladen, die sich durch verschiedene Beiträge ziehen. Damit sind zum einen Fragen zum richtigen Umgang mit einer Pluralität von Werten, Evidenzen und Wissenstypen in der Politikberatung sowie zur Rollendiffusion von Wissenschafler:innen angesprochen. Zum anderen ziehen sich Analysen von polarisierenden Diskussionen in (sozialen) Medien und deren Einfluss auf die Wahrnehmung und Gestaltung wissenschaftlicher Politikberatung durch eine Reihe der Workshopbeiträge.... weniger
Communicating research findings to the public in a clear but engaging manner is challenging, yet central for maximizing their societal impact. This systematic review aimed to derive evidence-based strategies for science communication from experimental studies. Three databases were searched in December 2022. Experimental studies published in English or German were included if they tested the effect of providing written information about science to adults aged 16+ years by assessing the impact on at least one of four domains of science communication aims (understanding and knowledge, attitudes and trust, intention and behavior, engagement). A total of 171 studies were included. Derived strategies include avoiding jargon, carefully structuring texts, including citations and expert sources, being mindful about how and when to indicate conflict or uncertainty in science, using neutral language, and highlighting Open Science principles and replicability. They can be used to communicate science effectively to lay audiences, benefitting society.
In recent years, the veracity of scientific findings has come under intense scrutiny in what has been called the “replication crisis.” This crisis is marked by the propagation of scientific claims which were subsequently contested, found to be exaggerated, or deemed false. This article describes the replication crisis and identifies examples of unreproducible results and irreplicable findings from across the biomedical and social sciences. Purported causes and potential remedies to the crisis are examined. It is argued that social work research suffers from the many analytic and methodological vices described here and that the profession is likely in crisis itself. Consequences for the discipline, as both a research and practice-based profession, are explored and paths forward are proposed.
Science and personal experiences in some cases seem to be two different ways of knowledge justification. The current “post-truth” era is characterized by a rise of personal beliefs and justifications. In order to address these phenomena from a perspective of beliefs, several constructs may be considered: Beliefs about the utility of science and of personal experiences, trust in science, and epistemic beliefs. Despite some research addressing each belief’s independent relation to information seeking behavior, we do not know much about the interrelationship of these beliefs. To address this research gap and to explore whether knowledge about how science works is related to these beliefs, a paper–pencil study with 315 university students of psychology, education, and teacher education was conducted. There was a high positive relationship of trust in science with justification-by-authority beliefs, and medium negative relationships of trust in science with uncertainty beliefs and personal-justification beliefs. Trust in science was positively related to the perceived utility of science. Epistemic beliefs were also related to utility beliefs. The number of methods courses taken and knowledge about how science works was related to trust in science and epistemic beliefs, but not to utility of science or utility of personal experiences. It is concluded that we should revisit our conceptualization of epistemic beliefs in the context of “post-truth”.
The emergence of large-scale replication projects yielding successful rates substantially lower than expected caused the behavioural, cognitive, and social sciences to experience a so-called ‘replication crisis’. In this Perspective, we reframe this ‘crisis’ through the lens of a credibility revolution, focusing on positive structural, procedural and community-driven changes. Second, we outline a path to expand ongoing advances and improvements. The credibility revolution has been an impetus to several substantive changes which will have a positive, long-term impact on our research environment.
Both within and outside of sociology, there are conversations about methods to reduce error and improve research quality—one such method is preregistration and its counterpart, registered reports. Preregistration is the process of detailing research questions, variables, analysis plans, etc. before conducting research. Registered reports take this one step further, with a paper being reviewed on the merit of these plans, not its findings. In this manuscript, I detail preregistration’s and registered reports’ strengths and weaknesses for improving the quality of sociological research. I conclude by considering the implications of a structural-level adoption of preregistration and registered reports. Importantly, I do not recommend that all sociologists use preregistration and registered reports for all studies. Rather, I discuss the potential benefits and genuine limitations of preregistration and registered reports for the individual sociologist and the discipline.
In the current psychological debate, low replicability of psychological findings is a central topic. While the discussion about the replication crisis has a huge impact on psychological research, we know less about how it impacts public trust in psychology. In this article, we examine whether low replicability damages public trust and how this damage can be repaired. Studies 1–3 provide correlational and experimental evidence that low replicability reduces public trust in psychology. Additionally, Studies 3–5 evaluate the effectiveness of commonly used trust-repair strategies such as information about increased transparency (Study 3), explanations for low replicability (Study 4), or recovered replicability (Study 5). We found no evidence that these strategies significantly repair trust. However, it remains possible that they have small but potentially meaningful effects, which could be detected with larger samples. Overall, our studies highlight the importance of replicability for public trust in psychology.
Current debates about the replication crisis in psychology take it for granted that direct replication is valuable, largely focusing on its role in uncovering questionable statistical practices. This paper takes a broader look at the notion of replication in psychological experiments. It is argued that all experimentation/replication involves individuation judgments and that research in experimental psychology frequently turns on probing the adequacy of such judgments. In this vein, I highlight the ubiquity of conceptual and material questions in research, arguing that replication has its place, but is not as central to psychological research as it is sometimes taken to be.
There is a robust scientific consensus concerning climate change and evolution. But many people reject these expert views, in favour of beliefs that are strongly at variance with the evidence. It is tempting to try to explain these beliefs by reference to ignorance or irrationality, but those who reject the expert view seem often to be no worse informed or any less rational than the majority of those who accept it. It is also tempting to try to explain these beliefs by reference to epistemic overconfidence. However, this kind of overconfidence is apparently ubiquitous, so by itself it cannot explain the difference between those who accept and those who reject expert views. Instead, I will suggest that the difference is in important part explained by differential patterns of epistemic deference, and these patterns, in turn, are explained by the cues that we use to filter testimony. We rely on cues of benevolence and competence to distinguish reliable from unreliable testifiers, but when debates become deeply politicized, asserting a claim may itself constitute signalling lack of reliability.
The present work examines the role of source vs. content cues for the confirmation bias, in which recipients spend more time with content aligning with preexisting attitudes. In addition to testing how both source and content cues facilitate this biased pattern of selective exposure, the study measures subsequent attitude polarization. An experiment (N = 120) presented messages with opposing political stances, associated with unbiased or slanted sources. Software tracked selective exposure in seconds, and attitudes were measured before, immediately after, and two days after message exposure. Further, information processing styles were assessed. The confirmation bias emerged regardless of source quality. Information processing styles moderated the confirmation bias as well as selective exposure to messages from unbiased vs. slanted sources. Selective exposure reinforced attitudes days later.
In this paper, we contend that what to teach about scientific reasoning has been bedeviled by a lack of clarity about the construct. Drawing on the insights emerging from a cognitive history of science, we argue for a conception of scientific reasoning based on six ‘styles of scientific reasoning’. Each ‘style’ requires its own specific ontological and procedural entities, and invokes its own epistemic values and constructs. Consequently, learning science requires the development of not just content knowledge but, in addition, procedural knowledge, and epistemic knowledge. Previous attempts to develop a coherent account of scientific reasoning have neglected the significance of either procedural knowledge, epistemic knowledge, or both. In contrast, ‘styles of reasoning’ do recognize the need for all three elements of domain-specific knowledge, and the complexity and situated nature of scientific practice. Most importantly, ‘styles of reasoning’ offer science education a means of valorizing the intellectual and cultural contribution that the sciences have made to contemporary thought, an argument that is sorely missing from common rationales for science education. Second, the construct of ‘styles of reasoning’ offers a more coherent conceptual schema for the contruct of scientific reasoning – one of the major goals of any education in the sciences.
Replication is vital for increasing precision and accuracy of scientific claims. However, when replications “succeed” or “fail,” they could have reputational consequences for the claim’s originators. Surveys of United States adults (N = 4,786), undergraduates (N = 428), and researchers (N = 313) showed that reputational assessments of scientists were based more on how they pursue knowledge and respond to replication evidence, not whether the initial results were true. When comparing one scientist that produced boring but certain results with another that produced exciting but uncertain results, opinion favored the former despite researchers’ belief in more rewards for the latter. Considering idealized views of scientific practices offers an opportunity to address incentives to reward both innovation and verification.
In contrast to the truncated view that replications have only a little to offer beyond what is already known, we suggest a broader understanding of replications: We argue that replications are better conceptualized as a process of conducting consecutive studies that increasingly consider alternative explanations, critical contingencies, and real-world relevance. To reflect this understanding, we collected and summarized the existing literature on replications and combined it into a comprehensive overall typology that simplifies and restructures existing approaches. The resulting typology depicts how multiple, hierarchically structured replication studies guide the integration of laboratory and field research and advance theory. It can be applied to (a) evaluate a theory's current status, (b) guide researchers' decisions, (c) analyze and argue for the necessity of certain types of replication studies, and (d) assess the added value of a replication study at a given state of knowledge. We conclude with practical recommendations for different protagonists in the field (e.g., authors, reviewers, editors, and funding agencies). Together, our comprehensive typology and the related recommendations will contribute to an enhanced replication culture in social psychology and to a stronger real-world impact of the discipline.
Scientists are dedicating more attention to replication efforts. While the scientific utility of replications is unquestionable, the impact of failed replication efforts and the discussions surrounding them deserve more attention. Specifically, the debates about failed replications on social media have led to worry, in some scientists, regarding reputation. In order to gain data-informed insights into these issues, we collected data from 281 published scientists. We assessed whether scientists overestimate the negative reputational effects of a failed replication in a scenario-based study. Second, we assessed the reputational consequences of admitting wrongness (versus not) as an original scientist of an effect that has failed to replicate. Our data suggests that scientists overestimate the negative reputational impact of a hypothetical failed replication effort. We also show that admitting wrongness about a non-replicated finding is less harmful to one's reputation than not admitting. Finally, we discovered a hint of evidence that feelings about the replication movement can be affected by whether replication efforts are aimed one's own work versus the work of another. Given these findings, we then present potential ways forward in these discussions.
Selective exposure research indicates that news consumers tend to seek out attitude-consistent information and avoid attitude-challenging information. This study examines online news credibility and cognitive dissonance as theoretical explanations for partisan selective exposure behavior. After viewing an attitudinally consistent, challenging, or politically balanced online news source, cognitive dissonance, credibility perceptions, and likelihood of selective exposure were measured. Results showed that people judge attitude-consistent and neutral news sources as more credible than attitude-challenging news sources, and although people experience slightly more cognitive dissonance when exposed to attitude-challenging news sources, overall dissonance levels were quite low. These results refute the cognitive dissonance explanation for selective exposure and suggest a new explanation that is based on credibility perceptions rather than psychological discomfort with attitude-challenging information.
Given their lack of background knowledge, laypeople require expert help when dealing with scientific information. To decide whose help is dependable, laypeople must judge an expert's epistemic trustworthiness in terms of competence, adherence to scientific standards, and good intentions. Online, this may be difficult due to the often limited and sometimes unreliable source information available. To measure laypeople's evaluations of experts (encountered online), we constructed an inventory to assess epistemic trustworthiness on the dimensions expertise, integrity, and benevolence. Exploratory (n = 237) and confirmatory factor analyses (n = 345) showed that the Muenster Epistemic Trustworthiness Inventory (METI) is composed of these three factors. A subsequent experimental study (n = 137) showed that all three dimensions of the METI are sensitive to variation in source characteristics. We propose using this inventory to measure assignments of epistemic trustworthiness, that is, all judgments laypeople make when deciding whether to place epistemic trust in-and defer to-an expert in order to solve a scientific informational problem that is beyond their understanding.
An attribution analysis of opinion change viewed message persuasiveness as a function of inferred communicator biases. Recipients infer a knowledge bias (KB) by believing that a communicator's knowledge about external reality is nonveridical and a reporting bias (RB) by believing that a communicator's willingness to convey an accurate version of external reality is compromised. In an experiment with 355 undergraduates, KB expectancies were established by portraying a communicator as having a strong commitment to values represented by the probusiness or proenvironment side of a controversial issue and RB expectancies by portraying his audience as having a strong commitment to one or the other side. In all conditions, the communicator advocated the proenvironment position. Therefore, recipients' expectancies were confirmed in the context of a proenvironment communicator and/or audience and disconfirmed in the context of a probusiness communicator and/or audience. Regardless of the type of bias that Ss expected, they were more persuaded and rated the communicator as more unbiased when their expectancies were disconfirmed. Confirmation of expectancies based on RB, but not KB, was associated with inferences of communicator insincerity and manipulativeness. (21 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Empirically analyzing empirical evidence
One of the central goals in any scientific endeavor is to understand causality. Experiments that seek to demonstrate a cause/effect relation most often manipulate the postulated causal factor. Aarts et al. describe the replication of 100 experiments reported in papers published in 2008 in three high-ranking psychology journals. Assessing whether the replication and the original experiment yielded the same result according to several criteria, they find that about one-third to one-half of the original findings were also observed in the replication study.
Science , this issue 10.1126/science.aac4716
More than 34 million people in the world are living with HIV/AIDS. Each year there are nearly three million new HIV infections and nearly two million AIDS-related deaths. In the face of these staggering statistics, there is a vocal group of individuals who deny reality and claim that HIV is harmless or may not even exist at all. Mirroring the AIDS pandemic itself, AIDS denialism is a threat to public health. AIDS denialism propagates the views of a few rogue scientists through press releases, social media, and a significant presence on the Internet. AIDS Denialists aim to undermine HIV testing, prevention, and treatment. Most tragic has been the adoption of AIDS Denialist views by public officials, perhaps none more infamous than former President of South Africa Thabo Mbeki. This article examines the psychology of AIDS denialism, focusing on its emergence from the earliest days of AIDS and its grounding in conspiracy thinking and medical mistrust. The article also describes the tactics and rhetoric of AIDS denialism and those who are most vulnerable to AIDS Denialist claims. Recommendations are offered for countering AIDS denialism by exposing its fake experts and baseless claims.
Effect sizes are the most important outcome of empirical studies. Most articles on effect sizes highlight their importance to communicate the practical significance of results. For scientists themselves, effect sizes are most useful because they facilitate cumulative science. Effect sizes can be used to determine the sample size for follow-up studies, or examining effects across studies. This article aims to provide a practical primer on how to calculate and report effect sizes for t-tests and ANOVA's such that effect sizes can be used in a-priori power analyses and meta-analyses. Whereas many articles about effect sizes focus on between-subjects designs and address within-subjects designs only briefly, I provide a detailed overview of the similarities and differences between within- and between-subjects designs. I suggest that some research questions in experimental psychology examine inherently intra-individual effects, which makes effect sizes that incorporate the correlation between measures the best summary of the results. Finally, a supplementary spreadsheet is provided to make it as easy as possible for researchers to incorporate effect size calculations into their workflow.
The widespread prevalence and persistence of misinformation in contemporary societies, such as the false belief that there is a link between childhood vaccinations and autism, is a matter of public concern. For example, the myths surrounding vaccinations, which prompted some parents to withhold immunization from their children, have led to a marked increase in vaccine-preventable disease, as well as unnecessary public expenditure on research and public-information campaigns aimed at rectifying the situation.
We first examine the mechanisms by which such misinformation is disseminated in society, both inadvertently and purposely. Misinformation can originate from rumors but also from works of fiction, governments and politicians, and vested interests. Moreover, changes in the media landscape, including the arrival of the Internet, have fundamentally influenced the ways in which information is communicated and misinformation is spread.
We next move to misinformation at the level of the individual, and review the cognitive factors that often render misinformation resistant to correction. We consider how people assess the truth of statements and what makes people believe certain things but not others. We look at people’s memory for misinformation and answer the questions of why retractions of misinformation are so ineffective in memory updating and why efforts to retract misinformation can even backfire and, ironically, increase misbelief. Though ideology and personal worldviews can be major obstacles for debiasing, there nonetheless are a number of effective techniques for reducing the impact of misinformation, and we pay special attention to these factors that aid in debiasing.
We conclude by providing specific recommendations for the debunking of misinformation. These recommendations pertain to the ways in which corrections should be designed, structured, and applied in order to maximize their impact. Grounded in cognitive psychological theory, these recommendations may help practitioners—including journalists, health professionals, educators, and science communicators—design effective misinformation retractions, educational tools, and public-information campaigns.
Experimenter bias occurs when scientists' hypotheses influence their results, even if involuntarily. Meta-analyses have suggested that in some domains, such as psychology, up to a third of the studies could be unreliable due to such biases. A series of experiments demonstrates that while people are aware of the possibility that scientists can be more biased when the conclusions of their experiments fit their initial hypotheses, they robustly fail to appreciate that they should also be more sceptical of such results. This is true even when participants read descriptions of studies that have been shown to be biased. Moreover, participants take other sources of bias-such as financial incentives-into account, showing that this bias neglect may be specific to theory-driven hypothesis testing. In combination with a common style of scientific reporting, bias neglect could lead the public to accept premature conclusions.
We examined whether epistemic beliefs predict students’ evaluation of documents. Undergraduates read two texts on climate change. Participants judged the trustworthiness of each text and then indicated the criteria for their rating. We found that readers who believe strongly in relying on personal interpretations rather than on authorities trusted both documents less and used the document’s content or their own opinion as criteria for judging trustworthiness. We also found that readers who believe that knowledge claims should be critically evaluated through logic and rules rated the science text as more trustworthy and used the criteria of their own opinion, author and content more than readers who believe in relying on their own experiences. These effects hold true while controlling for readers’ prior knowledge and text comprehensibility.
This chapter features the concept of ascribed epistemic authority offered as a unique perspective on source effects in social judgment. It assumes that both the self and external sources may be assigned different degrees of epistemic authority in different domains and that this determines the ways in which individuals process information, make decisions, and undertake actions. The present framework traces the socio-developmental aspects of epistemic authority assignments and considers individual differences in the distribution of authority assignments across sources. The chapter conceives of epistemic authority ascriptions as meta-cognitive beliefs about a source of information. It introduces a perspective on source effects framed from the subjective standpoint of the information's recipient. This perspective highlights the developmental, individual differences, self-related, and applied aspects of source phenomena. The treatment of source effects in several major models of persuasion is reviewed. A final discussion highlights the unique properties of the epistemic authority and considers its implications for the place of source effects in notions of information processing and human judgment.
Replication is one of the most important tools for the verification of facts within the empirical sciences. A detailed examination of the notion of replication reveals that there are many different meanings to this concept and the relevant procedures, but hardly any systematic literature. This paper analyzes the concept of replication from a theoretical point of view. It demonstrates that the theoretical demands are scarcely met in everyday work within the social sciences. Some demands are just not feasible, whereas others are constricted by restrictions relating to publication. A new classification scheme based on a functional approach that distinguishes between different types of replication is proposed. Next, it will be argued that replication addresses the important connection between existing and new knowledge. To do so it has to be applied explicitly and systematically. The paper ends with a description of procedures how this could be done and a set of recommendations how to handle the concept of replication in the future to exploit its potential to the full. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Background
The aim of this study was to develop an instrument to measure laypeople’s beliefs about the nature of medical knowledge and knowing (the EBAM). Such beliefs should be a target of increased research interest because they influence how people handle medical information, for example in shared decision making.
Methods
An online survey was completed by 284 participants. Items assessed different aspects of laypeople’s epistemic beliefs about medicine and explicitly focused on the appearance of medical knowledge in everyday life and the evaluation of different sources as a way to justify knowledge.
Results
Factor analysis yielded a five-factor solution for the instrument. Dimensions covered by the instrument are certainty of medical knowledge, credibility of medical textbooks, credibility of medical information on the Internet, justification of medical knowledge, and preliminarity of medical knowledge.
Conclusions
Results indicate that laypeople have meaningful beliefs about the nature of medical knowledge and the trustworthiness of different sources. The instrument developed seems promising for measuring laypeople’s epistemic beliefs about medicine, which may help to increase patients’ compliance in medical decision making.
An academic scientist's professional success depends on publishing. Publishing norms emphasize novel, positive results. As such, disciplinary incentives encourage design, analysis, and reporting decisions that elicit positive results and ignore negative results. Prior reports demonstrate how these incentives inflate the rate of false effects in published science. When incentives favor novelty over replication, false results persist in the literature unchallenged, reducing efficiency in knowledge accumulation. Previous suggestions to address this problem are unlikely to be effective. For example, a journal of negative results publishes otherwise unpublishable reports. This enshrines the low status of the journal and its content. The persistence of false findings can be meliorated with strategies that make the fundamental but abstract accuracy motive-getting it right-competitive with the more tangible and concrete incentive-getting it published. This article develops strategies for improving scientific practices and knowledge accumulation that account for ordinary human motivations and biases.
© The Author(s) 2012.
C. Glenn Begley and Lee M. Ellis propose how methods, publications and
incentives must change if patients are to benefit.
There is ample evidence that estimations of what other people know are often biased in the direction of one's own knowledge. Yet, it is still unclear if this bias is influenced by expertise. In Study 1, computer experts estimated the distribution of Internet concepts and general knowledge concepts among students. These estimations were compared with norm values and with estimations obtained from a sample of laypersons. Laypersons showed a stronger bias than experts. Study 2 revealed that knowledge estimations can be influenced by labeling knowledge items as specialist knowledge. The results are suggestive of ways in which communication between experts and laypersons could be enhanced. Especially in asynchronous communication situations, as for example in the use of e-mail based hot lines, if experts are to communicate effectively, they must make accurate assumptions about laypersons' knowledge of the topic.
The editorial policies of several prominent educational and psychological journals require that researchers report some measure of effect size along with tests for statistical significance. In analysis of variance contexts, this requirement might be met by using eta squared or omega squared statistics. Current procedures for computing these measures of effect often do not consider the effect that design features of the study have on the size of these statistics. Because research-design features can have a large effect on the estimated proportion of explained variance, the use of partial eta or omega squared can be misleading. The present article provides formulas for computing generalized eta and omega squared statistics, which provide estimates of effect size that are comparable across a variety of research designs.
There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
Transparent communication of research is key to foster understanding within and beyond the scientific community. An increased focus on reporting effect sizes in addition of p-value based significance statements or Bayes Factors may improve scientific communication with the general public. Across three studies (N = 652), we compared subjective informativeness ratings for five effect sizes, Bayes Factor, and commonly used significance statements. Results showed that Cohen’s U3 was rated as most informative. For example, 440 participants (69%) found U3 more informative than Cohen’s d while 95 (15%) found d more informative than U3, with 99 participants (16%) finding both effect sizes equally informative. This effect was not moderated by level of education. We therefore suggest that in general Cohen’s U3 is used when scientific findings are communicated. However, the choice of the effect size may vary depending on what a researcher wants to highlight (e.g., differences or similarities).
Previous research suggests that when individuals encounter new information, they interpret it through perceptual ‘filters’ of prior beliefs, relevant social identities, and messenger credibility. In short, evaluations are not based solely on message accuracy, but also on the extent to which the message and messenger are amenable to the values of one’s social groups. Here, we use the release of Pope Francis’s 2015 encyclical as the context for a natural experiment to examine the role of prior values in climate change cognition. Based on our analysis of panel data collected before and after the encyclical’s release, we find that political ideology moderated views of papal credibility on climate change for those participants who were aware of the encyclical. We also find that, in some contexts, non-Catholics who were aware of the encyclical granted Pope Francis additional credibility compared to the non-Catholics who were unaware of it, yet Catholics granted the Pope high credibility regardless of encyclical awareness. Importantly, papal credibility mediated the conditional relationships between encyclical awareness and acceptance of the Pope’s messages on climate change. We conclude by discussing how our results provide insight into cognitive processing of new information about controversial issues.
We attempt to replicate 67 papers published in 13 well-regarded economics journals using author-provided replication files that include both data and code. Some journals in our sample require data and code replication files, and other journals do not require such files. Aside from 6 papers that use confidential data, we obtain data and code replication files for 29 of 35 papers (83%) that are required to provide such files as a condition of publication, compared to 11 of 26 papers (42%) that are not required to provide data and code replication files. We successfully replicate the key qualitative result of 22 of 67 papers (33%) without contacting the authors. Excluding the 6 papers that use confidential data and the 2 papers that use software we do not possess, we replicate 29 of 59 papers (49%) with assistance from the authors. Because we are able to replicate less than half of the papers in our sample even with help from the authors, we assert that economics research is usually not replicable. We conclude with recommendations on improving replication of economics research.
The experimental studies presented here investigated whether discussing ethical implications of preliminary scientific results in a science blog would impact blog readers’ perception of the responsible scientist blogger’s epistemic trustworthiness (on the dimensions expertise, integrity, and benevolence). They also investigated whether it made a difference in who had brought forward the ethics aspects: the responsible scientist blogger or another expert. Results indicate that by the mere introduction of ethics, people infer something about the blogger’s communicative intentions: Introducing ethical aspects seems to raise vigilance about an expert’s benevolence and integrity. Moreover, ratings of epistemic trustworthiness differed depending on who added ethical arguments: If ethics were introduced by the scientist blogger himself, his benevolence and integrity were rated higher than when ethics were introduced by another expert. These results are relevant for science bloggers, science communicators, and researchers who study laypeople’s understanding of epistemic uncertainty within science.
Another social science looks at itself
Experimental economists have joined the reproducibility discussion by replicating selected published experiments from two top-tier journals in economics. Camerer et al. found that two-thirds of the 18 studies examined yielded replicable estimates of effect size and direction. This proportion is somewhat lower than unaffiliated experts were willing to bet in an associated prediction market, but roughly in line with expectations from sample sizes and P values.
Science , this issue p. 1433
Reproducibility is the cornerstone of science. If an effect is reliable, any competent researcher should be able to obtain it when using the same procedures with adequate statistical power. Two of the articles in this special section question the value of direct replication by other laboratories. In this commentary, I discuss the problematic implications of some of their assumptions and argue that direct replication by multiple laboratories is the only way to verify the reliability of an effect.
© The Author(s) 2013.
This introduction to the special issue Understanding the Public Understanding of Science: Psychological Approaches discusses some of the challenges people face in understanding science. We focus on people's inevitably bounded understanding of science topics; research must address how people make decisions in science domains such as health and medicine without having the deep and extensive understanding that is characteristic of domain experts. The articles reflect two broad streams of research on the public understanding of science—the learning orientation that seeks to improve understanding through better instruction and the communications orientation that focuses on attitudes about science and trust in scientists. Challenges to understanding science include determining the relevance of information, the tentativeness of scientific truth, distinguishing between scientific and nonscientific issues, and determining what is true and what is false. Studying the public understanding of science can potentially contribute to psychological theories of thinking and reasoning in modern societies.
Networked digital media present new challenges for people to locate information that they can trust. At the same time, societal reliance on information that is available solely or primarily via the Internet is increasing. This article discusses how and why digitally networked communication environments alter traditional notions of trust, and presents research that examines how information consumers make judgments about the credibility and accuracy of information they encounter online. Based on this research, the article focuses on the use of cognitive heuristics in credibility evaluation. Findings from recent studies are used to illustrate the types of cognitive heuristics that information consumers employ when determining what sources and information to trust online. The article concludes with an agenda for future research that is needed to better understand the role and influence of cognitive heuristics in credibility evaluation in computer-mediated communication contexts.
Publication bias remains a controversial issue in psychological science. The tendency of psychological science to avoid publishing null results produces a situation that limits the replicability assumption of science, as replication cannot be meaningful without the potential acknowledgment of failed replications. We argue that the field often constructs arguments to block the publication and interpretation of null results and that null results may be further extinguished through questionable researcher practices. Given that science is dependent on the process of falsification, we argue that these problems reduce psychological science's capability to have a proper mechanism for theory falsification, thus resulting in the promulgation of numerous "undead" theories that are ideologically popular but have little basis in fact.
© The Author(s) 2012.
Replicability of findings is at the heart of any empirical science. The aim of this article is to move the current replicability debate in psychology towards concrete recommendations for improvement. We focus on research practices but also offer guidelines for reviewers, editors, journal management, teachers, granting institutions, and university promotion committees, highlighting some of the emerging and existing practical solutions that can facilitate implementation of these recommendations. The challenges for improving replicability in psychological science are systemic. Improvement can occur only if changes are made at many levels of practice, evaluation, and reward.
The research area of interpersonal expectancy effects originally derived from a general consideration of the effects of experimenters on the results of their research. One of these is the expectancy effect, the tendency for experimenters to obtain results they expect, not simply because they have correctly anticipated nature's response but rather because they have helped to shape that response through their expectations. When behavioral researchers expect certain results from their human (or animal) subjects they appear unwittingly to treat them in such a way as to increase the probability that they will respond as expected.
In the first few years of research on this problem of the interpersonal (or interorganism) self-fulfilling prophecy, the “prophet” was always an experimenter and the affected phenomenon was always the behavior of an experimental subject. In more recent years, however, the research has been extended from experimenters to teachers, employers, and therapists whose expectations for their pupils, employees, and patients might also come to serve as interpersonal self-fulfilling prophecies.
Our general purpose is to summarize the results of 345 experiments investigating interpersonal expectancy effects. These studies fall into eight broad categories of research: reaction time, inkblot tests, animal learning, laboratory interviews, psychophysical judgments, learning and ability, person perception, and everyday life situations. For the entire sample of studies, as well as for each specific research area, we (1) determine the overall probability that interpersonal expectancy effects do in fact occur, (2) estimate their average magnitude so as to evaluate their substantive and methodological importance, and (3) illustrate some methods that may be useful to others wishing to summarize quantitatively entire bodies of research (a practice that is, happily, on the increase).
In research on selective exposure to information, people have been found to predominantly seek information supporting rather than conflicting with their opinion. In most of these studies, participants were allowed to search for as many pieces of information as they liked. However, in many situations, the amount of information that people can search for is restricted. We report four experiments addressing this issue. Experiment 1 suggests that objective limits regarding the maximum number of pieces of information the participants could search for increases the preference for selecting supporting over conflicting information. In Experiment 2, just giving participants a cue about information scarcity induces the same effect, even in the absence of any objective restrictions. Finally, Experiment 3 and 4 clarify the underlying psychological process by showing that information limits increase selective exposure to information because information search is guided by the expected information quality, which is basically biased towards supporting information, and information limits act to reinforce this tendency. Copyright © 2005 John Wiley & Sons, Ltd.
News reports of scientific research are rarely hedged; in other words, the reports do not contain caveats, limitations, or other indicators of scientific uncertainty. Some have suggested that hedging may influence news consumers' perceptions of scientists' and journalists' credibility (perceptions that may be related to support for scientific research and/or adoption of scientific recommendations). But whether hedging does affect audience perceptions is unknown. A multiple-message experiment (N = 601) found that across five messages, both scientists and journalists were viewed as more trustworthy (a) when news coverage of cancer research was hedged (e.g., study limitations were reported) and (b) when the hedging was attributed to the scientists responsible for the research (as opposed to scientists unaffiliated with the research).
This article summarizes and comments upon recent research on epistemological beliefs (i.e., beliefs about knowledge and knowing). I identify four emergent themes, outline directions for future research, and draw links between current theory and educational practice. The four themes pertain to the number relationship among, development, and measurement of epistemological beliefs. I address conceptual and methodological issues in future research. I also identify four educational implications: understanding teachers' beliefs, understanding students' beliefs, promoting critical thinking, and attempting to change teachers' and students' beliefs.
This study explored the domain specificity of students' beliefs about academic knowledge in three related studies. Using a four-factor model as an initial framework, a series of domain-specific items about mathematics and history was developed. In Study I, these items were administered to 182 undergraduates, and the psychometric properties and underlying factor structure were examined via exploratory factor analysis. In Study II, the modified instrument, the Domain-Specific Beliefs Questionnaire (DSBQ), was administered to 633 students and a confirmatory factor analysis was conducted. A subsample of participants' responses on the DSBQ was also compared to responses on Schommer's epistemological questionnaire. Study III involved a second confirmatory factor analysis using data from a new sample (n = 523). Those data were examined for potential gender differences. Overall, the results of this investigation suggested that students possess certain domain-specific beliefs about knowledge in mathematics and history. Further, a significant moderate relationship between the DSBQ and Schommer's questionnaire provided some evidence of domain-generality in undergraduates' epistemological beliefs.
This article presents a semantic differential called Connotative Aspects of Epistemological Beliefs (CAEB) developed to assess university students' epistemological beliefs with adjective pairs such as dynamic–static and objective–subjective. After discussing the theoretical background, data are reported from two validation studies. The aim of the studies was to examine the emerging factor structure in different domains and to test whether CAEB can be used to measure domain-dependent differences in students' beliefs. Results showed a reasonable two-factor solution. The results further confirmed that the CAEB could measure differences in students' epistemological beliefs about knowledge in different domains of natural sciences like genetics, physics and plant identification.
1. This indicates the limitations of the predictivity of disease models and also that the validity of the targets being investigated is frequently questionable, which is a crucial issue to address if success rates in clinical trials are to be improved. Candidate drug targets in industry are derived from various sources, including inhouse target identification campaigns, inlicensing and public sourcing, in particular based on reports published in the literature and presented at conferences. During the transfer of projects from an academic to a company setting, the focus changes from ‘interesting’ to ‘feasible/marketable’, and the financial costs of pursuing a full-blown drug discovery and development programme for a particular target could ultimately be hundreds of millions of Euros. Even in the earlier stages, investments in activities such as high-throughput screening programmes are substantial, and thus the validity of published data on potential targets is crucial for companies when deciding to start novel projects. To mitigate some of the risks of such investments ultimately being wasted, most pharmaceutical companies run in-house target validation programmes. However, validation projects that were started in our company based on exciting published data have often resulted in disillusionment when key data could not be reproduced. Talking to scientists, both in academia and in industry, there seems to be a general impression that many
An open investigation of the reproducibility of cancer biology research
- T M Errington
- E Iorns
- W Gunn
- F E Tan
- J Lomax
- B A Nosek
Errington TM, Iorns E, Gunn W, Tan FE, Lomax J and Nosek BA (2014) An open investigation of the reproducibility of cancer biology research. Elife 3: e04333.
Believe it or not: How much can we rely on published data on potential drug targets?