Figure 1 - uploaded by Camilla McMahon
Content may be subject to copyright.
Responses to the survey questions on self-rated topical and methodological knowledge. The top bar represents the teams' answers about their knowledge regarding religion and well-being and the bottom bar represents the teams' answers about their knowledge regarding methodology and statistics. For each item, the number to the left of the data bar (in brown/orange) indicates the percentage of teams that reported little to no knowledge. The number in the center of the data bar (in grey) indicates the percentage of teams that were neutral. The number to the right of the data bar (in green/blue) indicates the percentage of teams that reported (some) expertize.

Responses to the survey questions on self-rated topical and methodological knowledge. The top bar represents the teams' answers about their knowledge regarding religion and well-being and the bottom bar represents the teams' answers about their knowledge regarding methodology and statistics. For each item, the number to the left of the data bar (in brown/orange) indicates the percentage of teams that reported little to no knowledge. The number in the center of the data bar (in grey) indicates the percentage of teams that were neutral. The number to the right of the data bar (in green/blue) indicates the percentage of teams that reported (some) expertize.

Source publication
Article
Full-text available
The relation between religiosity and well-being is one of the most researched topics in the psychology of religion, yet the directionality and robustness of the effect remains debated. Here, we adopted a many-analysts approach to assess the robustness of this relation based on a new cross-cultural dataset (N = 10, 535 participants from 24 countries...

Contexts in source publication

Context 1
... different career stages and domains of expertize featured in the analysis teams are given in Table 1. In addition, Figure 1 shows the self-rated collective knowledge about the topic of religion and well-being and about methodology and statistics. As becomes evident, most of the analysis teams had more methodological and/or statistical expertize than substantive expertize; 80% of the teams reported considerable expertize with regard to methods and statistics compared to 31% with regard to religion and well-being, 19% compared to 17% was neutral, and 3% compared to 50% reported little to no knowledge, respectively. ...
Context 2
... most teams indicated that effort was (very) high, the majority also reported that frustration was (very) low and that they spent as much time as anticipated (see Figure 10). That is, in stage 1, 55% of the teams reported (very) high effort, 17% were neutral, and 28% reported (very) low effort. ...
Context 3
... hypothesis was tested against the null-hypothesis that there is no relation between reported effect sizes and subjective beliefs. As the subjective beliefs were measured on a 7-point Figure 10. Responses to the survey questions about effort (top), frustration (middle), and workload (bottom). ...
Context 4
... further investigate changes in belief over the course of the project, we assessed the correlation between the reported effect sizes and the change in belief (i.e., the difference between posterior and prior beliefs for both research questions). For research question 1, there was basically no evidence for or against a positive relation between effect size and change in belief: BF +0 = 1.81, r s = 0.18, Figure 11. Reported effect sizes (beta coefficients) and subjective beliefs about the likelihood of the hypothesis. ...
Context 5
... the posterior beliefs, on the other hand, it seems that the teams updated their beliefs about the plausibility of research question 2 based on the results of their analyzes. Note, however, that based on the scatterplot in Figure 11(D), we should not put too much weight on this finding, as it may be partly driven by two outliers. For research question 1, the updating of beliefs may not have happened because prior beliefs about research question 1 were already in line with the outcomes, i.e., most teams expected and reported evidence for a positive relation between religiosity and well-being, with little variation between teams. ...
Context 6
... hypothesis was tested against the nullhypothesis that reported effect sizes and self-reported expertize were not related. For research question 1, we found moderate evidence against a correlation between effect sizes and methodological Figure 12. Reported effect sizes (beta coefficients) and self-reported team expertize. ...
Context 7
... credible interval [−0.24, 0.09]). See Figure 12 for scatterplots of the data. ...

Similar publications

Article
Full-text available
Human engagement with religion and spirituality is pervasive across the world, yet the extent to which religious and/or spiritual involvement promotes well-being is controversial theoretically and empirically. In the largest meta-analysis of religion/spirituality and life satisfaction to date (k = 256, N = 666,085), an overall effect size was compu...

Citations

... The impact of beliefs on well-being also depends on the cultural context in which people live. A study of 10,535 participants from 24 countries, recruiting 120 analysis teams, provided evidence that the effect of religious belief systems on well-being depends on the cultural norms of religiosity, which refer to the perceived importance of religious beliefs and behaviours for the average person within a given culture (Hoogeveen et al., 2023). In addition, research has found that there appears to be a relationship between spiritual beliefs and SWB in religious countries, but not in secular ones (Pérez & Rohde, 2022). ...
Article
The study explored the impact of religiosity and spirituality indicators on the subjective, social and psychological well-being of individuals aged 18 to 45 years both directly and indirectly through meaning in life. A total of 199 Bulgarians took part in the study (Mage = 27.58; SD = 8.71; 82.4% females). The instruments of the study were: Brief Mental Health Continuum Short, Brief Multidimensional Measure of Religiosity and Spirituality, Meaning in Life Questionnaire. Analyses showed that spiritual forgiveness predicted all domains of well-being and religious support predicted subjective well-being, all performing positive small effects. Meaning in life acted as full mediator in the relationship between daily spiritual experiences, religious support, forgiveness and all the domains of well-being. Meaning in life partially mediated the relationships between religious support and subjective well-being as well as between forgiveness and social well-being. The findings address that religiosity-based social support, daily transcendent experiences, and forgiving to oneself and the others promote well-being in early adulthood, and the underlying mechanism can be explained by meaning in life. Practical implications of the findings and directions for future research were discussed.
... A fragile result is not necessarily incorrect, but it is a risk to the generalisability and replicability of the observation (Nosek et al., 2022). Several "Many Analysts" studies have revealed that the decisions researchers make when analysing data can have a significant impact on observed results (Botvinik-Nezer et al., 2020;Hoogeveen et al., 2023;Silberzahn et al., 2018). These decisions could relate to, for example, the treatment of outliers, the selection of parametric or non-parametric statistical tests, or the inclusion of covariates in models. ...
Conference Paper
Full-text available
Previous research in technology education has illustrated the need to increase transparency in how research is reported in technology education to improve credibility and enhance the repeatability of published works. Following the identification of a replication crisis is several social scientific fields, it became increasingly clear that analytic decisions made by quantitative researchers can have significant effects on observed results. The fact that making seemingly minor or arbitrary decisions whilst analysing data can impact results so much leads to two problematic scenarios. First, researchers may unknowingly produce unrobust results which are observed only by virtue of the analytic decisions they made rather than results which reflect reality or a broader population. Second, researchers may search through their data for a sequence of analytic decisions which provide a desirable result and report this without describing the entirety of the analytic process such that readers are unaware that the result is not robust. In light of these possibilities, this work is presented under the assumption that if either of the above two scenarios is occurring in technology education research that it is the first. By using a multiverse analysis to reanalyse data from a previously published study, the robustness of results pertaining to a specific hypothesis is illustrated. In a multiverse analysis, a researcher can specify a series of choices to be made in an analysis, such as which variables can be included or whether subgroups within the sample could be analysed separately, and then every possible scenario of decisions is examined. This process illustrates which results will be observed from particular analytic paths. Through this example, the multiverse analysis as a method for robustness checking is described and commentary is provided on the value of pre-registering research for increasing transparency within the quantitative technology education research literature.
... A large body of prior evidence suggests that religion has a generally positive impact on mental health and well-being, including positive associations with life satisfaction (Lim & Putnam, 2010), optimism (Salsman et al., 2005), subjective happiness (Lewis & Cruise, 2006), reduced anxiety and depression (Hayward & Krause, 2014), and physical health (Seybold & Hill, 2001). Although typically small in size, this effect is consistently observed in both cross-sectional (Hoogeveen et al., 2023;Koenig & Larson, 2001) and longitudinal research (Garssen et al., 2021). Moreover, the positive link between religion and well-being has been replicated in different cultural contexts (Cohen & Johnson, 2017;Hoogeveen et al., 2023), even though several studies on person-culture fit show that the existence of this relationship and its strength may largely depend on perceived cultural norms of religion (e.g. ...
... Although typically small in size, this effect is consistently observed in both cross-sectional (Hoogeveen et al., 2023;Koenig & Larson, 2001) and longitudinal research (Garssen et al., 2021). Moreover, the positive link between religion and well-being has been replicated in different cultural contexts (Cohen & Johnson, 2017;Hoogeveen et al., 2023), even though several studies on person-culture fit show that the existence of this relationship and its strength may largely depend on perceived cultural norms of religion (e.g. Diener et al., 2011;Gebauer et al., 2017;Pokimica et al., 2012;Stavrova et al., 2013). ...
Article
Recent estimates by the Pew Research Center show that Poland is one of the fastest secularizing countries in the world. Despite the growing role of such attitudes in Poland and other countries around the world, still little is known about the psychosocial functioning and mental health of nonbelievers. The present study aimed to fill in this knowledge gap by examining various factors that may contribute to nonbelievers’ well-being, including perceived social support, emotion regulation strategies, secular beliefs (humanism and belief in science), and clarity of self-beliefs. Five hundred Polish nonbelievers took part in this study, of whom 38% reported no nominal affiliation with any religious or secular group, 33% affiliated with Roman Catholics, and 22% identified as Atheists. Comparing these groups, we found that while Catholics and Atheists did not differ significantly in terms of their overall well-being, religious “nones” scored lower than Catholics on overall well-being (but not meaning in life or social support). In addition, it was found that—at the level of the whole sample—having greater clarity of self-beliefs, using functional emotion regulation strategies, as well as having greater perceived social support were all positively associated with the presence of meaning in life and well-being.
... A many-analysts approach involves different analysis teams that each attempt to address a particular research question using the same data but different statistical methods. This approach has been used to empirically assess analytic variability/robustness of research outcomes [60,[72][73][74][75][76][77][78][79][80][81]. In the current project, we directly invited experts from research groups that have relevant knowledge concerning our statistical question and design (i.e. ...
... Moreover, some research questions might be more appropriate than others for inviting different methodological experts; a research question and design that clearly dictates a t-test might leave little room for different analytic strategies. At the same time, our experience with a previous multi-analyst project emphasized that the employment of different methodological or statistical approaches should not be understated; we expected mostly multi-level regression models in the Many-Analysts Religion Projects, yet were surprised to witness the variety of analyses, including machine learning, structural equation modelling, t-test, multiverse analysis and network analysis [81,118]. Furthermore, we believe the several-analysts design could also be extended to involve different theoretical experts, instead of solely methodological specialists, somewhat similar to an adversarial collaboration. ...
Article
Full-text available
Paranormal beliefs encompass a wide variety of phenomena, including the existence of supernatural entities such as ghosts and witches, as well as extraordinary human abilities such as telepathy and clairvoyance. In the current study, we used a nationally representative sample ( N = 2534 ) to investigate the presence and correlates of paranormal beliefs among the secular Dutch population. The results indicated that most single paranormal phenomena (e.g. belief in clairvoyance) are endorsed by 10–20% of Dutch respondents; however, 55.6% of respondents qualify as paranormal believers based on the preregistered criterion that they believe in at least one phenomenon with considerable certainty. In addition, we invited four analysis teams with different methodological expertise to assess the structure of paranormal beliefs using traditional factor analysis, network analysis, Bayesian network analysis and latent class analysis (LCA). The teams’ analyses indicated adequate fit of a four-factor structure reported in a 1985 study, but also emphasized different conclusions across techniques; network analyses showed evidence against strong connectedness within most clusters, and suggested a five-cluster structure. The application of various analytic techniques painted a nuanced picture of paranormal beliefs and believers in The Netherlands and suggests that despite increased secularization, subgroups of the general population still believe in paranormal phenomena.
... Additionally, team science approaches, in which multiple researchers and laboratories collaborate to examine a particular research topic, can help better elucidate the robustness of a particular finding. Adversarial collaborations such as many-analysts approaches in which numerous independent research teams analyse the same dataset to address a research question can help mitigate individual-researcher bias 166 . Multiverse approaches that conduct target analyses across a comprehensive set of alternatively processed datasets might clarify the robustness of an effect across different data processing methods (for example, by excluding cases or transforming variables) 167 . ...
Article
Full-text available
The administration of classic psychedelics has been associated with well-being and mental health benefits as well as risks and adverse events. The acute subjective effects of psychedelics might have a causal role in these risks and therapeutic benefits, but inconsistencies and limitations in the conceptualization and measurement of these acute subjective effects hinder research and clinical advances. In this Review, we outline current characterizations and psychometric examinations of the acute subjective effects of psychedelics, evaluate the construct validity of commonly used measures and describe findings showing that specific acute subjective effects predict certain outcomes. We discuss how to balance the limitations of existing measures with methodological advances in practice and elaborate on well-known methods and other psychological processes that can help inform the creation of new measures. We suggest actionable recommendations for how the field can transcend current conceptualizations and provide guidance on best practices until the next generation of measures is validated. Sections
... The study reinforces the crucial role of pastoral care in supporting mental health, echoing extensive research that demonstrates the positive correlation between religious involvement and psychological well-being (Hoogeveen et al. 2022;Tsomokos and Dunbar 2023;Garssen et al. 2020). The provision of counselling services by pastoral care programmes is particularly vital in addressing the mental health challenges often encountered in higher education settings. ...
Article
Full-text available
This study investigates the impact of pastoral care on the religious and psychological well-being of students residing in St. John’s College, a Christian tradition-based residential hall in Hong Kong. The research employs a mixed-methods approach and comprises two complementary studies: a quantitative analysis (Study 1) involving 98 participants and a qualitative investigation (Study 2) with 26 participants. Study 1 utilised the Religious Commitment Inventory-10 (RCI-10) and the Warwick-Edinburgh Mental Well-being Scale (WEMWBS) to measure religious commitment and psychological well-being, respectively. Initial findings revealed significant correlations suggesting that higher levels of religious commitment engender improved psychological well-being. Study 2 explored these relationships further through semi-structured interviews, providing deeper insights into how pastoral care influences students’ religious practices and mental health. The qualitative data underscored the nuanced role of pastoral activities in fostering a supportive community atmosphere, enhancing both personal and communal religious experiences.
... The anthology edited by Amy L. Ai and colleagues is an impressive discussion of the usefulness of some two dozen assessment measures in cross-cultural research (Ai et al., 2021; see also Paniagua & Yamada, 2013). A growing number of research teams in the cognitive science of religion have also been developing measures and methods with cross-cultural validity (Hoogeveen, Sarafoglou, Aczel, et al., 2022;Hoyer & Reddish, 2019;Purzycki et al., 2016;Purzycki et al., 2018;Turchin et al., 2023;Sznycer, Xygalatas, Agey, et al., 2018;Watts, Gray, & Bulbulia, 2019). None of these studies are mentioned. ...
Article
The Varieties of Spiritual Experience: 21st Century Research and Perspectives, by David B. Yaden and Andrew B. Newberg. Oxford: Oxford University Press, 2022. x + 430 pages, £26.49. ISBN: 9780190665678
... (RQ1). As described by Hoogeveen, Sarafoglou, Aczel, et al. (2023) this was answered in the affirmative. These authors stated that regardless of the subjective analytic decisions of the various teams (e.g. ...
... Studies closest to this ideal are those relying on the multianalyst approach, where different analysts independently test the same hypothesis on the same data. Our review involves three papers that fulfill our inclusion criteria (see Materials and Methods for details), examining the variability in effect sizes due to analytical flexibility for five hypotheses (69)(70)(71). ...
Article
Full-text available
A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies.
... 11: 240125 [6] or a 'multi-analyst approach' [4,7]), different independent analysis teams analyse the same dataset (e.g. [16][17][18][19][20][21][22][23]). In both cases, the end result is an evaluation of the consistency of the observed outcomes across all analyses. ...
... Many-analysts projects are a recent innovation, but they have already been adopted in many different fields, including neuroscience [16,17,[24][25][26], economics [27,28], epidemiology [29], ecology [30,31], political science [18] and psychology [19][20][21][22][23][32][33][34][35]. Many of these projects concluded that different but justifiable analytic decisions led to diverging outcomes, sometimes with statistically significant effects in opposite directions (e.g. ...
... Many of these projects concluded that different but justifiable analytic decisions led to diverging outcomes, sometimes with statistically significant effects in opposite directions (e.g. [18,28,34], but see [23]). ...
Article
Full-text available
Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.