Nicholas Alvaro Coles’s research while affiliated with Stanford University and other places

Researchers are increasingly using machine learning to study physiological markers of emotional experience. In the present work, we evaluated the promises and limitations of this approach via a big team science competition. Twelve teams of researchers competed to predict self-reported core affective experiences using a multi-modal set of peripheral nervous system features. Models were trained and tested in multiple ways: with data divided by participants, emotions, inductions, and time periods. In 100% of tests, teams outperformed baseline models that made random predictions. In 46% of tests, teams also outperformed baseline models that relied on the simple average of ratings from training datasets. In a follow-up, three models judged most promising by competition organizers exhibited lower prediction accuracy when re-tested on data with simulated physiological randomness. These results bolster claims that machine learning can capture physiological markers of affective experience. More notably, though, results uncovered a methodological challenge: multiplicative constraints on generalizability. Inferences about the accuracy and theoretical implications of machine learning efforts depended not only on their architecture, but also how they were trained, tested, and evaluated. For example, some teams performed better when tested on observations from different (vs. the same) subjects seen during training. Such results could be interpreted as evidence of biologically innate patterns that are sensitive to context. However, such conclusions would be premature because other teams exhibited the opposite pattern. Taken together, results illustrate how big team science can be leveraged to understand the promises and limitations of machine learning methods in affective science and beyond.

When researchers collaboratively tackle challenging questions, such as those related to climate change, the impact of artificial intelligence, or the nature of consciousness, they often encounter disagreements that are difficult to resolve (1). When disagreements persist, what should the team members do? Options may seem limited to co-authoring a paper they disagree with, delaying in hopes that consensus will eventually be reached, or leaving without credit. However, there is a better approach: transparently documenting the disagreements.

Demand characteristics are a fundamental methodological concern in experimental psychology. Yet, little is known about the direction, magnitude, consistency, and mechanisms underlying their effects. We conducted a three-level meta-analysis of 195 effect sizes from 40 studies that provided experimental tests of demand effects by manipulating the hypothesis communicated to participants. Results indicated that demand characteristics tend to produce small overall increases in hypothesis-consistent responding (d = 0.22, 95% CI [0.11, 0.33]). However, these effects were extremely heterogeneous (between-study τ = 0.31; within-study σ = 0.20), with the estimated distribution of true effects ranging from d = 1.98 (a massive increase in hypothesis-consistent responding) to d = -1.44 (a massive increase in hypothesis-inconsistent responding). Contrary to conventional motivation accounts, we did not find evidence that demand effects were driven by post-hoc measures of participants’ motivation or opportunity to adjust their responses. We did, however, find robust evidence for accounts that emphasize the role of participants’ expectancies about the hypothesized effects. Similar findings emerged in a direct replication of one recent study included in the meta-analysis. Taken together, results underscore the importance – and challenges – of understanding and controlling for demand characteristics in experimental design.

The past decade has witnessed a proliferation of Big Team Science (BTS), endeavours where a comparatively large number of researchers pool their intellectual and/or material resources in pursuit of a common goal. Despite this burgeoning interest, there exists little guidance on how to create, manage, and participate in these collaborations. In this paper, we integrate insights from a multidisciplinary set of BTS initiatives to provide a how-to guide for BTS. We first discuss initial considerations for launching a BTS project, such as building the team, identifying leadership, governance, tools, and open science approaches. We then turn to issues related to running and completing a BTS project, such as study design, ethical approvals, and issues related to data collection, management, and analysis. Finally, we address topics that present special challenges for BTS, including authorship decisions, collaborative writing, and team decision making.

In response to the COVID-19 pandemic, the Psychological Science Accelerator coordinated three large-scale psychological studies to examine the effects of loss-gain framing, cognitive reappraisals, and autonomy framing manipulations on behavioral intentions and affective measures. The data collected (April to October 2020) included specific measures for each experimental study, a general questionnaire examining health prevention behaviors and COVID-19 experience, geographical and cultural context characterization, and demographic information for each participant. Each participant started the study with the same general questions and then was randomized to complete either one longer experiment or two shorter experiments. Data were provided by 73,223 participants with varying completion rates. Participants completed the survey from 111 geopolitical regions in 44 unique languages/dialects. The anonymized dataset described here is provided in both raw and processed formats to facilitate re-use and further analyses. The dataset offers secondary analytic opportunities to explore coping, framing, and self-determination across a diverse, global sample obtained at the onset of the COVID-19 pandemic, which can be merged with other time-sampled or geographic data.

Ideal partner preferences (i.e., ratings of the desirability of attributes like attractiveness or intelligence) are the source of numerous foundational findings in the interdisciplinary literature on human mating. Recently, research on the predictive validity of ideal partner preference-matching (i.e., do people positively evaluate partners who match versus mismatch their ideals?) has become mired in several problems. For one, articles exhibit discrepant analytic and reporting practices. Furthermore, different findings emerge across laboratories worldwide, perhaps because they sample different relationship contexts and/or populations. The current project—partnered with the Psychological Science Accelerator—can bring clarity to this literature. This registered report uses a highly powered design across multiple world regions to calculate preference-matching effect sizes and variability estimates for all relevant analytic tests. It also examines effects in different relationship contexts and subsamples (e.g., attraction, established relationships, recently formed relationships).

The recent proliferation of big team science is forcing researchers to revisit three issues around authorship: (1) What is an authorship-worthy contribution, (2) How should contributions be documented, and (3) How should disagreements among large teams of co-authors be handled?

The Funding and Finance Committee serves as an advisory board for the PSA’s Director and Associate Directors on all matters related to finance and funding. The Funding and Finance Committee will work with the PSA’s Director and Associate Directors (hereafter “the PSA Board”) to propose and manage the PSA’s annual operating budget, provide guidance to specific PSA projects about project-specific budgets when appropriate, propose a long-term financial plan to sustain the PSA, provide suggestions on how to balance the budget, and search for different fundraising activities and opportunities around the globe.

The COVID-19 pandemic (and its aftermath) highlights a critical need to communicate health information effectively to the global public. Given that subtle differences in information framing can have meaningful effects on behavior, behavioral science research highlights a pressing question: Is it more effective to frame COVID-19 health messages in terms of potential losses (e.g., “If you do not practice these steps, you can endanger yourself and others”) or potential gains (e.g., “If you practice these steps, you can protect yourself and others”)? Collecting data in 48 languages from 15,929 participants in 84 countries, we experimentally tested the effects of message framing on COVID-19-related judgments, intentions, and feelings. Loss- (vs. gain-) framed messages increased self-reported anxiety among participants cross-nationally with little-to-no impact on policy attitudes, behavioral intentions, or information seeking relevant to pandemic risks. These results were consistent across 84 countries, three variations of the message framing wording, and 560 data processing and analytic choices. Thus, results provide an empirical answer to a global communication question and highlight the emotional toll of loss-framed messages. Critically, this work demonstrates the importance of considering unintended affective consequences when evaluating nudge-style interventions.