Daniel M. Oppenheimer’s research while affiliated with Carnegie Mellon University and other places

Strategies designed to address specific problems often give rise to unintended, negative consequences that, while foreseeable, are overlooked during strategy formulation and evaluation. We propose that this oversight is not due to a lack of knowledge but rather a cognitive bias rooted in focalism—the tendency to focus narrowly on the primary objective, ignoring other relevant factors, such as potential consequences. We introduce the concept of consequence neglect, where problem solvers fail to generate or consider downstream effects of their solutions because these consequences are not central to the proximal goal. Across four studies, we provide evidence supporting this phenomenon. Specifically, we find that individuals rate strategies more negatively after being prompted to generate both positive and negative consequences, suggesting that negative outcomes are not naturally weighted unless attention is explicitly drawn to them. We conclude by discussing the broader implications of consequence neglect for policymaking, business, and more general problem solving, and offer directions for future research.

The rise of Large Language Model (LLM) chatbots, such as ChatGPT and Gemini, has revolutionized how we access information. These LLMs can answer a wide array of questions on nearly any topic. When humans answer questions, especially difficult or uncertain questions, they often accompany their responses with metacognitive confidence judgments indicating their belief in their accuracy. LLMs are certainly capable of – and willing to – provide confidence judgments, but it is currently unclear how accurate these confidence judgments are. To fill this gap in the literature, the present studies investigate the capability of LLMs to quantify uncertainty through confidence judgments. We compare the absolute and relative accuracy of confidence judgments made by four LLMs (ChatGPT, Bard/Gemini, Sonnet, Haiku) and human participants in both domains of aleatory uncertainty - NFL predictions (Study 1a; n = 502), and Oscar predictions (Study 1b; n = 109) – and domains of epistemic uncertainty - Pictionary performance (Study 2; n = 164), Trivia questions (Study 3; n = 110), and questions about life at a university (Study 4; n = 110). We find several commonalities between LLMs and humans, such as achieving similar levels of absolute and relative metacognitive accuracy (although LLMs tend to be slightly more accurate on both dimensions). Like humans, we also find that LLMs tend to be overconfident. However, we find that, unlike humans, LLMs – especially ChatGPT and Gemini – often struggle to adjust their confidence judgments based on past performance, highlighting a key metacognitive limitation.

The rise of Large Language Model (LLM) chatbots, such as ChatGPT and Gemini, has revolutionized how we access information. These LLMs can answer a wide array of questions on nearly any topic. When humans answer questions, especially difficult or uncertain questions, they often accompany their responses with metacognitive confidence judgments indicating their belief in their accuracy. LLMs are certainly capable of – and willing to – provide confidence judgments, but it is currently unclear how accurate these confidence judgments are. To fill this gap in the literature, the present studies investigate the capability of LLMs to quantify uncertainty through confidence judgments. We compare the absolute and relative accuracy of confidence judgments made by four LLMs (ChatGPT, Bard/Gemini, Sonnet, Haiku) and human participants in both domains of aleatory uncertainty - NFL predictions (Study 1a; n = 502), and Oscar predictions (Study 1b; n = 109) – and domains of epistemic uncertainty - Pictionary performance (Study 2; n = 164), Trivia questions (Study 3; n = 110), and questions about life at a university (Study 4; n = 110). We find several commonalities between LLMs and humans, such as achieving similar levels of absolute and relative metacognitive accuracy (although LLMs tend to be slightly more accurate on both dimensions). Like humans, we also find that LLMs tend to be overconfident. However, we find that, unlike humans, LLMs – especially ChatGPT and Gemini – often struggle to adjust their confidence judgments based on past performance, highlighting a key metacognitive limitation.

Collaborative learning - an educational paradigm in which students work together to master content and complete projects - has long been a staple of classroom pedagogy. However, the rapid development of Large Language Model chatbots (LLMs), such as ChatGPT, Gemini, and Claude, has created the potential for a new frontier in collaborative learning in which students collaborate not with other students, but with LLMs. In this classroom-based study, we tested a novel intervention in which college students taking an introductory level social science elective (n = 154) were asked multiple times throughout a semester to write argumentative essays, have them critiqued by LLMs, and improve their essays by incorporating the LLMs’ critiques into their arguments or countering the LLMs’ arguments. We found that students enjoyed working with LLMs and that collaborating with LLMs improved students’ argumentative writing skills over the course of the semester - even when essay quality was assessed before students elicited the LLMs’ critiques. Students also improved their prompt engineering skills and increased their self-efficacy for working with LLMs. These findings suggest that collaborative learning with LLMs may be a highly scalable, low-cost intervention for improving students’ argumentative writing skills at the college level. Implications and limitations of the intervention are discussed.

Despite their ability to answer complex questions, it is unclear whether generative chatbots should be considered experts in any domain. There are several important cognitive and metacognitive differences that separate human experts from generative chatbots. First, human experts’ domain knowledge is deep, efficiently structured, adaptive, and intuitive—whereas generative chatbots’ knowledge is shallow and inflexible, leading to errors that human experts would rarely make. Second, generative chatbots lack access to critical metacognitive capacities that allow humans to detect errors in their own thinking and communicate this information to others. Though generative chatbots may surpass human experts in the future—for now, the nature of their knowledge structures and metacognition prevent them from reaching true expertise.

When Mueller and Oppenheimer (2014) documented the longhand advantage—the finding that students learn better when they take notes using pen and paper rather than a laptop—the study went viral, influencing classroom laptop policies around the world. However, more recently Urry et al. (2021) followed up on the original finding but found no evidence for a longhand advantage. Similarly, the literature on whether interventions to improve executive functions lead to improved classroom performance is full of mixed and contradictory findings. Instructors who hope to use evidence-based interventions in the classroom are confronted with a dilemma when they encounter discrepant studies regarding whether or not an intervention is effective. One helpful approach to navigating this challenge is to consider the differences between direct replications and generalization studies. While direct replications recreate the conditions of the original study as closely as possible to determine whether the results are reliable and trustworthy, generalization studies investigate whether the results of the original study are robust to novel conditions, populations, or environments. Because no study can perfectly replicate the conditions of another study, it is not always clear whether a study should be considered a replication or generalization. To resolve this, we describe a framework based in the intervention literature that proposes five principles for evaluating the match between original studies and their follow-ups. We illustrate the utility of this framework using the discrepant findings from Urry et al. (2021) and Mueller and Oppenheimer (2014), and the mixed evidence from the executive control interventions literature as case studies. Broader implications and practicalities for interpreting findings from replication attempts are discussed.

Collaborative learning - an educational paradigm in which students work together to master content and complete projects - has long been a staple of classroom pedagogy. However, the rapid development of Large Language Model chatbots (LLMs), such as ChatGPT, Gemini, and Claude, has created the potential for a new frontier in collaborative learning in which students collaborate not with other students, but with LLMs. In this classroom-based study, we tested a novel intervention in which college students taking an introductory level social science elective (n = 154) were asked multiple times throughout a semester to write argumentative essays, have them critiqued by LLMs, and improve their essays by incorporating the LLMs’ critiques into their arguments or countering the LLMs’ arguments. We found that students enjoyed working with LLMs and that collaborating with LLMs improved students’ argumentative writing skills over the course of the semester - even when essay quality was assessed before students elicited the LLMs’ critiques. Students also improved their prompt engineering skills and increased their self-efficacy for working with LLMs. These findings suggest that collaborative learning with LLMs may be a highly scalable, low-cost intervention for improving students’ argumentative writing skills at the college level. Implications and limitations of the intervention are discussed.