Michael David Lee

University of California, Irvine | UCI · Department of Cognitive Sciences

The wisdom of the crowd is the finding that aggregating the judgments of many people can lead to surprisingly accurate group judgments. Usually statistical methods are used to aggregate people’s judgments, but there are advantages to using cognitive models instead. Crowd judgments based on cognitive modeling can (a) identify experts and amplify the...

We develop a Bayesian method for aggregating partial ranking data using the Thurstone model. Our implementation is a JAGS graphical model that allows each individual to rank any subset of items, and provides an inference about the latent true ranking of the items and the relative expertise of each individual. We demonstrate the method by analyzing...

We study how people solve the optimal stopping problem of buying an airline ticket. Over a set of problems, people were given 12 opportunities to buy a ticket ranging from 12 months before travel to 1 day before. The distributions from which prices were sampled changed over time, following patterns observed in industry analysis of flight ticket pri...

We apply cognitive modeling to improve the wisdom of the crowd in a spatial knowledge task. Participants provided point estimates for where 48 US cities are located and then, using the point estimate as a center point, chose a radius large enough that they believed the resulting circle was certain to contain the city’s location. Simple and radius-w...

The circular drift-diffusion model (CDDM) is a sequential sampling model designed to account for decisions and response times in decision-making tasks with a circular set of choice alternatives. We present and demonstrate a fully Bayesian implementation and extension of the CDDM. This development allows researchers to apply the CDDM to data from co...

The mnemonic discrimination task (MDT) is a widely used cognitive assessment tool. Performance in this task is believed to indicate an age-related deficit in episodic memory stemming from a decreased ability to pattern-separate among similar experiences. However, cognitive processes other than memory ability might impact task performance. In this s...

A task used in clinical settings to assess memory is a surprise free recall of animal names originally encountered as part of an earlier triadic comparison task. Modeling the free recall is challenging, because learning occurs incidentally, items are presented multiple times, and the items have semantic relationships to each other. To address this...

Drafting is a competitive task in which a set of decision makers choose from a set of resources sequentially, with each resource becoming unavailable once selected. How people make these choices raises basic questions about human decision making, including people’s sensitivity to the statistical regularities of the resource environment, their abili...

There are many ways to measure how people manage risk when they make decisions. A standard approach is to measure risk propensity using self-report questionnaires. An alternative approach is to use decision-making tasks that involve risk and uncertainty, and apply cognitive models of task behavior to infer parameters that measure people’s risk prop...

We study whether experts and novices differ in the way they make predictions about National Football League games. In particular, we measure to what extent their predictions are consistent with five environmental regularities that could support decision making based on heuristics. These regularities involve the home team winning more often, the tea...

The last 25 years have shown a steady increase in attention for the Bayes factor as a tool for hypothesis evaluation and model selection. The present review highlights the potential of the Bayes factor in psychological research. We discuss six types of applications: Bayesian evaluation of point null, interval, and informative hypotheses, Bayesian e...

Background Many commonly‐used multi‐trial wordlist memory (WLM) tests present list words in a shuffled order across learning trials, while others maintain a fixed order across trials. The ADAS‐Cog employs shuffling across trials in only one sequence for all assessed subjects. Therefore, to examine the impact of presentation order shuffling on the c...

Student evaluations of teaching are widely used to assess instructors and courses. Using a model-based approach and Bayesian methods, we examine how the direction of the scale, labels on scales, and the number of options affect the ratings. We conduct a within-participants experiment in which respondents evaluate instructors and lectures using diff...

The Balloon Analogue Risk Task (BART) is widely-used to measure risk propensity in theoretical, clinical, and applied research. In the task, people choose either to pump a balloon to increase its value at the risk of the balloon bursting and losing all value, or to bank the current value of the balloon. Risk propensity is most commonly measured as...

We study the wisdom of the crowd in three sequential decision-making tasks: the Balloon Analogue Risk Task (BART), optimal stopping problems, and bandit problems. We consider a behavior-based approach, using majority decisions to determine crowd behavior and show that this approach performs poorly in the BART and bandit tasks. The key problem is th...

Although the Kendall distance is a standard metric in computer science, it is less widely used in psychology. We demonstrate the usefulness of the Kendall distance for analyzing psychological data that take the form of ranks, lists, or orders of items. We focus on weighted extensions of the metric that allow for heterogeneity of item importance, it...

We use cognitive models to evaluate three theories of the change in semantic memory caused by Alzheimer’s disease. We use data from 14,096 clinical assessments of 3602 Alzheimer’s patients and their caregivers. Each patient completed a semantic memory task involving the odd-one-out comparison of animal names. Each patient was also independently eva...

Background This study sought to validate a pragmatic method to predict impending cognitive decline in Alzheimer’s disease (AD) patients, prior to the development of hallmark symptoms. The method, a Hierarchical Bayesian Cognitive Process (HBCP) model, uses item responses to a wordlist memory (WLM) test to generate digital cognitive biomarkers (Tabl...

Background: Recent Alzheimer’s disease (AD) trials have faced significant challenges to enroll pre-symptomatic or early stage AD subjects with biomarker positivity, minimal or no cognitive impairment, and likelihood to decline cognitively during a short trial period. Our previous study showed that digital cognitive biomarkers (DCB), generated by a...

The last 25 years have shown a steady increase in attention for the Bayes factor as a tool for hypothesis evaluation and model selection. The present review highlights the potential of the Bayes factor in psychological research. We discuss six types of applications: Bayesian evaluation of point null, interval, and informative hypotheses, Bayesian e...

We develop and demonstrate a method for inferring changes in strategy use, applicable to decision making in multi-attribute choice. The method is an extension of one developed by Lee, Gluck, and Walsh (Decision 6:335–368, 2019) and continues to rely on a Bayesian approach for inferring strategy switches based on spike-and-slab priors. The extension...

The principle of predictive irrelevance states that when two competing models predict a data set equally well, that data set cannot be used to discriminate the models and – for that specific purpose – the data set is evidentially irrelevant. To highlight the ramifications of the principle, we first show how a single binomial observation can be irre...

Optimal stopping problems require people to choose from a sequence of values presented sequentially, under the constraint that it is not possible to return to an earlier option. Usually, the distribution from which values are drawn is the same for each option in the sequence. We consider an extension in which the distributions change in a known way...

We model word-list learning over sequences of immediate and delayed free recall tasks to study the impact of memory impairment on episodic memory. We use a previously developed Multinomial Processing Tree (MPT) model of encoding, retrieval, and learning (Alexander et al., 2016), and apply it to behavioral data from thousands of patients tested tens...

Why is there no consensual way of conducting Bayesian analyses? We present a summary of agreements and disagreements of the authors on several discussion points regarding Bayesian inference. We also provide a thinking guideline to assist researchers in conducting Bayesian inference in the social and behavioural sciences.

The target article on robust modeling (Lee et al. in review) generated a lot of commentary. In this reply, we discuss some of the common themes in the commentaries; some are simple points of agreement while others are extensions of a practical or abstract nature. We also address a small number of disagreements or confusions.

In most decision-making domains, people recognize some stimuli but not others. The validity of recognition as a basis for making decisions about the stimuli is empirically unclear, but has strong implications for the development of theories and models of decision-making. Following the framework used to motivate the recognition heuristic (Gigerenzer...

In an attempt to increase the reliability of empirical findings, psychological scientists have recently proposed a number of changes in the practice of experimental psychology. Most current reform efforts have focused on the analysis of data and the reporting of findings for empirical studies. However, a large contingent of psychologists build mode...

Differential strategy use is a topic of intense investigation in developmental psychology. Questions under study are as follows: How do strategies change with age, how can individual differences in strategy use be explained, and which interventions promote shifts from suboptimal to optimal strategies? In order to detect such differential strategy u...

We apply the “wisdom of the crowd” idea to human category learning, using a simple approach that combines people's categorization decisions by taking the majority decision. We first show that the aggregated crowd category learning behavior found by this method performs well, learning categories more quickly than most or all individuals for 28 previ...

Whenever parameter estimates are uncertain or observations are contaminated by measurement error, the Pearson correlation coefficient can severely underestimate the true strength of an association. Various approaches exist for inferring the correlation in the presence of estimation uncertainty and measurement error, but none are routinely applied i...

The Iowa Gambling Task (IGT) is one of the most popular experimental paradigms for comparing complex decision-making across groups. Most commonly, IGT behavior is analyzed using frequentist tests to compare performance across groups, and to compare inferred parameters of cognitive models developed for the IGT. Here, we present a Bayesian alternativ...

In optimal stopping problems, people are asked to choose the best option out of a sequence of alternatives, under the constraint that they cannot return to an earlier option once it is rejected. We examine human performance on variations of the optimal stopping problem, with different environments and with different goals. Specifically, we consider...

People often interact with environments that can provide only a finite number of items as resources. Eventually a book contains no more chapters, there are no more albums available from a band, and every Pok?mon has been caught. When interacting with these sorts of environments, people either actively choose to quit collecting new items, or they ar...

The development of cognitive models involves the creative scientific formalization of assumptions, based on theory, observation, and other relevant information. In the Bayesian approach to implementing, testing, and using cognitive models, assumptions can influence both the likelihood function of the model, usually corresponding to assumptions abou...

Take-the-best is a decision-making strategy that chooses between alternatives, by searching the cues representing the alternatives in order of cue validity, and choosing the alternative with the first discriminating cue. Theoretical support for take-the-best comes from the “fast and frugal” approach to modeling cognition, which assumes decision-mak...

The practical advantages of Bayesian inference are demonstrated here through two concrete examples. In the first example, we wish to learn about a criminal’s IQ: a problem of parameter estimation. In the second example, we wish to quantify and track support in favor of the null hypothesis that Adam Sandler movies are profitable regardless of their...

In a top-n task, people produce a list of items that they believe are ordered relative to a criterion, and can include any number of items in their list. We develop Thurstonian cognitive models of the individual differences and decision-making processes involved in producing top-n lists, and apply it to the problem of inferring an aggregated list f...

Multidimensional scaling (MDS) models of mental representation assume stimuli are represented by points in a low-dimensional space, such that more similar stimuli are represented by points closer to each other. We consider possible individual differences in MDS representations, using the recently proposed K-INDSCAL model, which allows for both sub-...

The article reviews the field from a historical perspective, starting from the works of Fechner and Thurstone, and outlining the basic theoretical concepts in four traditional areas: learning, psychophysics, measurement and choice, and response times. More recent topics that have emerged from these areas are also briefly mentioned together with the...

We study the effect of memory impairment on triadic comparisons of animal names in a large clinical data set. We define eight groups of subjects in terms of their delayed free recall performance, and present standard analyses of the triadic comparison and free recall data that provide little insight into the effect of memory impairment on semantic...

Interval estimates – estimates of parameters that include an allowance for sampling uncertainty – have long been touted as a key component of statistical analyses. There are several kinds of interval estimates, but the most popular are confidence intervals (CIs): intervals that contain the true parameter value in some known proportion of repeated s...

People have all sorts of different knowledge and are able to express this information in many ways. One ubiquitous form of expression is through ranking, in which people generate a list of items with respect to the criterion of interest. In some structured circumstances — like experts predicting the final standings of all the teams in a sporting co...

Interval estimates – estimates of parameters that include an allowance for sampling uncertainty – have long been touted as a key component of statistical analyses. There are several kinds of interval estimates, but the most popular are confidence intervals (CIs): intervals that contain the true parameter value in some known proportion of repeated s...

Many decisions in the lives of animals and humans require a fine balance between the exploration of different options and the exploitation of their rewards. Do you buy the advertised car, or do you test drive different models? Do you continue feeding from the current patch of flowers, or do you fly off to another one? Do you marry your current part...

Hilbig and Moshagen (Psychonomic Bulletin & Review, 21, 1431-1443, 2014) recently developed a method for making inferences about the decision processes people use in multi-attribute forced choice tasks. Their paper makes a number of worthwhile theoretical and methodological contributions. Theoretically, they provide an insightful psychological moti...

The less-is-more effect predicts that people can be more accurate making paired-comparison decisions when they have less knowledge, in the sense that they do not recognize all of the items in the decision domain. The traditional theoretical explanation is that decisions based on recognizing one alternative but not the other can be more accurate tha...

p>Interval estimates -- estimates of parameters that include an allowance for sampling uncertainty -- have long been touted as a key component of statistical analyses. There are several kinds of interval estimates, but the most popular are confidence intervals (CIs): intervals that contain the true parameter value in some known proportion of repeat...

Diffusion models are widely-used and successful accounts of the time course of two-choice decision making. Most diffusion models assume constant boundaries, which are the threshold levels of evidence that must be sampled from a stimulus to reach a decision. We summarize theoretical results from statistics that relate distributions of decisions and...

The power fallacy refers to the misconception that what holds on average -across an ensemble of hypothetical experiments- also holds for each case individually. According to the fallacy, high-power experiments always yield more informative data than do low-power experiments. Here we expose the fallacy with concrete examples, demonstrating that a pa...

We study how people terminate their search for information when making decisions in a changing environment. In 3 experiments, differing in the cost of search, participants made a sequence of 2-alternative decisions, based on the information provided by binary cues they could search. Whether limited or extensive search was required to maintain accur...

Interval estimates – estimates of parameters that include an allowance for sampling uncertainty – have long been touted as a key component of statistical analyses. There are several kinds of interval estimates, but the most popular are confidence intervals (CIs): intervals that contain the true parameter value in some known proportion of repeated s...

We develop a cognitive modeling approach, motivated by classic theories of knowledge representation and judgment from psychology, for combining people's rankings of items. The model makes simple assumptions about how individual differences in knowledge lead to observed ranking data in behavioral tasks. We implement the cognitive model as a Bayesian...

We demonstrate the usefulness of cognitive models for combining human estimates of probabilities in two experiments. The first experiment involves people's estimates of probabilities for general knowledge questions such as "What percentage of the world's population speaks English as a first language?" The second experiment involves people's estimat...

In most decision-making situations, there is a plethora of information potentially available to people. Deciding what information to gather and what to ignore is no small feat. How do decision makers determine in what sequence to collect information and when to stop? In two experiments, we administered a version of the German cities task developed...

Faced with probabilistic relationships between causes and effects, quantum theory assumes that deterministic causes do not exist, and that only incomplete probabilistic expressions of knowledge are possible. As in its application to physics, this fundamental epistemological stance severely limits the ability of quantum theory to provide insight and...

The scale-invariant memory, perception, and learning (SIMPLE) model developed by Brown, Neath, and Chater (2007) formalizes the theoretical idea that scale invariance is an important organizing principle across numerous cognitive domains and has made an influential contribution to the literature dealing with modeling human memory. In the context of...

Bayesian inference has become a standard method of analysis in many fields of science. Students and researchers in experimental psychology and cognitive science, however, have failed to take full advantage of the new and exciting possibilities that the Bayesian approach affords. Ideal for teaching and self study, this book demonstrates how to do Ba...

Wills and Pothos (2012) reviewed approaches to evaluating formal models of categorization, raising a series of worthwhile issues, challenges, and goals. Unfortunately, in discussing these issues and proposing solutions, Wills and Pothos (2012) did not consider Bayesian methods in any detail. This means not only that their review excludes a major bo...

Identifying disease-modifying treatment effects in earlier stages of Alzheimer's disease (AD)-when changes are subtle-will require improved trial design and more sensitive analytical methods. We applied hierarchical Bayesian analysis with cognitive processing (HBCP) models to the Alzheimer's Disease Assessment Scale-Cognitive subscale (ADAS-Cog) an...

Formal models in psychology are used to make theoretical ideas precise and allow them to be evaluated quantitatively against data. We focus on one important-but under-used and incorrectly maligned-method for building theoretical assumptions into formal models, offered by the Bayesian statistical approach. This method involves capturing theoretical...

Game theory has been useful for understanding risk-taking and cooperative behavior. In the present study, subjects played the Hawk-Dove game with simulated and embodied (robotic) neu-ral agents which used a neurobiologically plausible model of action selection and adaptive behaviors. Subjects had their serotonin levels temporarily altered through a...

Heuristic decision-making models, like Take-the-best, rely on environmental regularities. They conduct a limited search, and ignore available information, by assuming there is structure in the decision-making environment. Take-the-best relies on at least two regularities: DIMINISHING RETURNS, which says that information found earlier in search is m...

Two experiments were conducted examining the effectiveness of visualizations of unstructured texts. The first experiment presented transcriptions of unrehearsed dialog and the second used emails. Both experiments showed an advantage in overall performance for semantically structured two-dimensional (2D) spatialized layouts, such as multidimensional...

Despite their theoretical appeal, Bayesian methods for the assessment of poor effort and malingering are still rarely used in neuropsychological research and clinical diagnosis. In this article, we outline a novel and easy-to-use Bayesian latent group analysis of malingering whose goal is to identify participants displaying poor effort when tested....

MacGregor and Ormerod (1996) have presented results purporting to show that human performance on visually presented traveling salesman problems, as indexed by a measure of response uncertainty, is strongly determined by the number of points in the stimulus array falling inside the convex hull, as distinct from the total number of points. It is argu...

Determining how cognition affects functional abilities is important in Alzheimer disease and related disorders. A total of 280 patients (normal or Alzheimer disease and related disorders) received a total of 1514 assessments using the functional assessment staging test (FAST) procedure and the MCI Screen. A hierarchical Bayesian cognitive processin...

The "wisdom of the crowd" phenomenon refers to the finding that the aggregate of a set of proposed solutions from a group of individuals performs better than the majority of individual solutions. Most often, wisdom of the crowd effects have been investigated for problems that require single numerical estimates. We investigate whether the effect can...

We apply a cognitive modeling approach to the problem of measuring expertise on rank ordering problems. In these problems, people must order a set of items in terms of a given criterion (e.g., ordering American holidays through the calendar year). Using a cognitive model of behavior on this problem that allows for individual differences in knowledg...

Book synopsis: Over a century ago, William James proposed that people search through memory much as they rummage through a house looking for lost keys. We scour our environments for territory, food, mates, and information. We search for items in visual scenes, for historical facts, and for the best deals on Internet sites; we search for new friends...

Inductive generalization, where people go beyond the data provided, is a basic cognitive capability, and it underpins theoretical accounts of learning, categorization, and decision making. To complete the inductive leap needed for generalization, people must make a key ''sampling'' assumption about how the available data were generated. Previous mo...

Hierarchical Bayesian methods offer a principled and comprehensive way to relate psychological models to data. Here we use them to model the patterns of information search, stopping and deciding in a simulated binary comparison judgment task. The simulation involves 20 subjects making 100 forced choice comparisons about the relative magnitudes of t...

Analyses of multi-attribute decision problems are dominated by accounts which assume people select from a repertoire of cognitive strategies to make decisions. This paper explores an alternative account based on sequential sampling and evidence accumulation. Two experiments varied aspects of a decision environment to examine competing models of dec...

The phenomenon of the ‘wisdom of the crowds’ refers to the finding that the aggregate of a set of proposed solutions from a group of individuals performs better than the majority of individual solutions. We investigated this effect in the context of planar Euclidean traveling salesperson problem (TSP). The goal in TSPs is to estimate the shortest t...