No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
A review of uncertainty visualization errors: Working memory as an explanatory theory
Abstract
Uncertainty communicators often use visualizations to express the unknowns in data, statistical analyses, and forecasts. Well-designed visualizations can clearly and effectively convey uncertainty, which is vital for ensuring transparency, accuracy, and scientific credibility. However, poorly designed uncertainty visualizations can lead to misunderstandings of the underlying data and result in poor decision-making. In this chapter, we present a discussion of errors in uncertainty visualization research and current approaches to evaluation. Researchers consistently find that uncertainty visualizations requiring mental operations, rather than judgments guided by the visual system, lead to more errors. To summarize this work, we propose that increased working memory demand may account for many observed uncertainty visualization errors. In particular, the most common uncertainty visualization in scientific communication (e.g., variants of confidence intervals) produces systematic errors that may be attributable to the application of working memory or lack thereof. To create a more effective uncertainty visualization, we recommend that data communicators seek a sweet spot in the working memory required by various tasks and visualization users. Further, we also recommend that more work be done to evaluate the working memory demand of uncertainty visualizations and visualizations more broadly.
... Speed and accuracy, however, often exhibit a trade-off where an individual's performance improves when taking longer to complete a task, producing complex covariance (for a review, see [35]). Within visualization research, some have advocated for a converging methods approach (i.e., using multiple observable phenomena beyond speed and accuracy) to provide evidence for a visualization's utility [68,71]. Scholars have also recommended incorporating individual differences (e.g., [31,32,71]), which quantify how different people vary in given abilities, such as graph literacy [66]. ...
... It is a limiting factor in the amount of mental effort that can be allocated to a given task and has been studied by researchers in psychology and cognitive science for decades (e.g., [2,4,18,60,71]). Theoretical work has recently suggested that working memory is likely the cognitive mechanism that produces reasoning errors with uncertainty visualizations (for a review, see [68]). As with other abilities (e.g., maximum running pace), individuals differ in their average capacity to utilize working memory (i.e., working-memory capacity) [20,44]. ...
... Individuals with lower working-memory capacity have fewer cognitive resources available to complete demanding tasks. Thus, if participants with low-working-memory capacity show worse performance with some uncertainty communication techniques compared to others, we would have indirect evidence that the poor-performing uncertainty communication techniques require more working memory [68]. ...
As uncertainty visualizations for general audiences become increasingly common, designers must understand the full impact of uncertainty communication techniques on viewers' decision processes. Prior work demonstrates mixed performance outcomes with respect to how individuals make decisions using various visual and textual depictions of uncertainty. Part of the inconsistency across findings may be due to an over-reliance on task accuracy, which cannot, on its own, provide a comprehensive understanding of how uncertainty visualization techniques support reasoning processes. In this work, we advance the debate surrounding the efficacy of modern 1D uncertainty visualizations by conducting converging quantitative and qualitative analyses of both the effort and strategies used by individuals when provided with quantile dotplots, density plots, interval plots, mean plots, and textual descriptions of uncertainty. We utilize two approaches for examining effort across uncertainty communication techniques: a measure of individual differences in working-memory capacity known as an operation span (OSPAN) task and self-reports of perceived workload via the NASA-TLX. The results reveal that both visualization methods and working-memory capacity impact participants' decisions. Specifically, quantile dotplots and density plots (i.e., distributional annotations) result in more accurate judgments than interval plots, textual descriptions of uncertainty, and mean plots (i.e., summary annotations). Additionally, participants' open-ended responses suggest that individuals viewing distributional annotations are more likely to employ a strategy that explicitly incorporates uncertainty into their judgments than those viewing summary annotations. When comparing quantile dotplots to density plots, this work finds that both methods are equally effective for low-working-memory individuals. However, for individuals with high-working-memory capacity, quantile dotplots evoke more accurate responses with less perceived effort. Given these results, we advocate for the inclusion of converging behavioral and subjective workload metrics in addition to accuracy performance to further disambiguate meaningful differences among visualization techniques.
... persists even after training on interpreting the forecast correctly 30 (see also 21 ). Indeed, both novices 13,[22][23][24][25][26]29 and published researchers in psychology, neuroscience, and medicine misinterpret confidence intervals 27 (for reviews of errors in uncertainty visualization, see 22,31 ). Of the 48 COVID-19 forecast line charts reviewed by Zhang et al. 1 , 60% used confidence intervals to convey uncertainty (see Figure 1 C and E for examples). ...
... Scholars postulate that one of the key reasons some uncertainty visualizations produce misinterpretation is that they evoke dichotomous thinking 22,[31][32][33] . For example, in the case of the Cone of Uncertainty, viewers see the cone as a "danger zone" and believe that when the size of the cone increases, more of the area on the map is within the danger zone 13 . ...
... In actuality, scientists intend the cone to show that they are less sure about where the storm's eye will be over time. Similarly, viewers of other intervals, such as 95% confidence intervals, succumb to dichotomous thinking leading to systematic biases (e.g., 27,29,34,35 , for reviews see, 22,31 ). ...
People worldwide use SARS-CoV-2 (COVID-19) visualizations to make life and death decisions about pandemic risks. Understanding how COVID-19 visualizations influence risk perceptions to improve pandemic communication is crucial. To examine how COVID-19 visualizations influence risk perception, we conducted two experiments ( N = 2,549) during the height of COVID-19 where we presented participants with 34 visualizations from the CDC of the same COVID-19 mortality data. We found that visualizing data using a cumulative scale consistently led to participants believing that they and others were at more risk than before viewing the visualizations. In contrast, visualizing the same data with a weekly incident scale led to variable changes in risk perceptions. Further, uncertainty forecast visualizations also affected risk perceptions, with visualizations showing six or more models increasing risk estimates more than the others tested. Differences between COVID-19 visualizations of the same data produce different risk perceptions, fundamentally changing viewers' interpretation of information.
... Confidence intervals have been shown to be ineffective at conveying uncertainty accurately (Belia et al., 2005;Correll & Gleicher, 2014;Joslyn & LeClerc, 2012;Padilla et al., 2017) and similar concerns might apply to the ellipse stimuli in the present study. Moreover, these displays have the drawback of implying an apparently discrete boundary in what is better understood as a continuous space (Padilla, Castro, et al., 2021). The ellipse stimuli in the present study enclose the central 50% and 95% of the underlying probability mass, but the distribution itself is continuous and there is no special distinction between a point just inside a contour and just outside a contour. ...
... These high correlations indicate that participants' performance relative to those in their condition was stable across these blocks. Understanding measurable correlates of these stable individual differences in probabilistic decisionmaking (Grounds & Joslyn, 2018) and use of visualized uncertainty (Padilla, Castro, et al., 2021), such as underlying skills or dispositional traits, could help us predict who might perform tasks like this better or worse, and it could help us identify additional practice or other interventions that might help improve performance. ...
Background: Every day, people must reason with uncertain information to make decisions that affect their lives and affect the performance of their jobs and organizations. Visualizations of data uncertainty can facilitate these decisions, but visualizations are often misunderstood or misused. Previous research has demonstrated that deliberate practice with uncertainty visualizations can improve decision-making in abstract conditions, but it is not yet known whether the learning gains from this practice will transfer to more concrete, realistic, and complex decision-making tasks. Objective: Here, we test the degree to which practice integrating multiple sources of uncertain information with abstract 2-d summary or ensemble displays improves performance on a similar transfer task involving decision-making with a 3-d virtual sand table. Method: We conducted an online study with 378 participants who completed an uncertainty integration task in a 3-d virtual sand table context using either summary or ensemble displays of uncertainty. Participants had previously practiced with the same display, the other display, or received no opportunity to practice. We analyzed response accuracy and speed and how they changed throughout the task. Results: Results suggest that deliberate practice with abstract uncertainty visualizations allows faster decision making in the new context but does not improve accuracy. In the 3-d task, the summary display generally yielded similar or better performance than the ensemble display. Learning gains from practice transferred to both same-type and different-type visualizations in the 3-d condition. Conclusions: The results suggest that practice in the 2-d task enhanced facility with the underlying probabilistic reasoning in a new context rather than just increasing visualization-specific understanding. This implies that deliberate practice can be a beneficial tool to improve reasoning with uncertainty, including across contexts and across visualization types. Materials: Stimuli, stimulus software, anonymized data, and analysis scripts and related code are available online at https://osf.io/5xdsg/?view_only=8d422629a3784f6a80cfeae40e59a078
... Visual designs for regression validation mean additional graphical elements that are shown to the user in addition to the trend line in the visualization. We consider common visual designs for showing regression results found in literature (e.g., [37,39,53]) and the strategies used by participants in experiment 1 (see Sec. 4.2.5). ...
... Confidence Interval Visualizing the confidence interval of a regression model is common in several different areas and applications, such as pandemic infection projections or weather forecasts [38,46,55]. It shows the uncertainty in the underlying model [37,39]. All models within the confidence interval should be considered valid in a statistical sense. ...
Visual validation of regression models in scatterplots is a common practice for assessing model quality, yet its efficacy remains unquantified. We conducted two empirical experiments to investigate individuals' ability to visually validate linear regression models (linear trends) and to examine the impact of common visualization designs on validation quality. The first experiment showed that the level of accuracy for visual estimation of slope (i.e., fitting a line to data) is higher than for visual validation of slope (i.e., accepting a shown line). Notably, we found bias toward slopes that are "too steep" in both cases. This lead to novel insights that participants naturally assessed regression with orthogonal distances between the points and the line (i.e., ODR regression) rather than the common vertical distances (OLS regression). In the second experiment, we investigated whether incorporating common designs for regression visualization (error lines, bounding boxes, and confidence intervals) would improve visual validation. Even though error lines reduced validation bias, results failed to show the desired improvements in accuracy for any design. Overall, our findings suggest caution in using visual model validation for linear trends in scatterplots.
... A second difference between the transparency maps and the other visualization conditions was the salience of the boundaries between the risk bands. Prior work on human comprehension of data visualizations has demonstrated that people tend to treat things inside of a visual boundary as being categorically different from things outside of the boundary, even when that is not actually the case [PRCR17,NS12,PCH21]. In the hue and isarithmic maps, the boundaries between risk bands were more visually salient than those in the transparency maps, particularly at higher levels of specificity. ...
People frequently make decisions based on uncertain information. Prior research has shown that visualizations of uncertainty can help to support better decision making. However, research has also shown that different representations of the same information can lead to different patterns of decision making. It is crucial for researchers to develop a better scientific understanding of when, why and how different representations of uncertainty lead viewers to make different decisions. This paper seeks to address this need by comparing geospatial visualizations of wildfire risk to verbal descriptions of the same risk. In three experiments, we manipulated the specificity of the uncertain information as well as the visual cues used to encode risk in the visualizations. All three experiments found that participants were more likely to evacuate in response to a hypothetical wildfire if the risk information was presented verbally. When the risk was presented visually, participants were less likely to evacuate, particularly when transparency was used to encode the risk information. Experiment 1 showed that evacuation rates were lower for transparency maps than for other types of visualizations. Experiments 2 and 3 sought to replicate this effect and to test how it related to other factors. Experiment 2 varied the hue used for the transparency maps and Experiment 3 manipulated the salience of the borders between the different risk levels. These experiments showed lower evacuation rates in response to transparency maps regardless of hue. The effect was partially, but not entirely, mitigated by adding salient borders to the transparency maps. Taken together, these experiments show that using transparency to encode information about risk can lead to very different patterns of decision making than other encodings of the same information.
... Developing techniques for data visualization to address data uncertainty is an important research problem, as the interpretability of uncertain data can significantly impact decision-making [JS03,Hul16,PCH21] in domains such as climate and geospatial modeling [MRH * 05, Pan01, KCG15, NCA * 19, DM20, WLWC23], medicine [RPHL14,HSSV22], and business intelligence applications [VKKG17]. For example, employing point estimates as part of uncertainty communication has been found to improve decisionmaking in contexts such as weather and transit. ...
Understanding and communicating data uncertainty is crucial for making informed decisions in sectors like finance and healthcare. Previous work has explored how to express uncertainty in various modes. For example, uncertainty can be expressed visually with quantile dot plots or linguistically with hedge words and prosody. Our research aims to systematically explore how variations within each mode contribute to communicating uncertainty to the user; this allows us to better understand each mode's affordances and limitations. We completed an exploration of the uncertainty design space based on pilot studies and ran two crowdsourced experiments examining how speech, text, and visualization modes and variants within them impact decision‐making with uncertain data. Visualization and text were most effective for rational decision‐making, though text resulted in lower confidence. Speech garnered the highest trust despite sometimes leading to risky decisions. Results from these studies indicate meaningful trade‐offs among modes of information and encourage exploration of multimodal data representations.
... Likewise, there is a lot of existing research on how graphs can convey the uncertainty of numerical information (e.g., Padilla et al., 2021). This, too, could be studied multimodally. ...
Modern society depends on numerical information, which must be communicated accurately and effectively. Numerical communication is accomplished in different modalities—speech, writing, sign, gesture, graphs, and in naturally occurring settings it almost always involves more than one modality at once. Yet the modalities of numerical communication are often studied in isolation. Here we argue that, to understand and improve numerical communication, we must take seriously this multimodality. We first discuss each modality on its own terms, identifying their commonalities and differences. We then argue that numerical communication is shaped critically by interactions among modalities. We boil down these interactions to four types: one modality can amplify the message of another; it can direct attention to content from another modality (e.g., using a gesture to guide attention to a relevant aspect of a graph); it can explain another modality (e.g., verbally explaining the meaning of an axis in a graph); and it can reinterpret a modality (e.g., framing an upwards-oriented trend as a bad outcome). We conclude by discussing how a focus on multimodality raises entirely new research questions about numerical communication.
... For example, logarithmic and threshold-based quantities often have asymmetric or multimodal distributions. Graphically, uncertainty is often represented as an error bar in scatter or bar plots and as a shaded area in line plots, although many variations exist [26]. ...
Uncertainty is an inherent aspect of aquatic remote sensing, originating from sources such as sensor noise, atmospheric variability, and human error. Although many studies have advanced the understanding of uncertainty, it is still not incorporated routinely into aquatic remote sensing research. Neglecting uncertainty can lead to misinterpretations of results, missed opportunities for innovative research, and a limited understanding of complex aquatic systems. In this article, we demonstrate how working with uncertainty can advance remote sensing through three examples: validation and match-up analysis, targeted improvement of data products, and decision-making based on information acquired through remote sensing. We advocate for a change of perspective: the uncertainty inherent in aquatic remote sensing should be embraced, rather than viewed as a limitation. Focusing on uncertainty not only leads to more accurate and reliable results but also paves the way for innovation through novel insights, product improvements, and more informed decision-making in the management and preservation of aquatic ecosystems.
... The FRM4SOC project 22 improved the state of the art by standardising protocols, intercomparing commonly used spectroradiometers, and improving methods for uncertainty estimation and propagation [92,97,108,110,114,120]. Additional recent research into uncertainty has included standardisation of robust comparison metrics [406,407] and terminology [472], identification and quantification of individual contributors to the overall uncertainty budget [208,218], and improvements to the visualisation and communication of uncertainty [278,473]. The most important recommendation for the future is to treat uncertainty as an integral part of the measurement process, meaning uncertainty and error should be characterised as comprehensively as possible, reported as consistently as possible, and propagated as accurately as possible. ...
Water is all around us and is vital for all aspects of life. Studying the various compounds and life forms that inhabit natural waters lets us better understand the world around us.
Remote sensing enables global measurements with rapid response and high consistency. Citizen science provides new knowledge and greatly increases the scientific and social impact of research.
In this thesis, we investigate several aspects of citizen science and remote sensing of water, with a focus on uncertainty and accessibility. We improve existing techniques and develop new methods to use smartphone cameras for accessible remote sensing of water.
... 35/100). Several studies have shown that icon-arrays are an effective method for communicating risk, such as simple ratio-based probability values [33]. This part-towhole representation of proportion refects the frequency of events and chances, providing a visual affordances for audiences with different statistical and visualization experience to grasp. ...
Visualizations today are used across a wide range of languages and cultures. Yet the extent to which language impacts how we reason about data and visualizations remains unclear. In this paper, we explore the intersection of visualization and language through a cross-language study on estimative probability tasks with icon-array visualizations. Across Arabic, English, French, German, and Mandarin, n = 50 participants per language both chose probability expressions - e.g. likely, probable - to describe icon-array visualizations (Vis-to-Expression), and drew icon-array visualizations to match a given expression (Expression-to-Vis). Results suggest that there is no clear one-to-one mapping of probability expressions and associated visual ranges between languages. Several translated expressions fell significantly above or below the range of the corresponding English expressions. Compared to other languages, French and German respondents appear to exhibit high levels of consistency between the visualizations they drew and the words they chose. Participants across languages used similar words when describing scenarios above 80% chance, with more variance in expressions targeting mid-range and lower values. We discuss how these results suggest potential differences in the expressiveness of language as it relates to visualization interpretation and design goals, as well as practical implications for translation efforts and future studies at the intersection of languages, culture, and visualization. Experiment data, source code, and analysis scripts are available at the following repository: https://osf.io/g5d4r/.
... Instead, our work focuses on the relative change in risk judgments as a critical component in determining the impact of visualizations on people's understanding of risk. Additionally, although we tested many data visualization types (34 visualizations total), this work did not comprehensively study every modern uncertainty visualization approach (for reviews, see 31,48 ). Further, while we included eight individual differences measures, which we selected based on prior COVID-19 research, we did not exhaustively test every individual differences measure that may have influenced participants' risk judgments. ...
People worldwide use SARS-CoV-2 (COVID-19) visualizations to make life and death decisions about pandemic risks. Understanding how these visualizations influence risk perceptions to improve pandemic communication is crucial. To examine how COVID-19 visualizations influence risk perception, we conducted two experiments online in October and December of 2020 (N = 2549) where we presented participants with 34 visualization techniques (available at the time of publication on the CDC’s website) of the same COVID-19 mortality data. We found that visualizing data using a cumulative scale consistently led to participants believing that they and others were at more risk than before viewing the visualizations. In contrast, visualizing the same data with a weekly incident scale led to variable changes in risk perceptions. Further, uncertainty forecast visualizations also affected risk perceptions, with visualizations showing six or more models increasing risk estimates more than the others tested. Differences between COVID-19 visualizations of the same data produce different risk perceptions, fundamentally changing viewers’ interpretation of information.
... Instead, our work focuses on the relative change in risk judgments as a critical component in determining the impact of visualizations on people's understanding of risk. Additionally, although we tested many data visualization types (34 visualizations total), this work did not comprehensively study every modern uncertainty visualization approach (for reviews, see 31,48 ). Further, while we included eight individual differences measures, which we selected based on prior COVID-19 research, we did not exhaustively test every individual differences measure that may have influenced participants' risk judgments. ...
Policy-makers and the general public have made decisions using COVID-19 data visualizations that have affected the health of the global population. However, the impact that such wide use of data visualizations has had on people's beliefs about their personal risk for COVID-19 is unclear. We conducted two experiments (N = 2,549) during the height of the COVID-19 epidemic in the United States to examine if real-time COVID-19 visualizations influenced participants' beliefs about the risk of the pandemic to themselves and others. This work also examined the impact of two elements of COVID-19 data visualizations, data properties (cumulative- vs. incident-death metrics) and uncertainty visualization techniques (historical data only, and forecasts with no uncertainty, vs. nine uncertainty visualization techniques). The results revealed that viewing COVID-19 visualizations with rising trends resulted in participants believing themselves and others at greater risk than before viewing the COVID-19 visualizations. Further, uncertainty visualization techniques that showed six or more models evoked the largest increases in risk estimates compared to the visualizations tested. These results could inform the design of public pandemic risk communication.
Visual validation of regression models in scatterplots is a common practice for assessing model quality, yet its efficacy remains unquantified. We conducted two empirical experiments to investigate individuals' ability to visually validate linear regression models (linear trends) and to examine the impact of common visualization designs on validation quality. The first experiment showed that the level of accuracy for visual estimation of slope (i.e., fitting a line to data) is higher than for visual validation of slope (i.e., accepting a shown line). Notably, we found bias toward slopes that are “too steep” in both cases. This lead to novel insights that participants naturally assessed regression with orthogonal distances between the points and the line (i.e., ODR regression) rather than the common vertical distances (OLS regression). In the second experiment, we investigated whether incorporating common designs for regression visualization (error lines, bounding boxes, and confidence intervals) would improve visual validation. Even though error lines reduced validation bias, results failed to show the desired improvements in accuracy for any design. Overall, our findings suggest caution in using visual model validation for linear trends in scatterplots
Non-expert users often find it challenging to perceive the reliability of computer vision systems accurately. In human–computer decision-making applications, users’ perceptions of system reliability may deviate from the probabilistic characteristics. Intuitive visualization of system recognition results within probability distributions can serve to enhance interpretability and support cognitive processes. Different visualization formats may impact users’ reliability perceptions and cognitive abilities. This study first compared the mapping relationship between users’ perceived values of system recognition results and the actual probabilistic characteristics of the distribution when using density strips, violin plots, and error bars to visualize normal distributions. The findings indicate that when density strips are used for visualization, users’ perceptions align most closely with the probabilistic integrals, exhibiting the shortest response times and highest cognitive arousal. However, users’ perceptions often exceed the actual probability density, with an average coefficient of 2.53 times, unaffected by the form of uncertainty visualization. Conversely, this perceptual bias did not appear in triangular distributions and remained consistent across symmetric and asymmetric distributions. The results of this study contribute to a better understanding of user reliability perception for interaction designers, helping to improve uncertainty visualization and thereby mitigate perceptual biases and potential trust risks.
In responsible AI development, the construction of AI systems with well-designed transparency and the capability to achieve transparency-adaptive adjustments necessitates a clear and quantified understanding of user states during the interaction process. Among these, trust and load are two important states of the user’s internal psychology, albeit often challenging to directly ascertain. Thus, this study employs transparency experiments involving multiple probabilistic indicators to capture users’ compliance and reaction times during the interactive collaboration process of receiving real-time feedback. Subsequently, estimations of trust and load states are established, leading to the further development of a state transition matrix. Through the establishment of a trust–workload model, probabilistic estimations of user states under varying levels of transparency are obtained, quantitatively delineating the evolution of states and transparency within interaction sequences. This research lays the groundwork for subsequent endeavors in optimal strategy formulation and the development of transparency dynamically adaptive adjustment strategies based on the trust–workload state model constraints.
Visualizing uncertainty is a difficult but important task. Many techniques for visualizing uncertainty are designed for a specific domain, such as cartography or scientific visualization, and the effectiveness of these techniques is tested within that domain (when it is tested at all). This makes it difficult to generalize the findings to other tasks and domains. Recent work in visualization psychology has begun to focus on this problem from the perspective of how different visualization techniques impact human cognitive processes, including perception, memory, and decision-making. Taking this perspective allows researchers to develop theories that can generalize across domains. This is a rich area for research, but given the large number of papers about uncertainty visualization, it can be difficult to know where to begin. The goal of this chapter is to provide a broad overview of what kinds of uncertainty visualization techniques have been developed in different domains, which ones have been evaluated with respect to their impact on human cognition, and where important gaps remain.
Visualizations today are used across a wide range of languages and cultures. Yet the extent to which language impacts how we reason about data and visualizations remains unclear. In this paper, we explore the intersection of visualization and language through a cross-language study on estimative probability tasks with icon-array visualizations. Across Arabic, English, French, German, and Mandarin,
n=50
participants per language both chose probability expressions —
e.g. likely, probable
— to describe icon-array visualizations (Vis-to-Expression), and drew icon-array visualizations to match a given expression (Expression-to-Vis). Results suggest that there is no clear one-to-one mapping of probability expressions and associated visual ranges between languages. Several translated expressions fell significantly above or below the range of the corresponding English expressions. Compared to other languages, French and German respondents appear to exhibit high levels of consistency between the visualizations they drew and the words they chose. Participants across languages used similar words when describing scenarios above 80% chance, with more variance in expressions targeting mid-range and lower values. We discuss how these results suggest potential differences in the expressiveness of language as it relates to visualization interpretation and design goals, as well as practical implications for translation efforts and future studies at the intersection of languages, culture, and visualization. Experiment data, source code, and analysis scripts are available at the following repository:
https://osf.io/g5d4r/
.
While uncertainty is present in most data analysis pipelines, reasoning with uncertainty is challenging for novices and experts alike. Fortunately, researchers are making significant advancements in the communication of uncertainty. In this article, we detail new visualization methods and emerging cognitive theories that describe how we reason with visual representations of uncertainty. We describe the best practices in uncertainty visualization and the psychology behind how each approach supports viewers' judgments. This article begins with a brief overview of conventional and state‐of‐the‐art uncertainty visualization techniques. Then, we take an in‐depth look at the pros and cons of each technique using cognitive theories that describe why and how the mind processes different types of uncertainty information.
There is a growing body of evidence that numerical uncertainty expressions can be used by non-experts to improve decision quality. Moreover, there is some evidence that similar advantages extend to graphic expressions of uncertainty. However, visualizing uncertainty introduces challenges as well. Here, we discuss key misunderstandings that may arise from uncertainty visualizations, in particular the evidence that users sometimes fail to realize that the graphic depicts uncertainty. Instead they have a tendency to interpret the image as representing some deterministic quantity. We refer to this as the deterministic construal error. Although there is now growing evidence for the deterministic construal error, few studies are designed to detect it directly because they inform participants upfront that the visualization expresses uncertainty. In a natural setting such cues would be absent, perhaps making the deterministic assumption more likely. Here we discuss the psychological roots of this key but underappreciated misunderstanding as well as possible solutions. This is a critical question because it is now clear that members of the public understand that predictions involve uncertainty and have greater trust when uncertainty is included. Moreover, they can understand and use uncertainty predictions to tailor decisions to their own risk tolerance, as long as they are carefully expressed, taking into account the cognitive processes involved.
When forecasting events, multiple types of uncertainty are often inherently present in the modeling process. Various uncertainty typologies exist, and each type of uncertainty has different implications a scientist might want to convey. In this work, we focus on one type of distinction between direct quantitative uncertainty and indirect qualitative uncertainty. Direct quantitative uncertainty describes uncertainty about facts, numbers, and hypotheses that can be communicated in absolute quantitative forms such as probability distributions or confidence intervals. Indirect qualitative uncertainty describes the quality of knowledge concerning how effectively facts, numbers, or hypotheses represent reality, such as evidence confidence scales proposed by the Intergovernmental Panel on Climate Change. A large body of research demonstrates that both experts and novices have difficulty reasoning with quantitative uncertainty, and visualizations of uncertainty can help with such traditionally challenging concepts. However, the question of if, and how, people may reason with multiple types of uncertainty associated with a forecast remains largely unexplored. In this series of studies, we seek to understand if individuals can integrate indirect uncertainty about how “good” a model is (operationalized as a qualitative expression of forecaster confidence) with quantified uncertainty in a prediction (operationalized as a quantile dotplot visualization of a predicted distribution). Our first study results suggest that participants utilize both direct quantitative uncertainty and indirect qualitative uncertainty when conveyed as quantile dotplots and forecaster confidence. In manipulations where forecasters were less sure about their prediction, participants made more conservative judgments. In our second study, we varied the amount of quantified uncertainty (in the form of the SD of the visualized distributions) to examine how participants’ decisions changed under different combinations of quantified uncertainty (variance) and qualitative uncertainty (low, medium, and high forecaster confidence). The second study results suggest that participants updated their judgments in the direction predicted by both qualitative confidence information (e.g., becoming more conservative when the forecaster confidence is low) and quantitative uncertainty (e.g., becoming more conservative when the variance is increased). Based on the findings from both experiments, we recommend that forecasters present qualitative expressions of model confidence whenever possible alongside quantified uncertainty.
Bayesian reasoning is common and critical in everyday life while the performance on Bayesian reasoning is rather poor. Previous studies showed that people could enhance their performance by applying cognitive resources under the natural frequency format condition. Working memory is one of the crucial cognitive resources in the reasoning process. However, the role of working memory on Bayesian reasoning remains unclear. In our study, we verified the effect of working memory on Bayesian reasoning by evaluating the performance of participants with high and low working memory span (WMS); we also investigated if working memory as a kind of cognitive resource can affect Bayesian reasoning performance by manipulating the cognitive load in a dual-task paradigm among participants with no-, low-, and high-loads. We found the following: (1) The Bayesian reasoning performance of high WMS participants was significantly higher than that of low WMS participants. (2) Performance under natural frequency condition was noticeably higher than that in standard probability condition. (3) Interaction between working memory and probability format was significant, and the performance of participants with high-load in natural frequency condition was higher when compared to those of participants with no- and low-load. Therefore, we can conclude that: (1) Working memory resource is a major factor in Bayesian reasoning. The performance of Bayesian reasoning is influenced by working memory span and working memory load. (2) A Bayesian facilitation effect exists, and replacing the standard probability format with a natural frequency format can significantly improve Bayesian performance. (3) Bayesian facilitation occurs only in participants with sufficient working memory resources.
Given the widespread use of visualizations to communicate hazard risks, forecast visualizations must be as effective to interpret as possible. However, despite incorporating best practices, visualizations can influence viewer judgments in ways that the designers did not anticipate. Visualization designers should understand the full implications of visualization techniques and seek to develop visualizations that account for the complexities in decision-making. The current study explores the influence of visualizations of uncertainty by examining a case in which ensemble hurricane forecast visualizations produce unintended interpretations. We show that people estimate more damage to a location that is overlapped by a track in an ensemble hurricane forecast visualization compared to a location that does not coincide with a track. We find that this effect can be partially reduced by manipulating the number of hurricane paths displayed, suggesting the importance of visual features of a display on decision making. Providing instructions about the information conveyed in the ensemble display also reduced the effect, but importantly, did not eliminate it. These findings illustrate the powerful influence of marks and their encodings on decision-making with visualizations. (PsycINFO Database Record (c) 2019 APA, all rights reserved).
Cognitive science has established widely used and validated procedures for evaluating working memory in numerous applied domains, but surprisingly few studies have employed these methodologies to assess claims about the impacts of visualizations on working memory. The lack of information visualization research that uses validated procedures for measuring working memory may be due, in part, to the absence of cross-domain methodological guidance tailored explicitly to the unique needs of visualization research. This paper presents a set of clear, practical, and empirically validated methods for evaluating working memory during visualization tasks and provides readers with guidance in selecting an appropriate working memory evaluation paradigm. As a case study, we illustrate multiple methods for evaluating working memory in a visual-spatial aggregation task with geospatial data. The results show that the use of dual-task experimental designs (simultaneous performance of several tasks compared to single-task performance) and pupil dilation can reveal working memory demands associated with task difficulty and dual-tasking. In a dual-task experimental design, measures of task completion times and pupillometry revealed the working memory demands associated with both task difficulty and dual-tasking. Pupillometry demonstrated that participants' pupils were significantly larger when they were completing a more difficult task and when multitasking. We propose that researchers interested in the relative differences in working memory between visualizations should consider a converging methods approach, where physiological measures and behavioral measures of working memory are employed to generate a rich evaluation of visualization effort.
A viewer can extract many potential patterns from any set of visualized data values. But that means that two people can see different patterns in the same visualization, potentially leading to miscommunication. Here, we show that when people are primed to see one pattern in the data as visually salient, they believe that nave viewers will experience the same visual salience. Participants were told one of multiple backstories about political events that affected public polling data, before viewing a graph that depicted those data. One pattern in the data was particularly visually salient to them given the backstory that they heard. They then predicted what nave viewers would find most visually salient on the visualization. They were strongly influenced by their own knowledge, despite explicit instructions to ignore it, predicting that others would find the same patterns to be most visually salient. This result reflects a psychological phenomenon known as the curse of knowledge, where an expert struggles to re-create the state of mind of a novice. The present findings show that the curse of knowledge also plagues the visual perception of data, explaining why people can fail to connect with audiences when they communicate patterns in data.
During the 2017 Spring Forecasting Experiment in NOAA’s Hazardous Weather Testbed, 62 meteorologists completed a survey designed to test their understanding of forecast uncertainty. Survey questions were based on probabilistic forecast guidance provided by the NSSL Experimental Warn-on-Forecast System for ensembles (NEWS-e). A mix of 20 multiple-choice and open-ended questions required participants to explain basic probability and percentile concepts, extract information using graphical representations of uncertainty, and determine what type of weather scenario the graphics depicted. Multiple-choice questions were analyzed using frequency counts, and open-ended questions were analyzed using thematic coding methods. Of the 18 questions that could be scored, 60%–96% of the participants’ responses aligned with the researchers’ intended response. Some of the most challenging questions proved to be those requiring qualitative explanations, such as to explain what the 70th-percentile value of accumulated rainfall represents in an ensemble-based probabilistic forecast. Additionally, participants providing answers not aligning with the intended response oftentimes appeared to consider the given information with a deterministic rather than probabilistic mindset. Applications of a deterministic mindset resulted in tendencies to focus on the worst-case scenario and to modify understanding of probabilistic concepts when presented with different variables. The findings from this survey support the need for improved basic and applied training for the development, interpretation, and use of probabilistic ensemble forecast guidance. Future work should collect data for a larger sample size to examine the knowledge gaps across specific user groups and to guide development of probabilistic forecast training tools.
Visualizations—visual representations of information, depicted in graphics—are studied by researchers in numerous ways, ranging from the study of the basic principles of creating visualizations, to the cognitive processes underlying their use, as well as how visualizations communicate complex information (such as in medical risk or spatial patterns). However, findings from different domains are rarely shared across domains though there may be domain-general principles underlying visualizations and their use. The limited cross-domain communication may be due to a lack of a unifying cognitive framework. This review aims to address this gap by proposing an integrative model that is grounded in models of visualization comprehension and a dual-process account of decision making. We review empirical studies of decision making with static two-dimensional visualizations motivated by a wide range of research goals and find significant direct and indirect support for a dual-process account of decision making with visualizations. Consistent with a dual-process model, the first type of visualization decision mechanism produces fast, easy, and computationally light decisions with visualizations. The second facilitates slower, more contemplative, and effortful decisions with visualizations. We illustrate the utility of a dual-process account of decision making with visualizations using four cross-domain findings that may constitute universal visualization principles. Further, we offer guidance for future research, including novel areas of exploration and practical recommendations for visualization designers based on cognitive theory and empirical findings.
Understanding uncertainty is critical for many analytical tasks. One common approach is to encode data values and uncertainty values independently, using two visual variables. These resulting bivariate maps can be difficult to interpret, and interference between visual channels can reduce the discriminability of marks. To address this issue, we contribute Value-Suppressing Uncertainty Palettes (VSUPs). VSUPs allocate larger ranges of a visual channel to data when uncertainty is low, and smaller ranges when uncertainty is high. This non-uniform budgeting of the visual channels makes more economical use of the limited visual encoding space when uncertainty is low, and encourages more cautious decision-making when uncertainty is high. We demonstrate several examples of VSUPs, and present a crowdsourced evaluation showing that, compared to traditional bivariate maps, VSUPs encourage people to more heavily weight uncertainty information in decision-making tasks.
Visualizations of uncertainty in data are often presented to the public without explanations of their graphical conventions and are often misunderstood by nonexperts. The "cone of uncertainty" used to visualize hurricane forecasts is a case in point. Here we examined the effects of explaining graphical conventions on understanding of the cone of uncertainty. In two experiments, participants were given instructions with and without an explanation of these graphical conventions. We examined the effect of these instructions on both explicit statements of common misconceptions and users interpretation of hurricane forecasts, specifically their predictions of damage from the hurricane over space and time. Enhanced instructions reduced misconceptions about the cone of uncertainty as expressed in explicit beliefs, and in one experiment also reduced predictions of damage overall. Examination of individual response profiles for the damage estimate task revealed qualitative differences between individuals that were not evident in aggregate response profiles. This research reveals mixed results for the effectiveness of instructions on comprehension of uncertainty visualizations and suggests a more nuanced approach that focuses on the individual's knowledge and beliefs about the domain and visualization.
Ensemble and summary displays are two widely used methods to represent visual-spatial uncertainty; however, there is disagreement about which is the most effective technique to communicate uncertainty to the general public. Visualization scientists create ensemble displays by plotting multiple data points on the same Cartesian coordinate plane. Despite their use in scientific practice, it is more common in public presentations to use visualizations of summary displays, which scientists create by plotting statistical parameters of the ensemble members. While prior work has demonstrated that viewers make different decisions when viewing summary and ensemble displays, it is unclear what components of the displays lead to diverging judgments. This study aims to compare the salience of visual features – or visual elements that attract bottom-up attention – as one possible source of diverging judgments made with ensemble and summary displays in the context of hurricane track forecasts. We report that salient visual features of both ensemble and summary displays influence participant judgment. Specifically, we find that salient features of summary displays of geospatial uncertainty can be misunderstood as displaying size information. Further, salient features of ensemble displays evoke judgments that are indicative of accurate interpretations of the underlying probability distribution of the ensemble data. However, when participants use ensemble displays to make point-based judgments, they may overweight individual ensemble members in their decision-making process. We propose that ensemble displays are a promising alternative to summary displays in a geospatial context but that decisions about visualization methods should be informed by the viewer’s task.
The topic of working memory (WM) is ubiquitous in research on cognitive psychology and on individual differences. According to one definition, it is a small amount of information kept in a temporary state of heightened accessibility; it is used in most types of communication and problem solving. Short-term storage has been defined as the passive (i.e., non-attention-based, nonstrategic) component of WM or, alternatively, as a passive store separate from an attention-based WM. Here I note that much confusion has been created by the use by various investigators of many, subtly different definitions of WM and short-term storage. The definitions are sometimes made explicit and sometimes implied. As I explain, the different definitions may have stemmed from the use of a wide variety of techniques to explore WM, along with differences in theoretical orientation. By delineating nine previously used definitions of WM and explaining how additional ones may emerge from combinations of these nine, I hope to improve scientific discourse on WM. The potential advantages of clarity about definitions of WM and short-term storage are illustrated with respect to several ongoing research controversies.
Many visual depictions of probability distributions, such as error bars, are difficult for users to accurately interpret. We present and study an alternative representation, Hypothetical Outcome Plots (HOPs), that animates a finite set of individual draws. In contrast to the statistical background required to interpret many static representations of distributions, HOPs require relatively little background knowledge to interpret. Instead, HOPs enables viewers to infer properties of the distribution using mental processes like counting and integration. We conducted an experiment comparing HOPs to error bars and violin plots. With HOPs, people made much more accurate judgments about plots of two and three quantities. Accuracy was similar with all three representations for most questions about distributions of a single quantity.
Many techniques have been proposed for visualizing uncertainty in geospatial data. Previous empirical research on the effectiveness of visualizations of geospatial uncertainty has focused primarily on user intuitions rather than objective measures of performance when reasoning under uncertainty. Framed in the context of Google’s blue dot, we examined the effectiveness of four alternative visualizations for representing positional uncertainty when reasoning about self-location data. Our task presents a mobile mapping scenario in which GPS satellite location readings produce location estimates with varying levels of uncertainty. Given a known location and two smartphone estimates of that known location, participants were asked to judge which smartphone produces the better location reading, taking uncertainty into account. We produced visualizations that vary by glyph type (uniform blue circle with border vs. Gaussian fade) and visibility of a centroid dot (visible vs. not visible) to produce the four visualization formats. Participants viewing the uniform blue circle are most likely to respond in accordance with the actual probability density of points sampled from bivariate normal distributions and additionally respond most rapidly. Participants reported a number of simple heuristics on which they based their judgments, and consistency with these heuristics was highly predictive of their judgments.
Complex span and visual arrays are two common measures of working memory capacity that are respectively treated as measures of attention control and storage capacity. A recent analysis of these tasks concluded that (1) complex span performance has a relatively stronger relationship to fluid intelligence and (2) this is due to the requirement that people engage control processes while performing this task. The present study examines the validity of these conclusions by examining two large data sets that include a more diverse set of visual arrays tasks and several measures of attention control. We conclude that complex span and visual arrays account for similar amounts of variance in fluid intelligence. The disparity relative to the earlier analysis is attributed to the present study involving a more complete measure of the latent ability underlying the performance of visual arrays. Moreover, we find that both types of working memory task have strong relationships to attention control. This indicates that the ability to engage attention in a controlled manner is a critical aspect of working memory capacity, regardless of the type of task that is used to measure this construct.
Understanding how people interpret and use visually presented uncertainty data is an important yet seldom studied aspect of data visualization applications. Current approaches in visualization often display uncertainty as an additional data attribute without a well-defined context. Our goal was to test whether different graphical displays (glyphs) would influence a decision about which of 2 weather forecasts was a more accurate predictor of an uncertain temperature forecast value. We used a statistical inference task based on fictional univariate normal distributions, each characterized by a mean and standard deviation. Participants viewed 1 of 5 different glyph types representing 2 weather forecast distributions. Three of these used variations in spatial encoding to communicate the distributions and the other 2 used nonspatial encoding (brightness or color). Four distribution pairs were created with different relative standard deviations (uncertainty of the forecasts). We found that there was a difference in how decisions were made with spatial versus nonspatial glyphs, but no difference among the spatial glyphs themselves. Furthermore, the effect of different glyph types changed as a function of the variability of the distributions. The results are discussed in the context of how visualizations might improve decision making under uncertainty. (PsycINFO Database Record (c) 2014 APA, all rights reserved).
We show how brain sensing can lend insight to the evaluation of visual interfaces and establish a role for fNIRS in visualization. Research suggests that the evaluation of visual design benefits by going beyond performance measures or questionnaires to measurements of the user's cognitive state. Unfortunately, objectively and unobtrusively monitoring the brain is difficult. While functional near-infrared spectroscopy (fNIRS) has emerged as a practical brain sensing technology in HCI, visual tasks often rely on the brain's quick, massively parallel visual system, which may be inaccessible to this measurement. It is unknown whether fNIRS can distinguish differences in cognitive state that derive from visual design alone. In this paper, we use the classic comparison of bar graphs and pie charts to test the viability of fNIRS for measuring the impact of a visual design on the brain. Our results demonstrate that we can indeed measure this impact, and furthermore measurements indicate that there are not universal differences in bar graphs and pie charts.
This paper presents two linked empirical studies focused on uncertainty visualization. The experiments are framed from two conceptual perspectives. First, a typology of uncertainty is used to delineate kinds of uncertainty matched with space, time, and attribute components of data. Second, concepts from visual semiotics are applied to characterize the kind of visual signification that is appropriate for representing those different categories of uncertainty. This framework guided the two experiments reported here. The first addresses representation intuitiveness, considering both visual variables and iconicity of representation. The second addresses relative performance of the most intuitive abstract and iconic representations of uncertainty on a map reading task. Combined results suggest initial guidelines for representing uncertainty and discussion focuses on practical applicability of results.
Dual-process and dual-system theories in both cognitive and social psychology have been subjected to a number of recently published criticisms. However, they have been attacked as a category, incorrectly assuming there is a generic version that applies to all. We identify and respond to 5 main lines of argument made by such critics. We agree that some of these arguments have force against some of the theories in the literature but believe them to be overstated. We argue that the dual-processing distinction is supported by much recent evidence in cognitive science. Our preferred theoretical approach is one in which rapid autonomous processes (Type 1) are assumed to yield default responses unless intervened on by distinctive higher order reasoning processes (Type 2). What defines the difference is that Type 2 processing supports hypothetical thinking and load heavily on working memory.
© The Author(s) 2013.
Data graphics can be a powerful aid to decision-making—if they are designed to mesh well with human vision and understanding. Perceiving data values can be more precise for some graphical types, such as a scatterplot, and less precise for others, such as a heatmap. The eye can extract some types of statistics from large arrays in an eyeblink, as quickly as recognizing an object or face. But perceiving some patterns in visualized numbers—particularly comparisons within a dataset—is slow and effortful, unfolding over a series of operations that are guided by attention and previous experience. Effective data graphics map important messages onto visual patterns that are easily extracted, likely to be attended, and as consistent as possible with the audience’s previous experience. User-centered design methods, which rely on iteration and experimentation to improve a design, are critical tools for creating effective data visualizations.
People naturally bring their prior beliefs to bear on how they interpret the new information, yet few formal models exist for accounting for the influence of users' prior beliefs in interactions with data presentations like visualizations. We demonstrate a Bayesian cognitive model for understanding how people interpret visualizations in light of prior beliefs and show how this model provides a guide for improving visualization evaluation. In a first study, we show how applying a Bayesian cognition model to a simple visualization scenario indicates that people's judgments are consistent with a hypothesis that they are doing approximate Bayesian inference. In a second study, we evaluate how sensitive our observations of Bayesian behavior are to different techniques for eliciting people subjective distributions, and to different datasets. We find that people don't behave consistently with Bayesian predictions for large sample size datasets, and this difference cannot be explained by elicitation technique. In a final study, we show how normative Bayesian inference can be used as an evaluation framework for visualizations, including of uncertainty.
Animated representations of outcomes drawn from distributions (hypothetical outcome plots, or HOPs) are used in the media and other public venues to communicate uncertainty. HOPs greatly improve multivariate probability estimation over conventional static uncertainty visualizations and leverage the ability of the visual system to quickly, accurately, and automatically process the summary statistical properties of ensembles. However, it is unclear how well HOPs support applied tasks resembling real world judgments posed in uncertainty communication. We identify and motivate an appropriate task to investigate realistic judgments of uncertainty in the public domain through a qualitative analysis of uncertainty visualizations in the news. We contribute two crowdsourced experiments comparing the effectiveness of HOPs, error bars, and line ensembles for supporting perceptual decision-making from visualized uncertainty. Participants infer which of two possible underlying trends is more likely to have produced a sample of time series data by referencing uncertainty visualizations which depict the two trends with variability due to sampling error. By modeling each participant's accuracy as a function of the level of evidence presented over many repeated judgments, we find that observers are able to correctly infer the underlying trend in samples conveying a lower level of evidence when using HOPs rather than static aggregate uncertainty visualizations as a decision aid. Modeling approaches like ours contribute theoretically grounded and richly descriptive accounts of user perceptions to visualization evaluation.
Everyday predictive systems typically present point predictions, making it hard for people to account for uncertainty when making decisions. Evaluations of uncertainty displays for transit prediction have assessed people’s ability to extract probabilities, but not the quality of their decisions. In a controlled, incentivized experiment, we had subjects decide when to catch a bus using displays with textual uncertainty, uncer tainty visualizations, or no-uncertainty (control). Frequency- based visualizations previously shown to allow people to bet ter extract probabilities (quantile dotplots) yielded better decisions. Decisions with quantile dotplots with 50 outcomes were (1) better on average, having expected payoffs 97% of optimal (95% CI: [95%,98%]), 5 percentage points more than con trol (95% CI: [2,8]); and (2) more consistent, having within- subject standard deviation of 3 percentage points (95% CI: [2,4]), 4 percentage points less than control (95% CI: [2,6]). Cumulative distribution function plots performed nearly as well, and both outperformed textual uncertainty, which was sensitive to the probability interval communicated. We discuss implications for realtime transit predictions and possible generalization to other domains.
People often have erroneous intuitions about the results of uncertain processes, such as scientific experiments. Many uncertainty visualizations assume considerable statistical knowledge, but have been shown to prompt erroneous conclusions even when users possess this knowledge. Active learning approaches been shown to improve statistical reasoning, but are rarely applied in visualizing uncertainty in scientific reports. We present a controlled study to evaluate the impact of an interactive, graphical uncertainty prediction technique for communicating uncertainty in experiment results. Using our technique, users sketch their prediction of the uncertainty in experimental effects prior to viewing the true sampling distribution from an experiment. We find that having a user graphically predict the possible effects from experiment replications is an effective way to improve one's ability to make predictions about replications of new experiments. Additionally, visualizing uncertainty as a set of discrete outcomes, as opposed to a continuous probability distribution, can improve recall of a sampling distribution from a single experiment. Our work has implications for various applications where it is important to elicit peoples' estimates of probability distributions and to communicate uncertainty effectively.
This review briefly examines the vast range of techniques used to communicate risk assessments arising from statistical analysis. After discussing essential psychological and sociological issues, I focus on individual health risks and relevant research on communicating numbers, verbal expressions, graphics, and conveying deeper uncertainty. I then consider practice in a selection of diverse case studies, including gambling, the benefits and risks of pharmaceuticals, weather forecasting, natural hazards, climate change, environmental exposures, security and intelligence, industrial reliability, and catastrophic national and global risks. There are some tentative final conclusions, but the primary message is to acknowledge expert guidance, be clear about objectives, and work closely with intended audiences.
Background
Effective risk communication is essential for informed decision making. Unfortunately, many people struggle to understand typical risk communications because they lack essential decision-making skills.
Objective
The aim of this study was to review the literature on the effect of numeracy on risk literacy, decision making, and health outcomes, and to evaluate the benefits of visual aids in risk communication.
Method
We present a conceptual framework describing the influence of numeracy on risk literacy, decision making, and health outcomes, followed by a systematic review of the benefits of visual aids in risk communication for people with different levels of numeracy and graph literacy. The systematic review covers scientific research published between January 1995 and April 2016, drawn from the following databases: Web of Science, PubMed, PsycINFO, ERIC, Medline, and Google Scholar. Inclusion criteria were investigation of the effect of numeracy and/or graph literacy, and investigation of the effect of visual aids or comparison of their effect with that of numerical information. Thirty-six publications met the criteria, providing data on 27,885 diverse participants from 60 countries.
Results
Transparent visual aids robustly improved risk understanding in diverse individuals by encouraging thorough deliberation, enhancing cognitive self-assessment, and reducing conceptual biases in memory. Improvements in risk understanding consistently produced beneficial changes in attitudes, behavioral intentions, trust, and healthy behaviors. Visual aids were found to be particularly beneficial for vulnerable and less skilled individuals.
Conclusion
Well-designed visual aids tend to be highly effective tools for improving informed decision making among diverse decision makers. We identify five categories of practical, evidence-based guidelines for heuristic evaluation and design of effective visual aids.
This review briefly examines the vast range of techniques used to communicate risk assessments arising from statistical analysis. After discussing essential psychological and sociological issues, I focus on individual health risks and relevant research on communicating numbers, verbal expressions, graphics and conveying deeper uncertainty. I then consider practice in a selection of diverse case studies, including gambling, the benefits and risks of pharmaceuticals, weather forecasting, natural hazards, climate change, environmental exposures, security and intelligence, industrial reliability, and catastrophic national and global risks. There are some tentative final conclusions, but the primary message is to acknowledge expert guidance, be clear about objectives, and to work closely with intended audiences.
Health and medical decisions are based, in part, on the extent to which individuals believe that they are likely to experience a health problem. These risk perceptions can be influenced by healthcare providers and public health practitioners who seek to enable people to make informed medical decisions and encourage them to engage in healthy behaviors. However, meaningful comprehension of risk information involves cognitive, intuitive, and affective processes, as well as contextual features that shape one?s perspective. This complexity makes it difficult to communicate risk information effectively. This chapter provides a very brief theoretical orientation to the concepts of risk perception and communication, and then focuses intensively on specific strategies for avoiding common mistakes made by well-meaning risk communicators: Adhering to basic good communication practices, recognizing limitations in numeracy, defining effective risk communication as more than simple recall of likelihood information, providing specific risk reduction recommendations, and using risk communication strategies that increase comprehension and evaluative meaning.
Data ensembles are often used to infer statistics to be used for a summary display of an uncertain prediction. In a spatial context, these summary displays have the drawback that when uncertainty is encoded via a spatial spread, display glyph area increases in size with prediction uncertainty. This increase can be easily confounded with an increase in the size, strength or other attribute of the phenomenon being presented. We argue that by directly displaying a carefully chosen subset of a prediction ensemble, so that uncertainty is conveyed implicitly, such misinterpretations can be avoided. Since such a display does not require uncertainty annotation, an information channel remains available for encoding additional information about the prediction. We demonstrate these points in the context of hurricane prediction visualizations, showing how we avoid occlusion of selected ensemble elements while preserving the spatial statistics of the original ensemble, and how an explicit encoding of uncertainty can also be constructed from such a selection. We conclude with the results of a cognitive experiment demonstrating that the approach can be used to construct storm prediction displays that significantly reduce the confounding of uncertainty with storm size, and thus improve viewers' ability to estimate potential for storm damage.
Uncertainty represented in visualizations is often ignored or
misunderstood by the non-expert user. The National Hurricane
Center displays hurricane forecasts using a track forecast cone,
depicting the expected track of the storm and the uncertainty in
the forecast. Our goal was to test whether different graphical
displays of a hurricane forecast containing uncertainty would
influence a decision about storm characteristics. Participants
viewed one of five different visualization types. Three varied
the currently used forecast cone, one presented a track with
no uncertainty, and one presented an ensemble of multiple
possible hurricane tracks. Results show that individuals make
different decisions using uncertainty visualizations with different
visual properties, demonstrating that basic visual properties
must be considered in visualization design and communication.
Ensemble coding supports rapid extraction of visual statistics about distributed visual information. Researchers typically study this ability with the goal of drawing conclusions about how such coding extracts information from natural scenes. Here we argue that a second domain can serve as another strong inspiration for understanding ensemble coding: graphs, maps, and other visual presentations of data. Data visualizations allow observers to leverage their ability to perform visual ensemble statistics on distributions of spatial or featural visual information to estimate actual statistics on data. We survey the types of visual statistical tasks that occur within data visualizations across everyday examples, such as scatterplots, and more specialized images, such as weather maps or depictions of patterns in text. We divide these tasks into four categories: identification of sets of values, summarization across those values, segmentation of collections, and estimation of structure. We point to unanswered questions for each category and give examples of such cross-pollination in the current literature. Increased collaboration between the data visualization and perceptual psychology research communities can inspire new solutions to challenges in visualization while simultaneously exposing unsolved problems in perception research.
Users often rely on realtime predictions in everyday contexts like riding the bus, but may not grasp that such predictions are subject to uncertainty. Existing uncertainty visualizations may not align with user needs or how they naturally reason about probability. We present a novel mobile interface design and visualization of uncertainty for transit predictions on mobile phones based on discrete outcomes. To develop it, we identified domain specific design requirements for visualizing uncertainty in transit prediction through: 1) a literature review, 2) a large survey of users of a popular realtime transit application, and 3) an iterative design process. We present several candidate visualizations of uncertainty for realtime transit predictions in a mobile context, and we propose a novel discrete representation of continuous outcomes designed for small screens, quantile dotplots. In a controlled experiment we find that quantile dotplots reduce the variance of probabilistic estimates by ~1.15 times compared to density plots and facilitate more confident estimation by end-users in the context of realtime transit prediction scenarios.
Many decisions are based on beliefs concerning the likelihood of uncertain events such as the outcome of an election, the guilt of a defendant, or the future value of the dollar. Occasionally, beliefs concerning uncertain events are expressed in numerical form as odds or subjective probabilities. In general, the heuristics are quite useful, but sometimes they lead to severe and systematic errors. The subjective assessment of probability resembles the subjective assessment of physical quantities such as distance or size. These judgments are all based on data of limited validity, which are processed according to heuristic rules. However, the reliance on this rule leads to systematic errors in the estimation of distance. This chapter describes three heuristics that are employed in making judgments under uncertainty. The first is representativeness, which is usually employed when people are asked to judge the probability that an object or event belongs to a class or event. The second is the availability of instances or scenarios, which is often employed when people are asked to assess the frequency of a class or the plausibility of a particular development, and the third is adjustment from an anchor, which is usually employed in numerical prediction when a relevant value is available.
For many years, uncertainty visualization has been a topic of research in several disparate fields, particularly in geographical visualization (geovisualization), information visualization, and scientific visualization. Multiple techniques have been proposed and implemented to visually depict uncertainty, but their evaluation has received less attention by the research community. In order to understand how uncertainty visualization influences reasoning and decision-making using spatial information in visual displays, this paper presents a comprehensive review of uncertainty visualization assessments from geovisualization and related fields. We systematically analyze characteristics of the studies under review, i.e., number of participants, tasks, evaluation metrics, etc. An extensive summary of findings with respect to the effects measured or the impact of different visualization techniques helps to identify commonalities and differences in the outcome. Based on this summary, we derive “lessons learned” and provide recommendations for carrying out evaluation of uncertainty visualizations. As a basis for systematic evaluation, we present a categorization of research foci related to evaluating the effects of uncertainty visualization on decision-making. By assigning the studies to categories, we identify gaps in the literature and suggest key research questions for the future. This paper is the second of two reviews on uncertainty visualization. It follows the first that covers the communication of uncertainty, to investigate the effects of uncertainty visualization on reasoning and decision-making.
Working memory comprises a central executive, responsible for reasoning, decision making, and coordinating the activities of two subsidiary systems. One of the subsystems, the articulatory loop, is responsible for retention of verbal material, while the other subsystem, the visuo-spatial scratch pad, is responsible for retention of visual or spatial information. A visual temporary memory system could be responsible for retaining the imaged pattern of numbers in specific locations in a matrix. In contrast, if the subject is genuinely required to retain a sequence of movements, rather than a static pattern, they would be more likely to rely on a spatial temporary memory system. This chapter reports the experiment that was designed to investigate this possibility. In this experiment, subjects were required to retain information from one of two kinds of visually presented displays, either color hues or the sequential order in which a series of squares was presented at different locations on the screen.
Behavioral decision research has demonstrated that judgments and decisions of ordinary people and experts are subject to numerous biases. Decision and risk analysis were designed to improve judgments and decisions and to overcome many of these biases. However, when eliciting model components and parameters from decisionmakers or experts, analysts often face the very biases they are trying to help overcome. When these inputs are biased they can seriously reduce the quality of the model and resulting analysis. Some of these biases are due to faulty cognitive processes; some are due to motivations for preferred analysis outcomes. This article identifies the cognitive and motivational biases that are relevant for decision and risk analysis because they can distort analysis inputs and are difficult to correct. We also review and provide guidance about the existing debiasing techniques to overcome these biases. In addition, we describe some biases that are less relevant because they can be corrected by using logic or decomposing the elicitation task. We conclude the article with an agenda for future research.
© 2015 Society for Risk Analysis.
When making an inference or comparison with uncertain, noisy, or incomplete data, measurement error and confidence intervals can be as important for judgment as the actual mean values of different groups. These often misunderstood statistical quantities are frequently represented by bar charts with error bars. This paper investigates drawbacks with this standard encoding, and considers a set of alternatives designed to more effectively communicate the implications of mean and error data to a general audience, drawing from lessons learned from the use of visual statistics in the information visualization community. We present a series of crowd-sourced experiments that confirm that the encoding of mean and error significantly changes how viewers make decisions about uncertain data. Careful consideration of design tradeoffs in the visual presentation of data results in human reasoning that is more consistently aligned with statistical inferences. We suggest the use of gradient plots (which use transparency to encode uncertainty) and violin plots (which use width) as better alternatives for inferential tasks than bar charts with error bars.
For decades, uncertainty visualisation has attracted attention in disciplines such as cartography and geographic visualisation, scientific visualisation and information visualisation. Most of this research deals with the development of new approaches to depict uncertainty visually; only a small part is concerned with empirical evaluation of such techniques. This systematic review aims to summarize past user studies and describe their characteristics and findings, focusing on the field of geographic visualisation and cartography and thus on displays containing geospatial uncertainty. From a discussion of the main findings, we derive lessons learned and recommendations for future evaluation in the field of uncertainty visualisation. We highlight the importance of user tasks for successful solutions and recommend moving towards task-centered typologies to support systematic evaluation in the field of uncertainty visualisation.
Three experiments demonstrated advantages over conventional deterministic forecasts for participants making temperature estimates and precautionary decisions with predictive interval weather forecasts showing the upper and lower boundaries within which the observed value is expected with a specified probability. Participants using predictive intervals were better able to identify unreliable forecasts, expected a narrower range of outcomes, and were more decisive than were participants using deterministic forecasts. Predictive interval format was also manipulated to determine whether adding visualizations enhanced understanding. Some participants using visualizations misinterpreted predictive intervals as expressions of diurnal fluctuations (deterministic forecasts). Almost no misinterpretations occurred when the predictive interval was expressed in text alone. Moreover, no advantages were found for visualizations over text-only formats, demonstrating that visualizations, especially those investigated in these studies, may not be suitable for expressing this concept. Thus, predictive intervals are both understandable and advantageous to non-expert decision makers, as long as they are carefully expressed. Copyright © 2013 John Wiley & Sons, Ltd.
The term working memory refers to a brain system that provides temporary storage and manipulation of the information necessary
for such complex cognitive tasks as language comprehension, learning, and reasoning. This definition has evolved from the
concept of a unitary short-term memory system. Working memory has been found to require the simultaneous storage and processing
of information. It can be divided into the following three subcomponents: (i) the central executive, which is assumed to be
an attentional-controlling system, is important in skills such as chess playing and is particularly susceptible to the effects
of Alzheimer's disease; and two slave systems, namely (ii) the visuospatial sketch pad, which manipulates visual images and
(iii) the phonological loop, which stores and rehearses speech-based information and is necessary for the acquisition of both
native and second-language vocabulary.
Research has demonstrated that icon arrays (also called "pictographs") are an effective method of communicating risk statistics and appear particularly useful to less numerate and less graphically literate people. Yet research is very limited regarding whether icon type affects how people interpret and remember these graphs.
1502 people age 35-75 from a demographically diverse online panel completed a cardiovascular risk calculator based on Framingham data using their actual age, weight, and other health data. Participants received their risk estimate in an icon array graphic that used 1 of 6 types of icons: rectangular blocks, filled ovals, smile/frown faces, an outline of a person's head and shoulders, male/female "restroom" person icons (gender matched), or actual head-and-shoulder photographs of people of varied races (gender matched). In each icon array, blue icons represented cardiovascular events and gray icons represented those who would not experience an event. We measured perceived risk magnitude, approximate recall, and opinions about the icon arrays, as well as subjective numeracy and an abbreviated measure of graphical literacy.
Risk recall was significantly higher with more anthropomorphic icons (restroom icons, head outlines, and photos) than with other icon types, and participants rated restroom icons as most preferred. However, while restroom icons resulted in the highest correlations between perceived and actual risk among more numerate/graphically literate participants, they performed no better than other icon types among less numerate/graphically literate participants.
Icon type influences both risk perceptions and risk recall, with restroom icons in particular resulting in improved outcomes. However, optimal icon types may depend on numeracy and/or graphical literacy skills.