Allan H. Murphy’s research while affiliated with Oregon State University and other places

In Bayesian inference and decision analysis, inferences and predictions are inherently probabilistic in nature. Scoring rules, which involve the computation of a score based on probability forecasts and what actually occurs, can be used to evaluate probabilities and to provide appropriate incentives for “good” probabilities. This paper review scoring rules and some related measures for evaluating probabilities, including decompositions of scoring rules and attributes of “goodness” of probabilites, comparability of scores, and the design of scoring rules for specific inferential and decision-making problems

Hailfinder is a Bayesian system that combines meteorological data and model with expert judgment, based on both experience and physical understanding, to forecast severe weather in Northeastern Colorado. The system is based on a model, known as a belief network (BN), that has recently emerged as the basis of some powerful intelligent systems. Hailfinder is the first such system to apply these Bayesian models in the realm of meteorology, a field that has served as the basis of many past investigations of probabilistic forecasting. The design of Hailfinder provides a variety of insights to designers of other BN-based systems, regardless of their fields of application.

In many forecasting situations, forecasts can be produced by several different methods. The ultimate objective of considering multiple methods may be to select a single method (the choosing scenario) or to aggregate the multiple forecasts into a single forecast (the combining scenario). Procedures for screening candidate forecasts—sufficiency in the choosing scenario and extraneousness in the combining scenario—are described here. Screening can identify forecasting methods that are dominated in the sense that their forecasts are clearly inferior to those of other methods or do not add any information to the combination of forecasts. These evaluation procedures are illustrated and contrasted by considering prototypical examples and an application involving precipitation probability forecasts. The value of screening is that it can reduce the set of candidate forecasting methods to a manageable number, which can then be evaluated in greater detail.

Evaluation of forecasts encompasses the processes of assessing both forecast quality and forecast value. These processes necessarily play key roles in any effort to improve forecasting performance or to enhance the usefulness of forecasts. A framework for forecast verification (the process of assessing forecast quality) based on the joint distribution of forecasts and observations — and on the conditional and marginal distributions derived from factorizations of this joint distribution — is described. The joint, conditional, and marginal distributions relate directly to basic aspects of forecast quality, and evaluation methods based on these distributions — and associated statistics and measures — provide a coherent, diagnostic approach to forecast verification. This approach — and its attendant methodology — is illustrated using a sample of probabilistic long-range weather forecasts. A decision-analytic approach to the problem of assessing the value of forecasts is outlined, and this approach is illustrated by considering the so-called fallowing-planting problem. In addition to providing estimates of the value of state-of-the-art and hypothetically improved long-range weather forecasts, the results of this case study illustrate some of the fundamental properties of quality/value relationships. These properties include the inherent nonlinearity of such relationships and the existence of quality thresholds below which the forecasts are of no value. The sufficiency relation is used to explore quality/value relationships; this relation embodies the conditions that must exist between the joint distributions of two forecasting systems to ensure that one system’s forecasts are better in all respects (i.e., in terms of quality and value) than the other system’s forecasts. The applicability of the sufficiency relation is illustrated by comparing forecasting systems that produce prototypical long-range weather forecasts. This application also demonstrates that quality/value reversals can occur when the multifaceted nature of forecast quality is not respected. Some outstanding problems in forecast evaluation are identified and briefly discussed. Recommendations are made regarding improvements in evaluation methods and practices.

The sufficiency relation, originally developed in the context of the comparison of statistical experiments, provides a sound basis for the comparative evaluation of forecasting systems. The importance of this relation resides in the fact that if forecasting system A can be shown to be sufficient for forecasting system B, then all users will find A's forecasts of greater value than B's forecasts regardless of their individual payoff structures. In this paper the sufficiency relation is applied to the problem of comparative evaluation of prototypical climate forecasting systems. The primary objectives here are to assess the basic applicability of the sufficiency relation in this context and to investigate the implications of this approach for the relationships among the performance characteristics of such forecasting systems. The results confirm that forecasting system A is sufficient for forecasting system B when the former uses more extreme probabilities more frequently than the latter. Further, in terms of the relatively simple forecasting systems considered here, it is found that system A may be sufficient for system B even if the former uses extreme forecasts less frequently, provided that A's forecasts are-to a certain degree-more extreme than B's forecasts. Conversely, system A cannot be shown to be sufficient for system B if the former uses less extreme forecasts more frequently than the latter. The advantages of the sufficiency relation over traditional performance measures in this context are also demonstrated. Several issues related to the general applicability of the sufficiency relation to the comparative evaluation of climate forecasts are discussed. Possible extensions of this work, as well as some implications of the results for verification procedures and practices in this context, are briefly described.

The Environmental Stewardship Scheme provides payments to farmers for the provision of environmental services based on agricultural foregone income. This creates a potential incentive compatibility problem which, combined with an information asymmetry on farm land heterogeneity, could lead to adverse selection of farmers into the scheme. However, the Higher Level Scheme (HLS) design includes some features that potentially reduce adverse selection. This paper studies the adverse selection problem of the HLS using a principal agent framework at the regional level. It is found that, at the regional level, the enrolment of more land from lower payment regions for a given budget constraint has led to a greater overall contracted area (and thus potential environmental benefit) which has had the effect of reducing the adverse selection problem. In addition, for landscape regions with the same payment rate (i.e. of the same agricultural value), differential weighting of the public demand for environmental goods and services provided by agriculture (measured by weighting an environmental benefit function by the distance to main cities) appears to be reflected into the regulator’s allocation of contracts, thereby also reducing the adverse selection problem.

Equitable skill scores are based on scoring matrices, which assign scores to the various combinations of forecast and observed events. The basic tenets of equitability require that i) all constant forecasts - and random forecasts - receive the same expected score, and ii) the elements of scoring matrices do not depend on the elements of performance matrices. Scoring matrices are assumed here to be symmetric and to possess other reasonable properties related to the nature of the underlying variable. To scale the elements of scoring matrices, the expected scores for constant and random forecasts are set equal to zero and the expected score for perfect forecasts is set equal to one. To obtain a unique scoring matrix, additional conditons must be imposed or some scores must be specified a priori. -from Authors

Commentary on Armstrong, J. Scott, and Collopy, Fred, (1992), “Error measures for generalizing about forecasting methods: Empirical comparisons,” International Journal of Forecasting, 8, 69-80.