Content uploaded by Saurabh Bansal

Author content

All content in this area was uploaded by Saurabh Bansal on Feb 18, 2018

Content may be subject to copyright.

Content uploaded by Saurabh Bansal

Author content

All content in this area was uploaded by Saurabh Bansal on Dec 27, 2016

Content may be subject to copyright.

This article was downloaded by: [128.118.207.64] On: 25 July 2017, At: 06:08

Publisher: Institute for Operations Research and the Management Sciences (INFORMS)

INFORMS is located in Maryland, USA

Operations Research

Publication details, including instructions for authors and subscription information:

http://pubsonline.informs.org

Using Experts’ Noisy Quantile Judgments to Quantify

Risks: Theory and Application to Agribusiness

http://orcid.org/0000-0001-5443-4144Saurabh Bansal, Genaro J. Gutierrez, John R. Keiser

To cite this article:

http://orcid.org/0000-0001-5443-4144Saurabh Bansal, Genaro J. Gutierrez, John R. Keiser (2017) Using Experts’ Noisy

Quantile Judgments to Quantify Risks: Theory and Application to Agribusiness. Operations Research

Published online in Articles in Advance 24 Jul 2017

. https://doi.org/10.1287/opre.2017.1627

Full terms and conditions of use: http://pubsonline.informs.org/page/terms-and-conditions

This article may be used only for the purposes of research, teaching, and/or private study. Commercial use

or systematic downloading (by robots or other automatic processes) is prohibited without explicit Publisher

approval, unless otherwise noted. For more information, contact permissions@informs.org.

The Publisher does not warrant or guarantee the article’s accuracy, completeness, merchantability, fitness

for a particular purpose, or non-infringement. Descriptions of, or references to, products or publications, or

inclusion of an advertisement in this article, neither constitutes nor implies a guarantee, endorsement, or

support of claims made of that product, publication, or service.

Copyright © 2017, INFORMS

Please scroll down for article—it is on subsequent pages

INFORMS is the largest professional society in the world for professionals in the fields of operations research, management

science, and analytics.

For more information on INFORMS, its publications, membership, or meetings visit http://www.informs.org

OPERATIONS RESEARCH

Articles in Advance,pp. 1–16

http://pubsonline.informs.org/journal/opre/ ISSN 0030-364X (print), ISSN 1526-5463 (online)

Using Experts’ Noisy Quantile Judgments to Quantify Risks:

Theory and Application to Agribusiness

Saurabh Bansal,aGenaro J. Gutierrez,bJohn R. Keiserc

aThe Pennsylvania State University, University Park, Pennsylvania 16802; bThe University of Texas at Austin, Austin, Texas 78712; cDow

AgroSciences, Marshalltown, Iowa 50158

Contact:

sub32@psu.edu,http://orcid.org/0000-0001-5443-4144 (SB); genaro@austin.utexas.edu (GJG); JRKeiser@dow.com (JRK)

Received: July 31, 2015

Revised: September 17, 2015; July 6, 2016;

November 2, 2016

Accepted: December 15, 2016

Published Online in Articles in Advance:

July 24, 2017

Subject Classiﬁcations: decision analysis: risk;

forecasting applications; industries:

agriculture/food

Area of Review: OR Practice

https://doi.org/10.1287/opre.2017.1627

Copyright: ©2017 INFORMS

Abstract. Motivated by a unique agribusiness setting, this paper develops an optim-

ization-based approach to estimate the mean and standard deviation of probability distri-

butions from noisy quantile judgments provided by experts. The approach estimates the

mean and standard deviations as weighted linear combinations of quantile judgments,

where the weights are explicit functions of the expert’s judgmental errors. The approach is

analytically tractable, and provides ﬂexibility to elicit any set of quantiles from an expert.

The approach also establishes that using an expert’s quantile judgments to deduce the

distribution parameters is equivalent to collecting data with a speciﬁc sample size and

enables combining the expert’s judgments with those of other experts. It also shows ana-

lytically that the weights for the mean add up to one and the weights for the standard

deviation add up to zero—these properties have been observed numerically in the liter-

ature in the last 30 years, but without a systematic explanation. The theory has been in

use at Dow AgroSciences for two years for making an annual decision worth $800 million.

The use of the approach has resulted in the following monetary beneﬁts: (i) ﬁrm’s annual

production investment has reduced by 6%–7% and (ii) proﬁt has increased by 2%–3%. We

discuss the implementation at the ﬁrm, and provide practical guidelines for using expert

judgment for operational uncertainties in industrial settings.

Funding:

This research was supported, in part, by grants from the Center for Supply Chain Research

at Penn State, and the Supply Chain Management Center of the McCombs School of Business at

The University of Texas at Austin.

Supplemental Material:

The online appendix is available at https://doi.org/10.1287/opre.2017.1627.

Keywords:

expert judgments

•

quantile judgments

•

estimating distributions

•

bootstrap

•

yield uncertainty

1. Introduction and Industry Motivation

1.1. Problem Context

Understanding and quantifying production-related

uncertainties is critical for decision making in busi-

nesses. The probability distributions for these uncer-

tainties are usually estimated using historical data

obtained during repetitive manufacturing. But these

data may not be available when ﬁrms frequently

launch new products in the market, e.g., at ﬁrms in the

semiconductor and in the agribusiness industry. In the

absence of historical data, these ﬁrms turn to domain

experts for obtaining subjective probability distribu-

tions (e.g., Baker and Solak 2014).

Prior literature (e.g., O’Hagan and Oakley 2004) ad-

vises that in these situations, one should avoid ob-

taining direct estimates of the standard deviation from

domain experts as this quantity is not intuitive to esti-

mate. This literature recommends obtaining experts’

input in the form of judgments for speciﬁc discrete

points on distributions, for example, judgments for spe-

ciﬁc quantiles, but also cautions that these judgments

are subject to judgmental errors (Ravinder et al. 1988).

However, a systematic approach that uses these judg-

ments to deduce the mean and standard deviation of

probability distributions, while explicitly modeling and

accounting for experts’ judgmental errors, is not yet estab-

lished. In this paper, we accomplish this task. Speciﬁ-

cally, we develop an approach to deduce the mean and

standard deviation using judgments provided by one

or multiple experts for distribution quantiles (or frac-

tiles). This approach is analytically tractable, provides

the ﬂexibility of using judgments for any set of quantiles

that an expert is willing to provide, and is amenable to

combining an expert’s quantile judgments with those

of other experts. The approach also establishes a novel

equivalence between the quality of an expert’s judg-

ments and the size of an experimental sample that is

equally informative about the distribution.

This approach was developed to manage a dynamic

new product development situation at Dow Agro-

Sciences (DAS) for an annual decision worth $800 mil-

lion. Analysis on the ﬁrm’s historical decisions shows

that the use of the approach has resulted in the fol-

lowing monetary beneﬁts: (i) ﬁrm’s annual production

investment has reduced by 6%–7% and (ii) proﬁt has

increased by 2%–3%. We also discuss the implementa-

tion of the approach developed at the ﬁrm, and prac-

1

Downloaded from informs.org by [128.118.207.64] on 25 July 2017, at 06:08 . For personal use only, all rights reserved.

Bansal et al.: Using Experts’ Judgments to Quantify Risks

2Operations Research, Articles in Advance, pp. 1–16, ©2017 INFORMS

tical guidelines for seeking expert judgment for opera-

tional uncertainties in industrial situations.

The rest of this paper is organized as follows. Sec-

tion 2provides an overview of our approach and

the contributions to the existing literature. Sections 3

and 4discuss a model for deducing mean and standard

deviation from quantile judgments, derive the solu-

tion and its structural properties. Section 5discusses

an equivalence of expertise with randomly collected

data and results for combining judgments from mul-

tiple experts. Section 6describes implementation of

the approach at DAS and quantiﬁcation of beneﬁts of

using the approach. Section 7concludes with insights

for practice.

2. Overview of Approach and Our

Contributions to the Existing Literature

2.1. Overview

We develop an optimization-based approach to esti-

mate the mean and standard deviation from quan-

tile judgments provided by an expert; speciﬁcally,

we obtain the minimum variance estimates of the

mean and standard deviation of yield distributions as

weighted averages of the quantile judgments provided

by the expert, subject to the constraint that the esti-

mates are unbiased. The model has two inputs. The

ﬁrst input is an identiﬁcation of the quantiles for which

the expert will provide judgments. For example, at

DAS, the expert chose to provide judgments for the

10th, 50th, and 75th quantiles because his software was

set to show these quantiles during data analysis, and he

was accustomed to thinking about these quantiles. The

second input is a quantiﬁcation of the noise present

in the expert’s judgments for the quantiles. This quan-

tiﬁcation is done separately by comparing the expert’s

judgments for the quantiles with the true values for

a number of distributions constructed using historical

data. We discuss this empirical estimation at DAS in

Section 6.

The solution to the optimization model assigns two

sets of weights to the quantile judgments (e.g., for

the 10th, 50th, and 75th quantiles). The ﬁrst set of

weights is for estimating the mean as a weighted aver-

age of quantile judgments. The second set of weights

is used similarly to estimate the standard deviation.

The weights are speciﬁc to the noise quantiﬁed in the

second input discussed above for the expert’s quantile

judgments.

2.2. Contributions to Literature

A large body of literature considers situations in which

experts provide their assessments for events with

binary outcomes (e.g., Ayvaci et al. 2017). In contrast,

we focus on situations where continuous distributions

need to be speciﬁed over the outcomes, and expert

judgment is sought to estimate these probability distri-

butions. Two streams of literature are relevant to our

focus: (i) models of judgmental errors and (ii) practice-

driven literature on the use of expert judgments.

2.2.1. Models on Model-Driven Theory on Judgments.

The ﬁrst stream of related literature is on model-driven

theory of judgmental errors. The existing literature on

expert judgments acknowledges the potential severity

of judgmental errors and focuses on developing elici-

tation guidelines for reducing judgmental errors (e.g.,

Koehler et al. 2002). In contrast, articles on moment

estimation from quantile judgments have explored the

problem of deducing moments from the median and

two additional symmetric quantiles (typically, the 5th

and 95th) or four additional symmetric quantiles (typ-

ically, the 5th, 25th, 75th, and 95th), numerically with a

key assumption: no judgmental errors are present. Pear-

son and Tukey (1965) and Keefer and Bodily (1983)

follow this paradigm. But no prior articles consider

the problem where subjective quantile judgments from

multiple experts need to be combined to deduce the

mean and standard deviation or the case where an

expert provides judgments for an arbitrary set of quan-

tiles that are diﬀerent from the standard ones men-

tioned above. We contribute to this literature by devel-

oping a tractable solution approach to this problem.

A salient feature of this approach is that one can use

any set of quantile judgments that an expert can pro-

vide (over the 5th, 25th, 75th, and 95th as discussed

in the prior literature) to estimate the mean and stan-

dard deviation. This feature is useful for practice since

an expert may not be willing to provide quantile judg-

ments for speciﬁc symmetric quantiles. For example,

the expert at DAS was habituated to seeing the 10th,

50th, and 75th quantiles for historical data on his soft-

ware and was willing to estimate only these quantiles.

The approach also provides the following struc-

tural insights. First, regardless of the magnitude of an

expert’s judgmental errors and the quantiles elicited,

the variance-minimizing weights for the estimation

of the mean and standard deviation add up to 1

and 0, respectively. This structural property explains

the numerical ﬁndings in Pearson and Tukey (1965),

Lau et al. (1999), among others, who all assume that

judgmental errors do not exist. Second, our approach

establishes a new quantiﬁcation of expertise: it speci-

ﬁes the size of a random sample that would provide

estimates of mean and standard deviation with the

same precision as that of the estimates obtained using

the expert’s judgments for quantiles. This equivalence

enables an objective comparison of experts. Finally, in

our approach, the optimal weights provide point esti-

mates and variability in the estimates for the moments.

This quantiﬁcation of variability of moment estimates

enables us to combine quantile judgments from multi-

ple experts in a rational and consistent manner.

Downloaded from informs.org by [128.118.207.64] on 25 July 2017, at 06:08 . For personal use only, all rights reserved.

Bansal et al.: Using Experts’ Judgments to Quantify Risks

Operations Research, Articles in Advance, pp. 1–16, ©2017 INFORMS 3

Prior literature, e.g., O’Hagan (1998), Stevens and

O’Hagan (2002), discusses the role of expert judgments

in the absence of data for constructing prior distri-

butions; when data become available, posterior dis-

tributions for parameters are obtained using Bayesian

updating. The literature discusses two cases. When

conjugate priors are used, the posterior distributions

are obtained in closed form. When conjugate prior

cannot be used, numerical approaches must be used

to obtain posterior distributions. We make two con-

tributions to this literature. First, we develop a novel

approach to obtain the prior distributions on the

parameters for the mean and standard deviations of

distributions using expert judgments for quantiles.

Second, we show that the joint prior distributions

are correlated and are not conjugate priors, and we

develop a Copula-based approach to obtain the poste-

rior distributions.

2.2.2. Practice-Driven Tools and Insights. Our contri-

butions to practice are as follows. We provide a step-by-

step approach to quantifying an expert’s judgmental

errors and then discuss some practical issues observed

during this quantiﬁcation at DAS. Speciﬁcally, we dis-

cuss a bootstrapping approach to separate judgmental

errors from sampling errors during the error quantiﬁ-

cation process. Then, we show that the information

provided by the expert is equivalent to ﬁve to six years

of data collection at DAS using our approach. Such

quantiﬁcation has not been reported in practice litera-

ture before. We also report that the expert at DAS was

reluctant to provide judgments for extreme quantiles

because of his inability to distinguish between random

variations and systematic reasons as causes of extreme

outcomes. This observation suggests that contextual

reasons and experts’ preferences can lead to elicitation

of quantiles that are diﬀerent from the standard val-

ues (median and two or four symmetric quantiles); our

approach is especially useful in such situations.

3. Analytical Model

We consider a real-valued continuous random vari-

able X, whose distribution is to be estimated. The pro-

bability density function (PDF) of Xis denoted as

φ(x;θ), where θ[θ1, θ2]tare the parameters of the

PDF, and µ1,µ2denote the mean and standard devi-

ation, respectively. Similar to Lindley (1987), O’Hagan

(2006), we assume that the distribution family is known

from the application context, but the parameters are

not known. This framework is especially relevant to a

number of operations contexts in which the parametric

family of probability distributions is known from the

historical data available or from formal models. The

cumulative distribution function (CDF) of Xis denoted

as Φ. A source of information such as an expert pro-

vides quantile judgments ˆ

xicorresponding to probabil-

ity CDF values pifor i1,2, . . . , m. In vector notation,

we denote the quantile judgments as ˆ

x[ˆ

x1, . . . , ˆ

xm]t

and probability values as p[p1, . . . , pm]t.

We seek to develop an approach to deduce µ1, µ2

from the quantile judgments ˆ

x. From theoretical and

application perspective, it is desirable that the ap-

proach’s formulation provides a unique solution to the

problem, preferably in closed form, and is amenable

to sensitivity analysis. Prior literature in this domain

(e.g., Keefer and Bodily 1983, Johnson 1998) also sug-

gests that for an ease of implementation, the approach

should be consistent with moment matching and

with other probability discretization practices in use,

e.g., program evaluation and review technique (PERT)

for project management. Our approach accomplishes

these objectives and additionally provides a quantiﬁca-

tion of the quality of expert’s judgments into an equiv-

alent sample size.

3.1. Preliminaries for Expert Judgments

We assume that the quantile judgments are obtained

using an underlying process or mental model (we dis-

cuss the mental model used by the expert at DAS,

in Figure 1, Section 6.1), which is error prone but is

used consistently for generating quantile judgments.

This assumption means that the expert’s judgmental

errors are stable during elicitation. We further assume

an additive error structure that is used frequently in the

literature (e.g., Ravinder et al. 1988): the quantile judg-

ment ˆ

xiis composed of a true value xiand an additive

error ei:

ˆ

xixi+ei.(1)

In vector notation, the error model is ˆ

xx+e. Con-

sistent with this literature, we assume that the error ei

has two parts: a systematic component or bias δiand

a random component or noise i, such that eiδi+i

and E[i]0. The bias δicaptures the average devi-

ation of the judgments for quantile ifrom the true

value. The noise icaptures the spread in the error

due to random variations. In vector notation, the bias

and residual variation are denoted as δand , respec-

tively. The noise is quantiﬁed in variance-covariance

matrix Ω. The diagonal elements of this matrix ωi i

Var(i)denote the variance in the unbiased judgment

of quantile i. The oﬀ-diagonal elements are covariances

of unbiased judgments ωi j Cov(i, j). We discuss the

empirical estimation of δand Ωseparately in Section 6,

and assume for now that these quantities are available.

From the biased judgments ˆ

x, the unbiased judg-

ments ˆ

qare obtained by removing bias as ˆ

qˆ

x−δ.

Substituting this relationship into ˆ

xx+ex+δ+,

we obtain

ˆ

qx+.(2)

The matrix Ωfor is used as an input in the optimiza-

tion model discussed next.

Downloaded from informs.org by [128.118.207.64] on 25 July 2017, at 06:08 . For personal use only, all rights reserved.

Bansal et al.: Using Experts’ Judgments to Quantify Risks

4Operations Research, Articles in Advance, pp. 1–16, ©2017 INFORMS

3.2. Optimization Problem

We seek to obtain the estimates of the mean ˆ

µ1and

standard deviation ˆ

µ2as pooled or weighted linear

functions of the debiased quantile judgments as ˆ

µk

wt

kˆ

q;k1,2with the weights w1≡ [w11 ,w12 , . . . , w1m]t

and w2≡ [w21 ,w22 , . . . , w2m]t. Since the unbiased judg-

ments ˆ

qare subject to noise , the estimates ˆ

µk;k1,2

have variances Var[wt

kˆ

q]. Smaller values of the vari-

ances of these estimates are desirable as it would imply

that the estimates are more precise. To this end, it is

desirable to select weights wk;k1,2that lead to a

small variance in the estimates ˆ

µk. We ﬁrst restate the

variance of estimates ˆ

µkin terms of the weights as

Var[ˆ

µk]Var[wt

kˆ

q]E[(wt

kˆ

q−E[wt

kˆ

q])2](3)

E[(wt

k(x+) − E[wt

k(x+)])2]

E[(wt

k(x+) − wt

kx)2]

E[wt

ktwk]wt

kΩwk.(4)

Prior literature (e.g., Bates and Granger 1969, Granger

1980) shows that only minimizing this variance is not

informative as it is minimized by setting wk0and

the resultant weighted linear estimate is always equal

to ˆ

µk0for all judgments ˆ

q. This literature suggests

adding constraints to make statistical estimates respon-

sive to forecasts or judgments. Our focus will be on

a speciﬁc class of such constraints. We seek variance-

minimizing weights such that the obtained estimates

wt

kˆ

qare unbiased, i.e., E[wt

kˆ

q]µk, leading to the fol-

lowing optimization formulations for k1,2:

min

wk

Var[ˆ

µk]wt

kΩwk

s.t. E[wt

kˆ

q]µk

(5)

Problem (5) consists of ﬁnding the weights wkthat

lead to the minimum variance unbiased estimates

of µk. In the next section, we determine these weights

for location-scale distributions using structural proper-

ties of these distributions. The focus on location-scale

distributions is motivated by their widespread applica-

tion in numerous operations management contexts (see

Kelton and Law 2006, for a list of these applications)

as well as their speciﬁc application at DAS, where the

in-house statistics team has shown using existing data

that yields are normally distributed. This analysis is in

Section 6.2.

4. Solution: Weights for

Quantile Judgments

In Section 4.1 we specialize the problem (5) for distri-

butions of a location-scale family, and obtain the opti-

mal weights for quantile judgments and the weights’

structural properties in Section 4.2.

4.1. Reformulation and Solution for Distributions

of a Location-Scale Family

We now assume that Xis a location-scale random vari-

able with location and scale parameters θ1∈and

θ2∈++, respectively, and transform the constraint

E[wt

kˆ

q]µkin formulation (5) using two properties of

location-scale distributions. The ﬁrst property enables

us to rewrite the left-hand side (LHS) of this constraint

as a function of θ1,θ2. If Xis a location-scale ran-

dom variable with PDF φ(·;θ), then a speciﬁc value x

that corresponds to probability pcan be expressed as

xθ1+θ2z, where zdenotes the value of standardized

random variable with the standardized PDF φ(·;[0,1]t)

for probability p(Casella and Berger 2002, p. 116). We

write this expression in vector form as

xZθ,(6)

where Zis the m×2matrix formed as Z[1,z],zis the

column vector of standardized quantiles correspond-

ing to the probabilities p, and 1is a column vector of

ones. Substituting (6) into the LHS of the error model

in (2), ˆ

qx+, it follows that

E[wt

kˆ

q]E[wt

k(x+)] wt

kE[x+]wt

kZθ.(7)

The second property, formalized in Lemma 1below,

enables us to rewrite the right-hand side (RHS) of

the unbiasedness constraint E[wt

kˆ

q]µkas a function

of θ1, θ2.

Lemma 1 (Characterization of Location-Scale Moments).

If Xis a location-scale random variable with parameters θ

[θ1, θ2]twith ﬁnite jth moments for j1,2, . . ., then,

(a) the raw moments E[Xj]are given by E[Xj]

Pj

i0j

iθj

1θj−i

2κj−iand

(b) the central moments are given by E[(X−µ1)j]

θj

2Pj

i0(−κ1)iκj−i,

where the constants κiare κ01and κjE[Zj]for j

1,2,....

The proof is in Appendix A1. The values of κjare

documented in the literature for location-scale distri-

butions (see, e.g., Johnson et al. 1994). For example,

for a normal distribution, we have (κ0, κ1, κ2)(1,0,1).

It follows from part (a) of Lemma 1that µ1E[X]

[1, κ1]θ. It follows from part (b) of the lemma that

variance of Xis equal to E[(X−µ1)2]θ2

2(κ2−κ2

1),

and therefore the standard deviation is equal to µ2

pE[(X−µ1)2]pθ2

2(κ2−κ2

1)[0,pκ2−κ2

1]θ. We write

both relationships in vector notation as

µkat

kθ(8)

with at

1[1, κ1]and at

2[0,pκ2−κ2

1].

Substituting (7) and (8) into the LHS and RHS of

the constraint E[wt

kˆ

q]µk, respectively, we obtain the

following condition on the weights wkfor the estimate

wt

kˆ

qto be unbiased. This condition will help us solve

the problem in a tractable form.

Bansal et al.: Using Experts’ Judgments to Quantify Risks

Operations Research, Articles in Advance, pp. 1–16, ©2017 INFORMS 5

Proposition 1. If Xis a random variable with a location-

scale distribution, the weighted linear estimator wt

kˆ

qis un-

biased for µk, if and only if the weights wksatisfy

Ztwkak;k1,2.(9)

Proof. By deﬁnition, the estimate wt

kˆ

qis unbiased if

and only if E[wt

kˆ

q]µk. Substituting (7) and (8) into

the LHS and RHS of the constraint, it follows that the

estimate is unbiased if and only if wt

kZθat

kθfor all

values of θ1and θ2. It follows that the estimator is

unbiased if and only if wt

kZat

k, i.e., Ztwkak;k

1,2.

The implication of the iﬀ in Proposition 1is that we

can replace the constraint E[wt

kˆ

q]µkin formulation

(5) with the condition on weights Ztwkak;k1,2.

After this substitution, we obtain the formulation for

k1,2as

min

wk

wt

kΩwk

s.t. Ztwkak.(10)

The matrix Ωis a covariance matrix, and therefore it

must be positive semideﬁnite. It follows that the prob-

lem (10) is a quadratic convex problem, and its solu-

tion is obtained by solving a Lagrange formulation of

the problem. The next result establishes this unique

solution.

Theorem 1. The weights that solve problem (10) are given

by w∗

kΩ−1Z(ZtΩ−1Z)−1ak.

The proof is in Appendix A2. The conspicuous fea-

ture of the optimal weights w∗

kis that they are explicit

functions of the expert’s precision encoded in Ω. There-

fore, a change in an expert’s precision in providing

quantile judgments will modify Ω, which, in turn,

will change the optimal weights w∗

kfor the quan-

tile judgments. Finally, we note that the variance of

the estimates Var[ˆ

µk]at the optimal weights is equal

to Var[ˆ

µk]w∗

k

tΩw∗

k, which simpliﬁes to Var[ˆ

µk]

at

k(ZtΩ−1Z)−1ak, establishing a direct link between the

variance in the estimates ˆ

µkto the expert-speciﬁc Ω.

4.2. Structural Properties and Generalization of

Results Available in Literature

The development thus far provides new generaliza-

tions and insights to the existing literature. First, in our

approach, the expert can provide judgments for any set

of quantiles that he is comfortable estimating, i.e., he

is no longer restricted to providing his judgments for

the 5th, 25th, 50th, 75th, and 95th quantiles as speci-

ﬁed in extant literature such as Lau et al. (1996). This

ﬂexibility is useful since we no longer need to convince

an expert to provide judgments for these speciﬁc quan-

tiles and instead can focus on understanding why the

expert believes that he can provide better judgments

for his chosen quantiles. We discuss one such exam-

ple in Section 6.2. The second generalization of our

approach is that it provides an analytical foundation

to a numerical property observed consistently in the

existing literature that the weights add up to a constant

as follows.

Proposition 2. The optimal weights for quantiles add up to

constants. Speciﬁcally, Pm

i1w∗

1i1and Pm

i1w∗

2i0.

The proof is in Appendix A3. This result is true

regardless of the numerical values of Ω; therefore, it

holds true even when the judgmental errors are arbi-

trarily small, e.g., when Ωlimλ→0λI. A number of

prior articles (Pearson and Tukey 1965, Perry and Greig

1975, Keefer and Bodily 1983, Johnson 1998) numer-

ically discuss the limiting case when the errors are

absent. They select speciﬁc numerical test cases of

means and standard deviations of a distribution and

obtain the 5th, 50th, and 95th quantiles or other spe-

ciﬁc symmetric quantiles for these cases. Then, they

consider various sets of candidate weights. For each set

of weights, they estimate the means of all test cases as

weighted linear combinations of the quantile values.

Finally, they identify the set of weights that result in

the smallest squared deviations between the true and

the estimated means over all cases. A similar analy-

sis provides the weights to obtain the standard devia-

tion. The weights recommended in this literature add

up to 1 and 0 for the mean and standard deviation,

respectively. Proposition 2establishes that the additiv-

ity properties observed numerically in these articles are

structural properties of probability distributions, and

hold true for any magnitude of judgmental errors.

Third, these additivity properties are also shared by

the weights assigned in the project management tech-

nique PERT to the estimates for the optimistic, pes-

simistic, and most likely scenarios. The weights for the

mean are (1/6, 4/6, 1/6), respectively, for the estima-

tion of the mean adding up to one, and (−1/6, 0, 1/6)

for the estimation of the standard deviation adding

up to zero. Fourth, one can show using straightfor-

ward algebra that our approach automatically assigns

lower weight to a quantile judgment that has large

noise. In situations where an expert provides a num-

ber of quantile judgments, this feature is useful in

identifying which quantile judgments have large noise

and are therefore not useful for the estimation of the

moments; the weights for these quantile judgments

will be negligible.

5. Data Equivalence, Multiple Experts, and

Other Relationships

In Section 5.1, we determine the size of a randomly

drawn sample that is equivalent in terms of precision to

the expert’s judgments. In Section 5.2 we discuss com-

bining judgments of one expert with the judgments

from other experts. In Section 5.3, we discuss the con-

Bansal et al.: Using Experts’ Judgments to Quantify Risks

6Operations Research, Articles in Advance, pp. 1–16, ©2017 INFORMS

sistency of the approach developed with least squares

and moment matching.

5.1. Equivalence between Expertise and

Size of a Random Sample

Expert input is sought for estimating probability dis-

tributions when collecting data is costly. The expert’s

quantile judgments, after using our approach, provide

point-estimates ˆ

µkand the variances in these estimates

Var[ˆ

µk]. We can compare this variance with the vari-

ance of the mean and standard deviation obtained

from a sample of random observations for Xif data

collection is possible. Speciﬁcally, it is well known that

a sample mean has a variance of σ2/N1, where N1is

the sample size. In our approach, the variance in the

estimate of the mean is equal to Var[ˆ

µ1]w∗

1

tΩw∗

1(see

Appendix A4 for proof), which can be simpliﬁed to

[1, κ1](ZtΩ−1Z)−1[1, κ1]t. By equating these two vari-

ances, we can determine the size of a randomly col-

lected sample that would provide the same precision

of the estimate of the mean as the expert does. We call

this size an equivalent sample size for the mean. A simi-

lar analysis provides the equivalent sample size for the

standard deviation. The next result provides expres-

sions of these equivalent sample sizes.

Proposition 3. The precision of the estimates ˆ

µkobtained

using an expert’s quantile judgments with judgmental error

matrix Ωis comparable to the precision of estimates obtained

from an iid sample of size Nk, where

N1µ2

2

[1, κ1](ZtΩ−1Z)−1[1, κ1]tand

N2≈

µ2

2P4

j0(−κ1)jκ4−j

(κ2−κ2

1)2−(P2

j0(−κ1)jκ2−j)2

(κ2−κ2

1)2

4[0,pκ2−κ2

1](ZtΩ−1Z)−1[0,pκ2−κ2

1]t.

The proof is in Appendix A5. This result has two

profound implications. First, using this result, multi-

ple experts can be compared objectively based on their

judgmental errors quantiﬁed in Ω. More speciﬁcally,

for two experts A and B with matrices ΩAand ΩB, the

ratios of equivalent sample sizes are given as NA

1/NB

1

([1, κ1](ZtΩ−1

BZ)−1[1, κ1]t)/([1, κ1](ZtΩ−1

AZ)−1[1, κ1]t)

and NA

2/NB

2([0,pκ2−κ2

1](ZtΩ−1

BZ)−1[0,pκ2−κ2

1]t)/

([0,pκ2−κ2

1](ZtΩ−1Z)−1

A[0,pκ2−κ2

1]t), and they are

independent of the true value of µ1, µ2. For exam-

ple, if NA

k/NB

k2, then the estimates of µkobtained

from expert A are two times as reliable as the esti-

mates obtained from expert B. This beneﬁt from using

expert A over expert B is equal to the beneﬁt from

doubling the sample size of experimental or ﬁeld

data for the purposes of estimating µk. Second, some

recent literature (e.g., Akcay et al. 2011) quantiﬁes the

marginal beneﬁt of improving estimates of probabil-

ity distributions by collecting more data before making

decisions under uncertainty. Proposition 3provides

a natural connection to these results by quantifying

the economic beneﬁts of improving the precision in

judgments.

5.2. Combining Estimates from Multiple Experts

The technical development extends to multiple experts

j1,2, . . . , nas follows. We ﬁrst construct the com-

bined matrix

Ω

Ω11 Ω12 ··· Ω1n

Ω12 Ω22 ··· Ω2n

.

.

.

Ω1nΩ2n··· Ωnn

where Ω11 is the m×mmatrix for residual errors of

expert 1, the matrix Ω12 is the m×mcovariance matrix

for the errors of experts 1 and 2, and so on. Then, the

matrix Ωis used in Theorem 1along with matrix Ztof

size 2×mn,Zt[Zt

1Zt

2, . . . , Zt

n], where each Zj[1,zj],

zjis the column vector of standardized quantiles corre-

sponding to the probabilities pjthat expert jhas cho-

sen to provide judgments for, and 1is a column vector

of ones. The use of Theorem 1 provides mn weights; the

ﬁrst mweights for the ﬁrst expert, the next mweights

for the second expert, and so on.

We discuss a special case of interest here. Suppose n

experts j1,2, . . . , nprovide judgments for the same

set of quantiles, i.e., Zt[Zt

0Zt

0, . . . , Zt

0], and the covari-

ance matrix of each expert jis given by Ωj j rjΩ0∀j

(i.e., the judgmental error structure of one expert is a

scaled version of another expert) and further assume

that the errors of any two experts are mutually inde-

pendent, i.e., all elements of Ωi j ,i,jare equal to 0.

The weights for the mn quantile judgments obtained

using Theorem 1 are denoted as w∗

kwith elements w∗

kt

for t1,2, . . . , mn and k1,2. The ﬁrst mweights are

for expert 1, the next mweights are for expert 2, and

so on. We can write these weights as w∗

k[w1

k, . . . , wn

k],

where wj

kis the vector of weights of expert j. We can

also decompose the weights w∗

kas the product of con-

stant αjfor expert jand a common weight vector of m

weights wc

kthat would be obtained if each expert was

the only one available, i.e., w∗

k[α1wc

k, . . . , αnwc

k]. The

values of αjand the relationships between wj

kand wc

k

are as follows.

Proposition 4. Consider experts j1,2, . . . , n, whose co-

variance matrices are rjΩ0;∀j, and further assume that the

judgmental errors across experts are mutually independent.

Then,

(i) If any expert jwas the only expert available, the opti-

mal weights for his unbiased judgments would be wc

k

t

at

k(Z0

tΩ−1

0Z0)−1Z0

tΩ−1

0independent of the value of rj.

(ii) When the quantile judgments of the nexperts are

considered simultaneously, the weights for each expert jare

obtained as wj

kαjwc

kwith αj(1/rj)/R, where R

Pn

j1(1/rj).

The proof is in Appendix A6. As an illustration, sup-

pose that we have two experts with r11and r22, i.e.,

Bansal et al.: Using Experts’ Judgments to Quantify Risks

Operations Research, Articles in Advance, pp. 1–16, ©2017 INFORMS 7

expert 2 is half as precise as expert 1. Further, consider

the case when

Ω0

80 30 35

30 22 30

35 30 68

for the estimation of the 10th, 50th, and the 75th quan-

tiles. If the quantile judgments of only expert jare con-

sidered separately, the estimation weights are obtained

as wcT

1[−0.167 1.484 −0.317]and wcT

2[−0.576

0.190 0.386]for either expert jby using

Zt1 1 1

−1.285 0 0.674

and Ω0in Theorem 1, as stated in part (i) of Proposi-

tion 4.

When both experts are available, the optimal weights

for their quantile judgments are obtained by multi-

plying the independent weights wc

kwith the expert-

speciﬁc marginal weight αjas wj

kαjwc

k. It follows

from part (ii) of the proposition that the expert-speciﬁc

constants are α1(1/r1)/(3/2)2/3and α2(1/r2)/

(3/2)1/3. The weights for the mean, for example,

are obtained as w1

1(2/3) × (−0.167,1.484,−0.317)

(−0.111,0.989,−0.211)and w2

1(1/3) × (−0.167,1.484,

−0.317)(−0.055,0.495,−0.105)for experts 1 and 2,

respectively. The same weights are also obtained di-

rectly by ﬁrst constructing the combined matrix

Ωr1Ω00

0r2Ω0

and using it in Theorem 1with matrix

Zt[Zt

0,Zt

0]

1 1 1 1 1 1

−1.285 0 0.674 −1.285 0 0.674 ,

which gives w∗

1[−0.111,0.989,−0.211,−0.055,0.495,

−0.105].

5.3. Relationship with Classical Least Squares

Regression and Moment Matching

We now discuss how our model and its solution is con-

sistent with (i) the classical least square minimization-

based regression framework and (ii) with moment

matching. In the classical regression framework, the

variance-covariance matrix is ΩKΩ0, where K>0

is a scalar, the diagonals elements of Ω0are equal

to 1, and the oﬀ-diagonal elements are equal to 0, i.e.,

Ω0I. In our context, this would be the noninforma-

tive case when the expert is equally good at estimating

all quantiles and his judgmental errors are mutually

independent. We showed in Theorem 1 that the opti-

mal weights are equal to w∗

kΩ−1Z(ZtΩ−1Z)−1ak.

Now, substituting ΩKI, we obtain the weights as

w∗

kZ(ZtZ)−1ak, or alternately, as the familiar kernel

of the ordinary least squares in the big parentheses:

w∗

k

takt((ZZt)−1Zt).

In the moment matching framework, we would seek

to minimize the squared deviations of the debiased

quantile judgments ˆ

qiobtained from the expert for

probability pifrom the unobserved values of mean

and standard deviation, i.e., we would seek to solve:

minµ1, µ2{Pm

i1(Φ−1(pi;µ1, µ2) − ˆ

qi)2}. This approach is

codiﬁed in many commercial software (e.g., @RISK)

and has been used in prior academic literature (e.g.,

Wallsten et al. 2013). For location-scale distributions,

Φ−1(pi;µ1, µ2)θ1+ziθ2. Using the properties that θ1

µ1− (κ1/pκ2−κ2

1)µ2and θ2(1/pκ2−κ2

1)µ2, we can

rewrite this problem as

min

µ1, µ2

m

X

i1µ1−κ1

pκ2−κ2

1

µ2+zi

µ2

pκ2−κ2

1

−ˆ

qi2

.

The next result establishes the classical least squares

and moment matching as a special case of our

approach.

Proposition 5. Consider the original optimization prob-

lem: minwwt

kΩwksubject to E[wt

kˆ

q]µk.

(1) This problem reduces to ordinary least squares solu-

tion when ΩKΩ0, where Ω0I.

(2) Consider the moment matching problem in the form

minµ1, µ2{Pm

i1(Φ−1(pi;µ1, µ2) − ˆ

qi)2}. It’s solution is ˆ

a

wT

aˆ

qfor a∈ {µ1, µ2}and it is identical to the solution

obtained from the original problem for ΩKΩ0, where

Ω0I.

The proof is in Appendix A7. The second part of

the proposition implies that given quantile judgments

and no information for the noise in the judgments,

the best estimates of the mean and standard deviation

(under quadratic penalty) for location-scale distribu-

tions are linear functions of the quantile judgments.

And these estimates coincide with solution obtained

in our approach for the noninformative case of ΩI.

Our approach extends the moment matching model to

account for expert’s judgmental errors as captured in

Ωas minwwt

kΩwksubject to E[wt

kˆ

q]µkfor the case

when information for the expert’s judgmental errors

is available, i.e., when Ω,KI. Finally, we note that

our approach is amenable to Bayesian updating using

Markov chain Monte Carlo methods, and we omit

details for sake of brevity. We next discuss the imple-

mentation of the approached developed at DAS.

6. Implementation Details and

Beneﬁts at DAS

6.1. Industry Context: Estimating Production Yield

Distributions for Hybrid Seeds

DAS produces seeds for various crops such as corn and

soybean, and sells these seeds to farmers. Our focus is

Bansal et al.: Using Experts’ Judgments to Quantify Risks

8Operations Research, Articles in Advance, pp. 1–16, ©2017 INFORMS

on the production of hybrid seed corn. DAS decides

annually how many acres of land to use to produce

hybrid seed corn. The yield, or amount of hybrid seed

corn obtained per acre of land by DAS, is uncertain.

Under this yield uncertainty, producing hybrid seed

corn on a large plot of land may result in a surplus

with a large up-front production cost if the realized

yield is high; using a small plot of land may result

in costly shortages if the realized yield is low. Math-

ematical models that incorporate the yield distribu-

tion can determine the optimal area of land that DAS

should use, but the historical yield data are not avail-

able for obtaining a statistical distribution. The unique

industry-speciﬁc reasons for this lack of historical data

are discussed next, but before discussing these reasons,

we note an important characteristic of our focus. Our

focus is on the production yield realized by DAS when

it produces hybrid seeds, and this is the context in

which the term yield will be used in the remainder of

the paper.

6.1.1. Biological Context for Expert Judgments. DAS

has a pool of approximately 125 types of parent or

purebred seed corn; each type has a unique genetic

structure, and these purebred varieties are used to pro-

duce hybrid varieties of seed corn. To replenish the

stock of a speciﬁc parent seed, DAS plants this seed

in a ﬁeld. Self-pollination among the plants produces

seeds of the same type, this is why the parent seed

are purebred seeds. This inbreeding is carried out reg-

ularly to maintain inventories of parent seeds, and

statistical distributions of the yields obtained during

this inbreeding process are available from historical

data. But these seeds are not sold to farmers; rather,

the seeds sold to farmers are hybrid seeds that are

obtained by cross-mating pairs of parent seeds. This

cross-mating occurs when two diﬀerent parent seeds

are planted in the ﬁeld. Corn plants have male and

female reproductive parts. Plants growing from the

Figure 1. (Color online) During Cross-Pollination, the Male YChanges the Inbred Yield Distribution of XShown on the Left

Genetic makeup XGenetic makeup X

Female parent

Inbred seed:

• Yield data available

• Benchmark for hybrid yields

Expert’s judgment to

assess possible impact of Y

Self-pollination

Controlled

cross-pollination

Hybrid seed:

• Limited yield data available

Genetic makeup Y

Male parent

Notes. The expert’s mental model involves judgments about changes in the location and/or spread of the distribution due to Y. Possible

distributions after crossbreeding are shown in dotted lines on the right.

parent seeds of one type; say, X, are treated chemi-

cally and physically (in a process called detasseling,

e.g., see http://ntrdetassel.com/detasseling/) to make

them act as female, and the plants growing from the

parent seeds of the other type; say, Y, are made to act

as male. The cross-pollination between these parents

produces the hybrid seeds. DAS oﬀers more than 200

varieties of hybrid seed corn every year in the market

targeted to diverse soil and climate zones of the con-

tinental United States. Each variety is obtained from a

diﬀerent set of parents.

Due to the rapid pace of innovation in this indus-

try, the average life of hybrid varieties is short. DAS

produces and sells most hybrid varieties only three or

four times before replacing them with new hybrids.

Therefore, suﬃcient historical yield data necessary to

obtain statistical distributions are not available for most

hybrid seeds. In the absence of these data, DAS relies

on a yield expert to estimate the yield distributions

for producing the seeds. Before describing the process

the expert uses, we note that a set of hybrid seeds has

also been produced and sold repetitively. The historical

yield data of these hybrids serve an important purpose

in our estimation approach.

6.1.2. Expert’s Mental Model. The yield expert at DAS

uses a mental model for estimating the yield distribu-

tion for the production of a hybrid seed without his-

torical data. This model is illustrated in Figure 1for

the hybrid seed obtained by crossing varieties Xand

Y. Female parent plants (type X) provide the body on

which the hybrid seed grows; the male parent plants

(type Y) provide the pollen to fertilize the female plant.

Since the female plant nurtures the seed, the available

statistical distribution for the inbreeding for type X

provides a statistical benchmark (left part of Figure 1)

for the hybrid seed. The male parent aﬀects this distri-

bution during cross-pollination through its pollinating

power and other genetic characteristics, leading to var-

Bansal et al.: Using Experts’ Judgments to Quantify Risks

Operations Research, Articles in Advance, pp. 1–16, ©2017 INFORMS 9

ious likely distributions as shown in dotted lines in the

right part of the ﬁgure. This may include a shift in the

median and/or changes in the spread of the distribu-

tion. The expert’s contextual knowledge for the biology

of both parents provides him with insights into how

the distribution might change during cross-pollination.

6.1.3. Practice at DAS Before New Approach. In the

past, the yield expert has adjusted the median of the

inbreeding female distribution higher or lower to pro-

vide an estimate of the median yield for the production

of hybrid seed. Thus the estimate of the median seed

production yield has been based on indirect data and

is judgmental in nature. This median yield was used

for production planning decisions as follows. Man-

agers would ﬁrst calculate the area needed as area

demand/median yield judgment and then increase it

by 20% or 30% to account for high proﬁt margins. In

our interactions, managers articulated the need for a

rigorous approach to estimate the spread in the uncer-

tain yields, which could then be used to determine

the number of acres for each hybrid using optimiza-

tion models. Furthermore, since the yield expert is

required to provide judgments for almost 200 hybrid

seeds within a span of two weeks every seed produc-

tion season, it was necessary to develop an approach

that could be implemented within this time window.

Our analytical approach accomplishes these tasks.

In Section 6.2, we discuss how our approach con-

tinues to use the expert’s judgments for the median

(that he has estimated in the past) to exploit the mental

model that he has developed and used over years as

well as two additional quantiles selected by him. DAS

makes the production planning decision once a year,

typically during January–February. Our approach was

ﬁrst used in 2014, and has been in use since then. In

the ﬁrst step of our implementation, we determined

the bias vector δ, the matrix Ωof judgmental errors,

and the matrix Zcorresponding to the quantiles for

which the expert will provide judgments. This was

done using historical yield data for a set of hybrid

seeds that have been produced repetitively in the past.

Details of this step are in Section 6.2. From these quan-

tities, we obtained the optimal weights w∗

k;k1,2.

Details of this determination are in Section 6.3. Finally,

we quantiﬁed the beneﬁt from using our approach

using the data from the 2014 production planning deci-

sions. This analysis is presented in Section 6.4. In Sec-

tion 6.5, we discuss the integration of our approach

into DAS’s operational decision making.

6.2. Implementation: Data Collection at DAS and

Calibration of Judgmental Errors

The ﬁrst task during the implementation was to col-

lect data from the ﬁrm to determine the appropriate

distribution to use to model yield uncertainty, and to

calibrate the expert. We ﬁrst describe this data and the

Table 1. Tests to Accept/Reject Normality of Historical Yield

Data for a Subset of Seeds

Test: pvalue pvalue pvalue

H0: Data are normal for seed 1 for seed 2 for seed 3

Kolmogorov-Smirnov test >0.15 >0.15 >0.15

Anderson-Darling test 0.90 0.51 0.51

Lilliefors-van Soest test >0.20 >0.20 >0.20

Cramer-von Mises test 0.92 0.56 0.59

Ryan-Joiner test >0.10 >0.10 >0.10

statistical tests performed to determine the paramet-

ric family of yield distributions. Then, we describe the

process used for calibrating the expert at DAS.

We asked DAS to identify a set of hybrid seeds

that have been produced repetitively in the last few

years. Overall, DAS found L22 such hybrid seeds

indexed by l1,2, . . . , 22 and provided us with the

historical yield data for these seeds. Using this data

and other sources, we sought to determine the appro-

priate parametric family to model yield distribution.

First, we analyzed the available yield data and ran

a battery of tests, including the Kolmogorov-Smirnov

test, Anderson-Darling test, and Lilliefors-van Soest

test and found that they all failed to reject the hypoth-

esis that the data was normally distributed. Table 1

shows the results for three such seeds. These tests only

conﬁrm that normality cannot be ruled out. We then

ran a second test to see if normality provided the best

ﬁt with the data. In this test, we determined the para-

metric family with the best ﬁt with the data using the

chi-square test and Anderson-Darling test. The can-

didate distributions were the normal distribution, the

gamma distribution, the uniform distribution, the log-

normal distribution, the beta distribution, the Gumbel

distribution, the exponential distribution, the Weibull

distribution, the logistic distribution, and the inverse

normal distribution. The normal distribution was the

best ﬁt on the Anderson-Darling test for all hybrid

seeds. On the chi-square test, the normal distribution

provided the best ﬁt for a majority of the seeds.

In addition to this statistical proof for the hybrid

seeds, DAS has extensive data for inbred seeds that sup-

ports normality. Since the biological factors at play dur-

ing plant growth are the same in hybrid seeds, the yields

of the hybrid seed corn also would be normal. Recently,

Comhaire and Papier (2015) also provided statistical

evidence for normality of yields during seed corn pro-

duction. After identifying the normal distribution to be

appropriate, we determined the quantiles that the yield

expert at DAS was comfortable estimating (to obtain Z

for the normal distribution), as well as determined his

error structure (δand Ω), as described next.

6.2.1. Step 1: Selection of Quantiles for Elicitation and

Determination of Z.For each of the hybrid seeds l

1,2, . . . , L, we asked the expert to select three quantiles

Bansal et al.: Using Experts’ Judgments to Quantify Risks

10 Operations Research, Articles in Advance, pp. 1–16, ©2017 INFORMS

to estimate, without looking at the historical yield data

of these hybrids. The selection of three quantiles (rather

than more than three) was motivated by existing liter-

ature that suggests that three quantiles perform almost

as well as ﬁve quantiles (Wallsten et al. 2013), as well

as the time constraints faced by the expert. The expert

is an agricultural-scientist; he is well trained in statis-

tics and has worked extensively with yield data. His

quantitative background and experience were helpful

as he clearly understood the probabilistic meaning and

implications of quantiles. The ﬁrst quantile he selected

was the 50th quantile since he has estimated this quan-

tile regularly in the last few years. The extant literature

also has established that estimating this quantile has the

intuitive 50–50 high-low interpretation that managers

understand well (O’Hagan 2006).

We then asked the expert to provide us with his

quantile judgments for two other quantiles, one in each

tail of the yield distribution, that he was comfortable

estimating. The yield expert chose to provide his judg-

ments for the 10th and 75th quantiles for several rea-

sons. First, he has developed familiarity with these

quantiles in the last few years: his statistical software

(JMP) typically provides a limited number of quantile

values, including these two quantiles during data anal-

ysis, and he is accustomed to thinking about them. Sec-

ond, the expert suggested the use of these asymmetric

quantiles because if asked for symmetric quantiles, he

would intuitively “estimate one-tail quantile and calcu-

late the other symmetric quantile using the properties

of the normal distribution.” This will be equivalent to

estimating only one quantile instead of two.

Finally, the expert was not comfortable in provid-

ing judgments for quantiles that were further out in

the tails, such as the 1st and the 95th quantiles. This

reluctance was interesting and highlighted some subtle

disconnects between theory and practice. Some arti-

cles (e.g., Lau et al. 1998, Lau and Lau 1998) have sug-

gested weights for extreme quantiles such as 1 per-

centile, assuming no judgmental errors. However, the

expert found it diﬃcult to estimate extreme quantiles.

Speciﬁcally, he was concerned that he might not be

able to diﬀerentiate between random variations (that

we seek to capture) and acts of nature such as torna-

dos and ﬂoods (that we seek to exclude since the yield

expert cannot predict these events) that lead to extreme

outcomes.

We then determined the matrix Zfor the 10th, 50th,

and 75th quantiles. For the normal distribution, this

matrix is calculated as

Z

1−1.28

1 0

1 0.67

,

where the value of −1.28 is equal to the inverse of the

standard normal distribution at the probability 0.1 and

so on.

6.2.2. Step 2: Elicitation Sequence and Consistency

Check. For each distribution l, we obtained the three

quantile judgments ˆ

xil (pi);i1,2,3;l1,2, . . . , 22;pi

0.1,0.5,0.75 from the expert; the expert did not have

access to the historical yield data for these hybrids

during this estimation. We obtained the expert’s judg-

ments in two rounds. In Round 1, for each hybrid l,

the expert followed his usual procedure for studying

the yield distribution for the female parent, looking

at the properties of the male parent and providing

his judgment for the median (see Figure 1). We then

asked the expert to provide his quantile judgments for

the 10th and 75th quantiles, in that order. This cus-

tomized sequence is consistent with the extant liter-

ature that suggests ﬁrst obtaining an assessment for

50–50 odds (Garthwaite and Dickey 1985), and then

focusing further on quantiles in the tails. In Round 2 of

estimation, to encourage a careful reconﬁrmation of the

judgments provided in Round 1, we used a feedback

mechanism. We used the information from two quan-

tile judgments to make deductions about the third one,

and then asked the expert to validate these deductions.

If the expert did not concur with the deductions, we

provided him an opportunity to ﬁne tune the quantile

judgments.

As an example, consider a speciﬁc seed for which the

expert provided values of 15, 70, and 100 for the 10th,

50th, and 75th quantiles, respectively. The stated val-

ues of the 10th and 50th quantiles imply a mean yield

of 70 and standard deviation of 42.92 for normally dis-

tributed yields. These two values imply that there is

a 50% chance that the yield will be between 41 and

99 (the implied 25th and 75th quantile). We asked the

yield expert the following question: “Your estimate of

the 10th quantile implies that there is a 50% chance that

the yield will be between 41 and 99. If you think that

this range should be narrower, please consider increas-

ing the estimate of the 10th quantile. If you think the

range should be wider, please consider decreasing the

estimate of the 10th quantile.” We implemented this

feedback in an automated fashion so that the values

in the feedback question were generated automatically

using his quantile estimates. The expert could revisit

his input and the accompanying feedback question any

number of times before moving to the next feedback

question for the judgment for the 75th quantile (using

the deduced 35th and 85th quantile values obtained

from his judgments for the 50th and 75th quantiles).

After ﬁnishing this feedback, he moved to the next

seed. Throughout this process, we emphasized that the

objective of this ﬁne tuning was to help him reﬂect on

his estimates carefully without leading him to any spe-

ciﬁc set of numbers. Analysis showed that after this

feedback, the standard deviations reduced by 33% for

the tail quantiles in round 2, conﬁrming that the feed-

back was indeed helpful to the expert in improving the

quality of his estimates.

Bansal et al.: Using Experts’ Judgments to Quantify Risks

Operations Research, Articles in Advance, pp. 1–16, ©2017 INFORMS 11

6.2.3. Step 3: Separation of Sampling Errors Using

Bootstrapping. After elicitation was complete, we

quantiﬁed the judgmental errors by comparing the

expert’s stated values for the quantiles with the val-

ues obtained from the historical data. For our analysis

in Section 2, we assumed that the true values of the

quantiles xiwere available. However, since the num-

ber of data points for each seed at DAS was limited

(the largest sample size was 53), the quantile values

obtained from the data were subject to sampling varia-

tions that needed to be explicitly accounted for. Specif-

ically, let ˜

xidenote the value of quantile ifor the empir-

ical distribution. Then, for the true value xiand the

expert’s estimate ˆ

xi, we have the following decomposi-

tion of errors:

ˆ

xi−˜

xi(ˆ

xi−xi)+(xi−˜

xi)(11)

Total Error Judgmental Error +Sampling Error.(12)

The comparison of the expert’s assessment ˆ

xiwith

the empirical value ˜

xihas two sources of errors: the

expert’s judgmental error and the sampling error. The

judgmental error is the diﬀerence between the quan-

tile judgment and the true quantile (ˆ

xi−xi). The sam-

pling error (xi−˜

xi)captures the data variability that is

present because the empirical distribution is based on

a random sample of limited size from the population.

The expert did not see the historical data, therefore

both sources of errors can be considered to be mutually

independent.

Writing (11) in a vector form, we have ˆ

x−˜

x(ˆ

x−x)+

(x−˜

x). It follows that the total bias is equal to

E[ˆ

x−˜

x]E[(ˆ

x−x)] +E[(x−˜

x)]

δtδ+δs,(13)

where δtis the total bias, and δand δsare the expert’s

judgmental bias and the sampling bias, respectively.

The expert’s judgmental bias is computed as δδt−δs.

Similarly, the variance in the estimates of quantiles,

assuming independence of the data-speciﬁc sampling

error and the expert-speciﬁc judgmental error, is

Var[ˆ

x−˜

x]Var[(ˆ

x−x)] +Var[(x−˜

x)].

We can write this equation in matrix notation as

ΩtΩ+Ωs,(14)

Table 2. Variance-Covariance Matrix and Biases After Bootstrap Adjustment

ˆ

Ωt

113.41 50.09 46.83

50.09 42.92 51.46

46.82 51.46 93.37

ˆ

Ωs

34.42 20.71 13.50

20.71 21.00 21.16

13.49 21.16 25.20

ˆ

Ωˆ

Ωt−ˆ

Ωs

78.99 29.38 33.33

29.38 21.92 30.30

33.33 30.30 68.17

ˆ

δt9.43 0.94 −2.48 ˆ

δs−1.05 0.00 0.55 ˆ

δˆ

δt−ˆ

δs10.48 0.94 −3.03

where Ωis the matrix of covariances of judgmental

errors and needs to be estimated for use in our analyt-

ical development described earlier. This matrix is esti-

mated as ΩΩt−Ωs. The matrix Ωmust be checked

for positive deﬁniteness to be able to take an inverse to

obtain the weights using Theorem 1. We next discuss

the estimation of δtand Ωtusing DAS’s data and the

estimation of δsand Ωsusing bootstrapping. Note that

with a large number of historical observations, Ω'Ωt,

δ'δt, and the bootstrapping approach is not required.

For DAS’s data, the total bias δtand matrix Ωtwere

determined using the expert’s assessments as follows.

In each of the two rounds of elicitation, the expert’s

quantile judgments ˆ

xil (pi);i1,2,3for hybrid lwere

compared to the quantiles of the empirical distribu-

tion, ˜

xil (pi). The diﬀerences provided the total errors

ˆ

et

il ˆ

xil (pi) − ˜

xil (pi). The average error ˆ

δt

iPL

l1ˆ

eil /L

provided the total bias for each quantile. The vector

of biases ˆ

δt

iconstituted ˆ

δt. We then obtained unbiased

errors as ˆ

eu

il ˆ

et

il −ˆ

δt

i; using these, we estimated the

3×3variance-covariance matrix ˆ

Ωt. As discussed ear-

lier, a comparison of ˆ

Ωtfrom the ﬁrst round without

feedback and the second round with feedback showed

that the feedback reduced the spread of the errors sig-

niﬁcantly (by 30%). The covariance matrix ˆ

Ωtand the

bias ˆ

δtobtained after the second round are shown in

Table 2.

The sampling bias δsand the variance-covariance

matrix Ωswere estimated by bootstrapping as fol-

lows. We had data y1l,y2l, . . . , ynllfor seed land cor-

responding quantiles ˜

xil estimated using these data.

For each distribution l, we drew a sample indexed pof

size nlwith replacement from the data y1l,y2l, . . . , ynll

and obtained the quantiles for this bootstrapping sam-

ple, ˜

xilp. We repeated the process for p1,2, . . . , P

times. Then, we obtained the diﬀerences ∆ilp

(˜

xilp −˜

xil ), determined the average diﬀerence ¯

∆il

Pp∆ilp/P, and calculated the unbiased diﬀerences

∆u

ilp ∆ilp −¯

∆il . From these 3×Punbiased diﬀer-

ences, we obtained the covariance matrix ˆ

Ωsl for seed l.

To ensure a stable variance-covariance matrix ˆ

Ωsl , we

used a large value of P,P1,000,000. Finally, Ωswas

estimated as ˆ

ΩsPˆ

Ωsl /L, implying that each covari-

ance matrix ˆ

Ωsl is equally likely to be present for each

elicitation in the future. The sampling bias for quan-

tile iwas estimated as ˆ

δs

iPl¯

∆il /L. The vector of these

biases constituted ˆ

δs. For DAS’s data, the values of ˆ

Ωs

and the bias vector ˆ

δsare shown in Table 2. The esti-

mated judgmental bias ˆ

δwas obtained as ˆ

δˆ

δt−ˆ

δs,

Bansal et al.: Using Experts’ Judgments to Quantify Risks

12 Operations Research, Articles in Advance, pp. 1–16, ©2017 INFORMS

and the estimated matrix of judgmental errors ˆ

Ωwas

obtained as ˆ

Ωˆ

Ωs, and are shown in Table 2.

6.3. Implementation: Determination of Weights

For the variance-covariance matrix ˆ

Ωin Table 2and the

matrix

Z

1−1.28

1 0

1 0.67

,

Theorem 1provides the weights w∗

1[−0.18,1.51,

−0.33]for estimating the mean and w∗

2[−0.58,0.20,

0.38]for estimating the standard deviation to be used

on the expert’s judgments for the 10th, 50th, and 75th

quantiles for yield distributions for hybrid seeds with-

out historical data. For these results, the following

regime was used at DAS in 2014 for estimating the pro-

duction yield distributions of each of more than 100

hybrid varieties that did not have historical yield data.

First, the expert estimated 10th, 50th, and 75th quan-

tiles ˆ

xfor the yield distribution of that hybrid seed. He

provided judgments for these quantiles using the same

mental model that he used during calibration, i.e., he

looked at the historical statistical distribution of the

production yield of the female parent on his computer,

considered the pollinating power and other biological

factors of the male parent, and then provided the quan-

tile judgments for the hybrid.

From this information, the debiased estimates were

obtained as ˆ

qˆ

x−ˆ

δby subtracting the biases ˆ

δ1

10.48,ˆ

δ20.94,ˆ

δ3−3.03. Next, the mean and stan-

dard deviation were obtained using the weights above

on the debiased estimates, ˆ

µ1w∗

1

tˆ

qand ˆ

µ2w∗

2

tˆ

q.

Finally, these estimates were used in an optimization

framework that an in-house team was developing in

parallel to determine the optimal area of land to pro-

duce each hybrid. Since 2014, this approach has formed

the basis of decisions worth $800 million annually.

Equally important, since the approach leveraged the

expert’s experience and intuition, which he had been

using for a few years, the decision to implement the

approach at DAS was reached quickly.

6.4. Estimation of Monetary Beneﬁts Using

Managerial Decisions

6.4.1. Status Quo Approach for Comparison. Before

adopting our approach for estimating the mean and

standard deviation for yield distributions, DAS used

the following method to determine the area of land

to use to grow each hybrid. The expert provided his

estimate of the median ˆ

x2. The production manager

used this point estimate to determine the area to use as

Qh(D/ˆ

x2)f, where Dwas the demand of the seed,

ˆ

x2was the median value provided by the expert and

fwas the risk adjustment factor of 1.2 or 1.3 based on

a subjective high/low perceived uncertainty in yield.

This framework provides a benchmark for quantifying

the beneﬁt of using our approach.

6.4.2. Measures for Quantifying Beneﬁts from Our

Approach Over Status Quo.The benchmark status quo

approach aﬀected the ﬁrm’s ﬁnances systematically in

three ways. First, the proﬁt margins of the seed did not

inﬂuence the acreage decision at all even though they

clearly should aﬀect the decision. Second, only two

values of the factor fdid not completely capture the

complete range of yield standard deviations that were

present in the portfolio. After our approach was imple-

mented to estimate the mean and standard deviation,

the ﬁrm used them as inputs to a stochastic optimiza-

tion problem for expected proﬁt maximization, i.e.,

the ﬁrm determine Q∗arg maxQ{−cQ +p E[h(Q,D)]},

where his the revenue function, cis the per acre

cost, and pis the selling price per bag. This process

change was a direct consequence of the availability of

the standard deviation. One could then calculate the

optimal ratio f∗Q∗ˆ

x2/D. At Dow, these ratios varied

from 1 to 1.4, suggesting that the use of only 1.2 or

1.3 was not optimal. The dollar capital investment in

a seed is equal to: Capital Investment $4,500 ×Area,

as the per acre cost of growing seed corn is approx-

imately $4,500 (the number is modiﬁed to preserve

conﬁdentiality). A reduction in the area used for grow-

ing hybrids directly translates into a reduction in ini-

tial capital investment, with the savings being equal to

P4,500 × (Qh−Q∗), where the summation is over all

200 seeds. Over the complete portfolio, the cost savings

were signiﬁcant, as we discuss shortly. This reduction

in the cost is the ﬁrst measure for quantifying the ben-

eﬁt of our approach.

Third, as we documented in earlier sections, the yield

expert’s judgments for the median ˆ

x2has judgmental

error. When using the status quo approach, this judg-

mental error leads to an error in the calculation of the

Unadjusted Area demand/ˆ

x2. This error was further

ampliﬁed by the use of the scaling factor f>1dur-

ing the calculation of the adjusted area using Qh

(demand/ˆ

x2)f. For some hybrids, this error in the cal-

culation of adjusted area can be very large and may

result in a substantial suboptimal decision with a sub-

stantial loss in proﬁt. This loss of proﬁt is the second

measure for quantifying the beneﬁts of our approach.

The beneﬁt on the two measures was quantiﬁed

using historical decisions made at DAS, as discussed

below.

6.4.3. Analysis for Quantifying the Beneﬁts. Due to

conﬁdentiality concerns, we do not provide here the

speciﬁc numerical values for all 200 seeds, and instead,

focus on the process used and the beneﬁts observed.

Our approach was ﬁrst used in 2014 to make the

production planning decision. For a number of seeds

involved in this decision, we documented the area used

for the annual crop plan in two ways: (a) status quo

approach and (b) using yield distributions estimated

using our approach. In approach (a), we determined the

Bansal et al.: Using Experts’ Judgments to Quantify Risks

Operations Research, Articles in Advance, pp. 1–16, ©2017 INFORMS 13

area used as Qh(D/ˆ

x2)fat f1.2,1.3for the median

estimates ˆ

x2provided by the expert. In approach (b),

we estimated the mean and standard deviation of the

yield distribution from the expert’s quantile judgments

using our approach and then determined the optimal

area using a proﬁt-maximization formulation Q∗(dis-

cussed in Bansal and Nagarajan 2017), which needs

the speciﬁcation of yield distributions to determine

the optimal area. The acreage decisions obtained from

our approach were implemented at DAS along with

a record of the decisions made using the status quo

approach that would have been made in the absence of

our approach.

The beneﬁt of using our approach was estimated

using the sale data available at the end of the season.

Speciﬁcally, an in-house business analytics team com-

pared the cost of using the area that our approach

recommended with the cost of using the status quo

approach. These results showed that the annual pro-

duction investment decreased by 6%–7% using our

approach. Equally important, DAS did not see a drop

in the service levels of the seeds after the adoption of

this new approach for estimating yield distributions.

Subsequently, an analysis was performed on the

proﬁt. For this analysis, the key item was that the

demand, yield, revenue, and proﬁt for each hybrid had

been observed by the end of the year. For each hybrid,

these quantities provided the revenue if the area in the

status quo approach had been used. From this revenue,

the cost was subtracted to obtain the proﬁt. Compar-

ing the proﬁt from this status quo approach with actual

proﬁt suggested that our approach led to between 2%

and 3% improvement in proﬁt. These documented ben-

eﬁts have led to a continuous use of our approach for

estimating yield distributions at the ﬁrm. We next dis-

cuss how this approach has been integrated into DAS’s

operations, but ﬁrst, we discuss some nonmonetary

beneﬁts accrued.

6.4.4. Nonmonetary Beneﬁts. Several features of our

approach were perceived to be of managerial impor-

tance during the implementation. First, it provided

a unique quantiﬁcation of the quality of the expert’s

judgments. This quantiﬁcation was important for the

ﬁrm in understanding the beneﬁt of identifying and

training experts in other seed businesses (soybean, cot-

ton, etc.) for which new varieties are being developed.

Speciﬁcally, at DAS, the yield distributions have a vari-

ance of µ2

2≈400 on average. At ˆ

Ωshown in Table 2,

the variance w∗

1

tΩw∗

118. Using Proposition 3, it fol-

lows that our approach extracts information from the

expert’s quantile judgments that is equivalent to the

information provided by 400/Var(ˆ

µ1)400/18 ≈22

data points. We were told that this is equivalent to

approximately ﬁve to six years of test data at DAS. Sec-

ond, the approach provides a rational eﬀort to estimate

the variability in production yields, enabling the yield

expert to support his estimates for yield distributions

with scientiﬁc tools.

6.5. Integration into Firm’s Operations

After the initial implementation in 2014, DAS recog-

nized the value of formal statistical modeling and

analysis for yield forecasting and production planning

decisions. The ﬁrm created a new business analytics

group, and two members of this group were tasked

with developing optimization protocols to inform

DAS’s operations. The team was composed of trained

statisticians with experience in biostatistics. This niche

skill set was considered necessary since the yield dis-

tributions and other properties of seeds are driven by

biology, and an understanding of plant biology as well

as statistics would enable the team to develop context-

informed models.

For the annual production planning decision, the

team implements the approach in the following man-

ner. The production planning decision is made every

year a few weeks before the advent of spring. In the

weeks preceding this decision, the team obtains a list

of hybrid seeds from the seed business manager that

are under consideration for being oﬀered to the market.

The portfolio of hybrid seeds oﬀered changes annu-

ally and this information is necessary for the team to

estimate yield distributions to support the production

planning decision. The team then sends this list to the

yield expert who is located at a diﬀerent geographi-

cal location. This expert does travel back and forth to

the team’s location, nevertheless, DAS has emphasized

the development of computer-based tools that can be

accessed from anywhere. The yield expert obtains this

list and provides his judgments for yield distributions.

The team of statisticians processes these quantile judg-

ments using the process described earlier to deduce

means and standard deviations. A list of these val-

ues is then sent back to the business analytics team

that is responsible for making the production planning

decision.

The business analytics team then uses an optimiza-

tion framework to determine the number of acres that

should be used to grow each hybrid seed. Yield distri-

butions constitute the major source of stochasticity in

this model. Under this uncertainty, the model seeks to

balance the trade-oﬀ between using a very large or a

very small area. The per acre tilling and land lease cost

is high and using a large area of land needs up-front

high investment and could lead to a surplus inventory

of hybrid seeds. The use of a small area of land requires

less up-front capital investment in the production, but

can lead to shortages. Estimating the yield distribu-

tions enables the ﬁrm to optimize this trade-oﬀ in a

mathematical fashion, in addition to providing a quan-

titative decision support.

Bansal et al.: Using Experts’ Judgments to Quantify Risks

14 Operations Research, Articles in Advance, pp. 1–16, ©2017 INFORMS

7. Discussion and Future Research

7.1. Summary of Approach

In changing environments, historical data do not exist

to provide probability distributions of various uncer-

tainties. In such environments, judgments are sought

from experts. But expert judgments are prone to judg-

mental errors. In this paper, we develop an analytical

approach for deducing the parameters of probability

distributions from a set of quantile judgments pro-

vided by an expert, while explicitly taking the expert’s

judgmental errors into account.

From a theory-building perspective, the optimiza-

tion approach proposed is consistent with moment

matching, has a unique analytically tractable solution,

and is amenable for comparative static analysis. The

approach also provides an analytical foundation for

results documented numerically in the prior literature.

From a practice perspective, a salient feature of the

approach is that an expert is no longer required to pro-

vide judgments for the median and speciﬁc symmetric

quantiles studied in the literature, but can provide his

judgments for any set of quantiles. The approach also

establishes a novel equivalence between an expert’s

quantile judgments and a sample size of randomly col-

lected data; this equivalence is useful for ranking and

comparing experts objectively. Finally, the modeling

framework explains a consistent numerical ﬁnding in

the prior literature that the weights for the mean and

the standard deviation add up to 1 and 0, respectively.

Equally important, it provides for a linear pooling

of quantile judgments from multiple experts, thereby

providing a practical toolkit for combining judgments

in practice.

From an implementation perspective, the approach

has several features that make it viable for an easy

adoption by ﬁrms. First, it usesjudgments for any three

or more quantiles that an expert is comfortable pro-

viding. In a speciﬁc application at DAS, we used the

yield expert’s judgments for the 10th, 50th, and 75th

quantiles to deduce the mean and standard deviations

of a large number of yield uncertainties. The expert

chose to estimate these quantiles based on his experi-

ence with obtaining and using these quantiles in his

data analysis responsibilities. Second, the ﬁnal out-

come of the approach is a set of weights that are used to

estimate means and standard deviations as weighted

linear functions of quantile judgments. The implemen-

tation of this procedure requires simple mathemati-

cal operations that can be performed in a spreadsheet

environment, and it has led to an expedited adoption

at DAS. Third, the weights are speciﬁc to the expert and

capture how good he is at providing estimates of vari-

ous quantiles. This explicit incorporation of an expert’s

judgmental errors is useful since we can then deter-

mine how the estimated parameters (and the decision

based on this estimated distribution) will vary as the

quality of the expert’s judgmental errors improve or

deteriorate. More speciﬁcally, in using Theorem 1, one

can analytically determine how the weights wchange

when the variance-covariance matrix Ωchanges.

7.2. Other Potential Approaches

In this section, we discuss three other potential ap-

proaches to obtain mean and standard deviation from

quantile judgments: parameter estimation through

entropy minimization, by minimizing sum of absolute

errors, and by nonparametric approaches.

In relative entropy methods, the entropy of the dis-

tribution obtained from iid randomly sampled data

relative to a benchmark distribution is computed to

evaluate the similarity of two distributions. In our prob-

lem, only three imperfect quantile judgments are avail-

able from the expert. Therefore the conventional theory

available for comparing distributions with iid randomly

sampled data using entropy-based measures is not

directly applicable. Motivated by the weighted linear

approach suggested by moment matching (in Propo-

sition 5), one possibility is to estimate moments from

quantile judgments as ˆ

µjPm

i1wji ˆ

qi;j1,2, where

the quantile judgments ˆ

qicorrespond to probabilities

pi. For the normal distribution, the cross-entropy or the

Kullback-Leibler (KL) distance of the estimates ˆ

µjfrom

true values µjis given as (Duchi 2007)

KL log ˆ

µ2

µ2

+µ2

2+(µ1−ˆ

µ1)2

2ˆ

µ2

2

−1

2.

For each debiased quantile judgment, ˆ

qiµ1+ziµ2+

i, where the term iis the noise in the judgment, and

therefore E[i]0; then it follows that

µ1Em

X

i1

ˆ

qiw1iµ1

m

X

i1

w1i+µ2

m

X

i1

ziw1i+Em

X

i1

iw1i,

and since E[Pm

i1iw1i]0, this implies that (i)

Pm

i1w1i1and (ii) Pm

i1ziw1i0. Similarly, since µ2

E[Pm

i1ˆ

qiw2i], it follows that (iii) Pm

i1w2i0and (iv)

Pm

i1ziw1i1.

Using the properties (i)–(iv), the KL distance can be

expressed as

KL log µ2+Pm

i1w2ii

µ2

+µ2

2+(Pm

i1w1ii)2

2(µ2+Pm

i1w2ii)2−1

2.(15)

This KL distance is a random variable, which is a

function of the estimation errors i, thus a plausible

approach would be to select the weights wji that mini-

mize the expected value of the KL distance, E[KL]. The

limiting behavior of E[KL]provides a point of com-

parison between this approach and the one developed

earlier in this paper. As the expert becomes increas-

ingly more reliable, we have on the limit E[KL] → 0as

Var(i) → 0, for any values of w1iand w2ithat satisfy

Bansal et al.: Using Experts’ Judgments to Quantify Risks

Operations Research, Articles in Advance, pp. 1–16, ©2017 INFORMS 15

conditions (i)–(iv). Since the value of KL is nonnega-

tive by construction, on the limit any such (w1i,w2i)

minimize E[KL]. Moreover, since in the optimization,

we can select 2mweights and we have only four con-

straints, anytime we elicit more than two quantiles, in

general, we may have an inﬁnite number of optimal

weight combinations. The unique weights obtained by

our approach automatically satisfy conditions (i)–(iv),

hence they also optimize E[KL]on the limit.

With respect to the general case of this approach

(minimizing (15)), we make three observations:

1. Notice from Equation (15), that E[KL]is a nontriv-

ial function of the entire error covariance matrix Ω, and

obtaining the E[KL]-minimizing weights will require

numerical optimization.

2. The above deﬁnition of E[KL]requires knowledge

of µ2, which we do not have.

3. The uniqueness of the weights is not guaranteed.

Comparing this estimation approach with the one

proposed and implemented at Dow, we can appreci-

ate an important diﬀerence. Both approaches would

require us to estimate the covariance matrix Ωfrom

the calibration data set. But the E[KL]minimization

approach also requires knowledge of the parameter µ2,

which Dow did not have. The estimation approach

developed in Sections 3–5does not require this knowl-

edge. These challenges associated with the E[KL]mini-

mization approach will need to be addressed by future

research before the approach can be used in practice.

The problem of estimating distribution parameters

by minimizing the sum of absolute errors (instead of

the sum of squared errors) is stated as minwik Pj|Piwi k ·

ˆ

qji −ˆ

µjk |, where ˆ

µjk is the mean (k1)and stan-

dard deviation (k2)of the calibration distribution j.

The optimal weights for this model are not obtainable

in closed form, rather this problem must be solved

numerically using a linear programming formulation,

and it not always has a unique solution (Harter 1977,

Bassett and Koenker 1978, Chen et al. 2008). Further-

more, there is no direct relationship between the sum

of squared errors and sum of absolute errors for the

data. Due to these two issues, the equivalent sample

size for an expert, akin to the result in Proposition 3,

cannot be determined.

Nonparametric methods explore various functional

forms to ﬁt data, while minimizing the squared dis-

tances between the ﬁtted and true values. The Spline

ﬁtting approach ﬁts one or more splines of various

degrees to the data. The recommended functional form

for the predictive model tends to be sensitive to the

data (Härdle et al. 2012) and, in our context, may

change with the inclusion/exclusion of even one prob-

ability distribution in the calibration set. Similarly,

the additive kernel model can be sensitive to tun-

ing parameters, which need to be selected subjectively

(Härdle et al. 2012). This sensitivity and subjectivity

in model recommendation implies, in our context, that

the nonparametric model for new seeds may have to be

modiﬁed for every season, which could be undesirable

when a ﬁrm seeks to develop a stable and transpar-

ent model for a repetitive use. Finally, a direct least

squares analysis provides a strong basis for using the

linear functional form used in the paper. Proposition 5

shows that the conventional least squares formulation

to deduce mean and standard deviation from quantile

judgments for location-scale distribution results in the

estimation of mean and standard deviation as linear

combinations of the quantile judgments. Our approach

exploits this result and develops it further in the form

of tractable and ease to use results discussed in vari-

ous propositions.

7.3. Future Research

A scant but important stream of literature has quanti-

ﬁed the beneﬁt of a more reliable estimation of oper-

ational uncertainties. Akcay et al. (2011), in collabora-

tion with SmartOps Corporation, show that using the

demand information computed from 20 data points

over 10 data points for inventory decision making

reduces the operating cost by 10% (Tables 2—4, p. 307).

Our quantiﬁcation provides a new addition to this lit-

erature, especially when the information for an uncer-

tainty is obtained from an expert. In the future, this

quantiﬁcation should be sharpened using Monte Carlo

simulation studies for the seed industry as well as other

industries. Future research should also explore tighter

connections between the yield of hybrid seed produc-

tion and the genomes of both parents crossed. This

industry is making signiﬁcant investments in genetic

research, and a large amount of genomic information

for some corn varieties is becoming available. Unfortu-

nately, currently, this task is daunting as corn has one

of the most complex plant genomes with some mapped

varieties showing sequences of more than two billion

genes (Dolgin 2009); this is in stark contrast with the

sparsity of the yield data available. Finally, an impor-

tant requirement for the approach developed for a use,

in practice, is that we calibrate the experts by compar-

ing their quantile judgments with the true values for

some distributions that are speciﬁc to the context, and

for which historical data is available at the ﬁrm. How-

ever, this data may not be available in all businesses.

The future research should explore whether it is pos-

sible to calibrate experts on almanac events, and then

use this information for estimating probability distri-

butions speciﬁc to the business.

Acknowledgments

The authors gratefully acknowledge the suggestions made by

three anonymous reviewers, associate editor, and area edi-

tor Andres Weintraub, which resulted in a much improved

paper. The authors thank Dow AgroSciences, especially

Sue Gentry and J. D. Williams, for their support in this

Bansal et al.: Using Experts’ Judgments to Quantify Risks

16 Operations Research, Articles in Advance, pp. 1–16, ©2017 INFORMS

collaboration. The ﬁrst version of the paper was developed

when the ﬁrst author was visiting the Department of Supply

Chain and Operations at University of Minnesota. The Lab-

oratory for Economics, Management and Auctions (LEMA)

at Penn State provided laboratory settings to test the the-

ory developed in the paper before its ﬁeld deployment. The

authors also thank Mike Blanco, Marilyn Blanco, Murali

Haran, and Dennis Lin at Penn State for their help during a

revision.

References

Akcay A, Biller B, Tayur S (2011) Improved inventory targets in the

presence of limited historical demand data. Manufacturing Ser-

vice Oper. Management 13(3):297–309.

Ayvaci MUS, Ahsen ME, Raghunathan S, Gharibi Z (2017) Timing

the use of breast cancer risk information in biopsy decision mak-

ing. Production Oper. Management. Forthcoming.

Baker E, Solak S (2014) Management of energy technology for sus-

tainability: How to fund energy technology research and devel-

opment. Production Oper. Management 23(3):348–365.

Bansal S, Nagarajan M (2017) Product portfolio management with

production ﬂexibility in agribusiness. Oper. Res. 65(4):914–930.

Bassett G Jr, Koenker R (1978) Asymptotic theory of least absolute

error regression. J. Amer. Statist. Assoc. 73(363):618–622.

Bates JM, Granger CWJ (1969) The combination of forecasts. Oper.

Res. Quart. 451–468.

Casella G, Berger RL (2002) Statistical Inference, 2n ed. (Duxbury

Press, Paciﬁc Grove, CA).

Chen K, Ying Z, Zhang H, Zhao L (2008) Analysis of least absolute

deviation. Biometrika 95(1):107–122.

Comhaire P, Papier F (2015) Syngenta uses a cover optimizer to deter-

mine production volumes for its European seed supply chain.

Interfaces 45(6):501–513.

Dolgin E (2009) Maize genome mapped. Nature News 1098.

Duchi J (2007) Derivations for Linear Algebra and Optimization. Working

paper, University of California, Berkeley, Berkeley, CA.

Garthwaite PH, Dickey JM (1985) Double- and single-bisection meth-

ods for subjective probability assessment in a location-scale fam-

ily. J. Econometrics 29(1–2):149–163.

Granger CWJ (1980) Forecasting in Business and Economics (Academic

Press).

Härdle WK, Müller M, Sperlich S, Werwatz A (2012) Nonparametric

and Semiparametric Models (Springer , New York).

Harter HL (1977) Nonuniqueness of least absolute values regression.

Comm. Statist.-Theory and Methods 6(9):829–838.

Johnson D (1998) The robustness of mean and variance approxima-

tions in risk analysis. J. Oper. Res. Soc. 49(3):253–262.

Johnson NL, Kotz S, Balakrishnan N (1994) Continuous Univariate Dis-

tributions, Vol. 1, Wiley Series in Probability and Mathematical

Statistics: Applied Probability and Statistics (Wiley, New York).

Keefer DL, Bodily SE (1983) Three-point approximations for contin-

uous random variables. Management Sci. 29(5):595–609.

Kelton WD, Law AM (2006) Simulation Modeling and Analysis, 4th ed.

(McGraw Hill, New York).

Koehler DJ, Brenner L, Griﬃn D (2002) The calibration of expert

judgment: Heuristics and biases beyond the laboratory. Heuris-

tics and Biases: The Psychology of Intuitive Judgment (Cambridge

University Press, New York).

Lau HS, Lau AHL (1998) An improved PERT-type formula for stan-

dard deviation. IIE Trans. 30(3):273–275.

Lau HS, Lau AHL, Ho CJ (1998) Improved moment-estimation for-

mulas using more than three subjective fractiles. Management

Sci. 44(3):346–351.

Lau HS, Lau AHL, Kottas JF (1999) Using Tocher’s curve to con-

vert subjective quantile-estimates into a probability distribution

function. IIE Trans. 31(3):245–254.

Lau AHL, Lau HS, Zhang Y (1996) A simple and logical alternative

for making PERT time estimates. IIE Trans. 28(3):183–192.

Lindley DV (1987) Using expert advice on a skew judgmental distri-

bution. Oper. Res. 35(5):716–721.

O’Hagan A (1998) Eliciting expert beliefs in substantial practical

applications. J. Roy. Statist. Soc.: Ser. D (The Statistician)47(1):

21–35.

O’Hagan A (2006) Uncertain Judgements: Eliciting Experts’ Probabilities,

Vol. 35 (John Wiley & Sons, Chichester, UK).

O’Hagan A, Oakley JE (2004) Probability is perfect, but we can’t elicit

it perfectly. Reliability Engrg. System Safety 85(1–3):239–248.

Pearson ES, Tukey JW (1965) Approximate means and standard devi-

ations based on distances between percentage points of fre-

quency curves. Biometrika 52(3–4):533.

Perry C, Greig ID (1975) Estimating the mean and variance of subjec-

tive distributions in pert and decision analysis. Management Sci.

21(12):1477–1480.

Ravinder HV, Kleinmuntz DN, Dyer JS (1988) The reliability of sub-

jective probabilities obtained through decomposition. Manage-

ment Sci. 34(2):186–199.

Stevens JW, O’Hagan A (2002) Incorporation of genuine prior infor-

mation in cost-eﬀectiveness analysis of clinical trial data. Inter-

nat. J. Tech. Assessment in Health Care 18(04):782–790.

Wallsten TS, Nataf C, Shlomi Y, Tomlinson T (2013) Forecasting

values of quantitative variables. Paper presented at SPUDM24,

Barcelona, Spain, August 20, 2013.

Saurabh Bansal is an assistant professor of supply chain

management and information systems, and a faculty member

of operations research at the Pennsylvania State University.

His research focuses on developing mathematical models,

algorithms, and protocols to estimate business risks and opti-

mize business operations under risks.

Genaro J. Gutierrez is an associate professor of informa-

tion risk and operations management at McCombs School of

Business, The University of Texas at Austin, where he teaches

operations management and supply chain analytics. He is

the Director of the Executive MBA Program that McCombs

School oﬀers in Mexico City.His current research interests

include, in general, the incorporation of data analytics in

the supply chain management domain. Speciﬁc research

projects include: combination of statistical and judgmental

approaches for estimating demand, data-driven models to

optimize the supply chain for digital advertising, procure-

ment of traded commodities, and reliability models for fore-

casting and procurement of high cost spare parts. Recent

publications of Professor Gutierrez have appeared in Manage-

ment Science, Operations Research, IIE Transactions, and Euro-

pean Journal of Operations Research.

John R. Keiser is the global technical expert for corn seed

production research, and is responsible for providing guid-

ance and coordination between all corn production research

programs globally, as well as technical oversight for the NA

Production Research program. He earned a PhD in Crop Pro-

duction and Physiology from Iowa State University.