Insightful or Wishful: Lawyers' Ability to Predict Case Outcomes

Abstract

Lawyers’ litigation forecasts play an integral role in the justice system. In the course of litigation, lawyers constantly make strategic decisions and/or advise their clients on the basis of their perceptions and predictions of case outcomes. The study investigated the realism in predictions by a sample of attorneys (n=481) across the United States who specified a minimum goal to achieve in a case set for trial. They estimated their chances of meeting this goal by providing a confidence estimate. After the cases were resolved, case outcomes were compared with the predictions. Overall, lawyers were overconfident in their predictions, and calibration did not increase with years of legal experience. Female lawyers were slightly better calibrated than their male counterparts and showed evidence of less overconfidence. In an attempt to reduce overconfidence, some lawyers were asked to generate reasons why they might not achieve their stated goals. This manipulation did not improve calibration.

Full text

INSIGHTFUL OR WISHFUL:
Lawyers’ Ability to Predict Case Outcomes
Jane Goodman-Delahunty
Charles Sturt University, Manly, New
South Wales, Australia
Pa¨r Anders Granhag
University of Gothenburg, Gothenburg,
Sweden
Maria Hartwig
John Jay College of Criminal Justice Elizabeth F. Loftus
University of California
Lawyers’ litigation forecasts play an integral role in the justice system. In the course of
litigation, lawyers constantly make strategic decisions and/or advise their clients on the
basis of their perceptions and predictions of case outcomes. The study investigated the
realism in predictions by a sample of attorneys (n⫽481) across the United States who
specified a minimum goal to achieve in a case set for trial. They estimated their chances
of meeting this goal by providing a confidence estimate. After the cases were resolved,
case outcomes were compared with the predictions. Overall, lawyers were overconfident
in their predictions, and calibration did not increase with years of legal experience.
Female lawyers were slightly better calibrated than their male counterparts and showed
evidence of less overconfidence. In an attempt to reduce overconfidence, some lawyers
were asked to generate reasons why they might not achieve their stated goals. This
manipulation did not improve calibration.
Keywords: legal decision making, case predictions, confidence judgments,
metacognitive realism
Outcome Predictions in Legal Decision Making
Human cognition is not only retrospective but also prospective: People look
toward an imagined future and make various assessments and judgments on the basis
of possible future events (Flavell, 2004; Kahneman, 1973; Meeks, Hicks & Marsh,
2007). As an important part of prospective reasoning, people establish goals that they
attempt to accomplish (Gollwitzer & Schaal, 1998). These goals are associated with
confidence judgments, which are defined as subjective assessments of the probability
of attaining the desired goal (Koriat, 2002; Perfect & Schwartz, 2002).
In the course of regular legal practice, judgments and meta-judgments of
future goals are an important aspect of a wide range of litigation-related decisions
Jane Goodman-Delahunty, Australian Graduate School of Policing and School of Psychology,
Charles Sturt University, Manly, New South Wales, Australia; Pa¨r Anders Granhag, Department of
Psychology, University of Gothenburg, Gothenburg, Sweden; Maria Hartwig, Department of
Psychology, John Jay College of Criminal Justice, New York; Elizabeth F. Loftus, Psychology &
Social Behavior, Criminology, Law & Society, Cognitive Sciences, and School of Law, University
of California at Irvine.
This research was supported by a grant from the Fund for Dispute Resolution. We are grateful
to Patti Rognlin, Norm Thompson, Susan Rhodes, Steve Teller, and many other students for data
collection, and to Marijke Malsch, Steve Penrod, Nicole Doering, and Berenike Waubert de Puiseau
for helpful advice on the article.
Correspondence concerning this article should be addressed to Jane Goodman-Delahunty,
Australian Graduate School of Policing, Charles Sturt University, Manly Campus, P.O. Box 168,
Manly, New South Wales, 2088 Australia. E-mail: jdelahunty@csu.edu.au
Psychology, Public Policy, and Law
2010, Vol. 16, No. 2, 133–157 © 2010 American Psychological Association
1076-8971/10/$12.00 DOI: 10.1037/a0019060
133

(English & Sales, 2005). From the moment when a client first consults a lawyer
until the matter is resolved, lawyers must establish goals in a case and estimate the
likelihood that they can achieve these goals. The vast majority of lawyers
recognize that prospective judgments are integral features of their professional
expertise. For example, a survey of Dutch criminal lawyers acknowledged that
90% made predictions of this nature in some or all of their real-life cases (Malsch,
1990). The central question addressed in the present study was the degree of
accuracy in lawyers’ forecasts of case outcomes. To explore this question, we
contacted a broad national sample of U.S. lawyers who predicted their chances of
achieving their goals in real-life cases and provided confidence ratings in their
predictions.
Prediction of success is of paramount importance in the system for several
reasons. In the course of litigation, lawyers constantly make strategic decisions
and/or advise their clients on the basis of these predictions. Attorneys make
decisions about future courses of action, such as whether to take on a new client,
the value of a case, whether to advise the client to enter into settlement negoti-
ations, and whether to accept a settlement offer or proceed to trial. Thus, these
professional judgments by lawyers are influential in shaping the cases and the
mechanisms selected to resolve them. Clients’ choices and outcomes therefore
depend on the abilities of their counsel to make reasonably accurate forecasts
concerning case outcomes. For example, in civil cases, after depositions of key
witnesses or at the close of discovery, the parties reassess the likelihood of success
at trial in light of the impact of these events.
In summary, whether lawyers can accurately predict the outcome of a case has
practical consequences in at least three areas: (a) the lawyer’s professional
reputation and financial success; (b) the satisfaction of the client; and (c) the
justice environment as a whole. Litigation is risky, time consuming, and expen-
sive. The consequences of judgmental errors by lawyers can be costly for lawyers
and their clients, as well as an unnecessary burden on an already overloaded
justice system. Ultimately, a lawyer’s repute is based on successful calculations of
case outcome. A lawyer who advises clients to pursue litigation without delivering
a successful outcome will not have clients for long. Likewise, a client will be most
satisfied with a lawyer who is accurate and realistic when detailing the potential
outcomes of the case. At the end of the day, it is the accurate predictions of the
lawyer that enable the justice system to function smoothly without the load of
cases that were not appropriately vetted by the lawyers.
Several lines of research support the proposition that lawyers’ assessments of
goals are central in the legal system. Some research singled out the attorney’s
estimate of the probability of success as the most crucial variable in shaping
decisions whether to litigate or settle a case in controversy (Schuck, 1986;
Wittman, 1988). An analysis of factors that influence assessments of outcomes in
contingent-fee cases revealed three perspectives of salience: that of the client; that
of an outside observer (e.g., a judge); and that of the lawyer, on the basis of past
experience and knowledge of how similar cases were resolved (Jay, 1989). The
lawyer’s viewpoint was regarded as the most important factor in shaping out-
comes in civil contingent-fee cases.
In civil litigation, both for plaintiffs who often retain counsel on a contingent-
fee basis and for defendants who typically do not, cost control and efficient
134 GOODMAN-DELAHUNTY, GRANHAG, HARTWIG, AND LOFTUS

representation may be achieved by resolving the case before trial. This can be
accomplished through negotiation, mediation, arbitration, or some other form of
alternative dispute resolution. In criminal cases, plea bargaining is routine
(Horowitz & Willging, 1984; Ross, 1970). A lawyer who cannot accurately
predict the outcome of a case or who does not thoroughly and efficiently
appreciate the litigation risks may ignore alternatives to trial and advise the client
to reject reasonable settlement offers. A lawyer who underestimates potential
outcomes may advise the client to accept an unreasonably lower amount in
settlement than is warranted.
The National Conference of Federal Trial Judges and the American Bar
Association conducted a survey to determine which factors were perceived by
lawyers as the most instrumental in facilitating settlement. The factor cited more
than twice as frequently as any other factor was the ability of the settlement judge
to objectively evaluate the strengths and weaknesses of a case (Brazil, 1985).
Although lawyers sometimes acknowledge that their ability to objectively assess
their cases may be suboptimal, settlement judges rely on lawyers to make these
determinations. Failure of lawyers to make proper predictions can impair settle-
ment and litigation performance (Harley, 2007).
Confidence and Calibration in Lawyers’ Predictions
We contend that lawyers’ forecasts related to the achievement of important
goals play a pivotal role in practical legal decision making, but little is known
about the reliability of these predictions. To what extent are they realistic?
Psychological studies of human decision-making processes in a wide variety of
contexts have revealed that overconfidence is a ubiquitous phenomenon. Re-
cently, psychological researchers distinguished three forms of overconfidence:
overestimation, overplacement, and overprecision (Moore & Healy, 2008). Over-
estimation is the inflated perception of one’s ability, performance, or chance of
success. Overplacement describes the relative judgment of oneself in comparison
with others. Overprecision is “excessive certainty regarding the accuracy of one’s
beliefs” (Moore & Healy, 2008, p. 4).
Overconfidence has been observed in social judgments, self-predictions, and
professional predictions, in retrospective as well as prospective judgments (All-
wood & Granhag, 1999; Dunning, Griffin, Milojkovic, & Ross, 1990; Lichten-
stein & Fischhoff, 1977; Paese & Feuer, 1991; Vallone, Griffin, Lin, & Ross,
1990; Von Winterfeldt & Edwards, 1986). With regard to confidence judgments
about achieving future goals, this calibration deficit implies that decision makers
systematically overestimate their capacity to reach stated goals.
Metacognitive Realism
Research has identified a number of predictors of calibration (Dougherty,
2001; Gonza´lez-Vallejo & Bonham, 2007; Hogarth, 2001; Hogarth, Gibbs,
McKenzie, & Marquis, 1991). For instance, underconfidence or underestimation
is common when people estimate their performance on easy tasks or, in the case
of skilled individuals, on tasks on which they regard their chances of performing
well as high (Moore & Healy, 2008). Overestimation is more common on difficult
tasks.
135LAWYERS’ ABILITY TO PREDICT CASE OUTCOMES

Outcome feedback is an important factor in the development of well-
calibrated metacognitive judgments (Hogarth et al., 1991; Murphy & Winkler,
1971; Keren, 1987; Stone & Opel, 2000). To increase calibration accuracy,
one needs to learn whether previous predictions were correct. Although
lawyers might receive some outcome feedback, several factors bias their
judgments in the direction of overconfidence. First, particular professional
demands within the framework of the adversarial legal system may contribute
to this tendency: Lawyers have an ethical obligation to advocate zealously in
representing their clients. In this endeavor, lawyers must behave persuasively
and express confidence in their position. Academics have noted, “lawyers need
to feel and display overconfidence in order to attract clients, and, later, to keep
those clients convinced that their interests are well served” (Loftus & Wa-
genaar, 1988, p. 450; see also Simon, 1988). These attitudinal demands for
high levels of confidence might skew lawyers’ reasoning toward overconfi-
dence even when they attempt to realistically assess the likelihood of success.
Confidence may also increase over time to justify commitment to a goal
(Brehm, 1956). Previous research revealed that expert predictions of the
likelihood of events were influenced by the person on whose behalf they were
working and that overconfidence was more prevalent when the expert had
some degree of commitment to, and emotional investment in, the outcome
(Rehm & Gadenne, 1990). More generally, lawyers may engage in wishful
thinking: They might believe a certain outcome is probable simply because of
a desire to reach it (Babad, Hills, & O’Driscoll, 1992; Kunda, 1990).
Another factor that might affect the realism of lawyers’ assessments of
future goals is perception of control (Dollinger, 2008; Langens, 2007). The
extent to which an individual believes he or she can take steps to increase the
likelihood of a desirable outcome has been shown to bias confidence estimates
in those outcomes. When an event is perceived to be controllable, overcon-
fidence is likely (Weinstein, 1980, 1982, 1983). This bias is linked to what
Langer (1975) called an illusion of control, defined as “an expectancy of a
personal success probability inappropriately higher than the objective proba-
bility would warrant” (p. 313). In more recent studies, these results held only
in easy tasks, perhaps because the illusion of control is stronger. On perform-
ing hard tasks, people tend to perceive their performance as worse than it
really is compared with that of others (Moore & Cain, 2007). Lawyers may
underestimate or fail to appreciate the extent to which situational factors (e.g.,
unpredictable judges or juries) may rob them of the ability to control the
outcome (cf. Dunning et al., 1990).
Temporal circumstances may also influence the realism of judgments of
success. Predictions can be made long before the outcome is known or relatively
close to the outcome. The temporal sequence of predicting has been identified as
an important factor affecting confidence estimates (Braun & Yaniv, 1992; Gilov-
ich, Kerr, & Husted Medvec, 1993; Hogarth & Makridakis, 1981; Rehm &
Gadenne, 1990). Events more distant in time are judged less probable than those
closer in time (Keren & Raaijmakers, 1988; Milburn, 1978; Rehm & Gadenne,
1990). Thus, lawyers’ confidence levels about achieving their goals may increase
as the trial date approaches.
136 GOODMAN-DELAHUNTY, GRANHAG, HARTWIG, AND LOFTUS

Empirical Research on Calibration in Lawyers’ Predictions
The findings on lawyers’ predictions in three published studies on this topic
reflect the general finding in judgment and decision-making research that people
display a tendency for overconfidence. In the first study, criminal defense lawyers
attending legal education seminars stated a goal that they hoped to achieve on
behalf of a client and provided a confidence estimate expressing the likelihood
that they would achieve this goal (e.g., a 75% chance of an acquittal). The results
showed evidence of overconfidence, which was replicated in a second sample of
civil defense lawyers (Loftus & Wagenaar, 1988).
A larger study of decision making by criminal lawyers working in the
context of an inquisitorial legal system in the Netherlands demonstrated that
lawyer confidence was somewhat related to case outcomes: In that sample,
lawyers with high confidence levels were more likely to be successful than
their counterparts whose confidence estimates were lower, indicating some
predictive validity in their judgments (Malsch, 1990). The influence on
calibration of demographic variables (e.g., age, gender, years of practice,
specialization or area of expertise) was explored. Older lawyers (regardless of
years in practice) were better calibrated than younger colleagues. Female
lawyers were significantly better calibrated than male lawyers, and specialized
lawyers were better calibrated than general practitioners. A link between type
of criminal case to metacognitive realism also emerged, in that lawyers were
more overconfident in cases involving violent crimes against property and
drug-related offenses (compared with nonviolent crimes against property and
violent crimes against persons, such as assault).
Overview of the Present Research
The present research extends previous studies on realism in lawyers’ pro-
spective judgments by exploring predictions of outcome success in a large
national sample of U.S. lawyers. These lawyers represented parties in civil and
criminal cases (i.e., members of the plaintiff’s and the defense bar in civil cases
and prosecutors and defense counsel in criminal cases). We predicted that lawyers
would be prone to overconfidence, and we sought to further explore factors linked
to calibration in previous research (Gilovich et al., 1993; Malsch, 1989).
Assuming that we would observe more overconfidence than underconfidence,
we examined whether a cognitive intervention task would reduce overconfidence
and improve calibration. For example, social judgments have been corrected by
requiring participants to consider the opposite (Lord, Lepper, & Preston, 1984).
The technique we used involves encouraging participants to generate arguments
counter to their own predictions and has been used successfully in the past to
improve realism in confidence judgments for general knowledge questions (Ko-
riat, Lichtenstein, & Fischhoff, 1980; Vreugdenhil, 1993). This study built on
previous research by investigating the extent to which this debiasing technique
was associated with more realistic levels of confidence in the prospective judg-
ment of lawyers.
137LAWYERS’ ABILITY TO PREDICT CASE OUTCOMES

Method
Participants
Participants were 481 litigating attorneys, including new graduates and seasoned
practitioners, from 44 states across the United States. Eligible participants were
identified by means of two sampling procedures. The majority (59%) were contacted
in person at continuing legal education (CLE) seminars. Data were collected at 19
different CLE seminars in diverse geographic regions. The remaining participants
(41%) were contacted by telephone on the basis of information sampled randomly
from local bar directories for states located in the eastern, western and midwestern
regions of the country. An introductory letter was mailed to 1,700 randomly selected
lawyers inviting them to participate in the study.
For several reasons, potential participants who were mailed a letter did not
participate. First, a group of 523 (30.8%) potential participants could not be
reached. In some cases, the letter was returned, indicating that the attorney was no
longer at the listed address; in other cases, the telephone interviewer was informed
that the attorney had moved without providing forwarding information. Some
attorneys were never available for an interview, despite five or more attempts to
reach them. This left a total of 1,177 potential participants contacted by mail.
A sizeable number of the 1,177 (272, or 23%) attorneys who were contacted
were ineligible either because their practice did not include trial work or because
they had no cases expected to come to trial within the study period. A third group
of attorneys declined to participate, either at the outset of the interview or at some
later point in the procedure. For example, 78 (6.6%) attorneys refused to disclose
their minimum goal or to express their confidence in attaining their goal. In all,
859 lawyers completed the initial survey. Further attrition occurred because many
of the 859 lawyers who participated in the initial survey did not complete the
follow-up interview within the study period. A substantial number of their cases
(331, or 39%) were omitted from further analysis because the trial date in the case
selected was continued beyond the termination date for the study. A small number
of cases (46, or 5%) that were concluded within the study period were excluded
from further analysis for other reasons. In some of these cases, substitute counsel
had been appointed, so individual calibration measures were infeasible. In others,
the initially cooperative lawyers simply declined to continue participating in the
study. A handful of lawyers who changed business addresses between the time of
the initial interview and the time of the expected trial date were excluded, because
no forwarding address was available and they could not be located. Finally, a few
cases were excluded because the lawyers were unable to recall which case it was
that they had selected to discuss or because the lawyers were unable to report any
information about the case outcome. In all, follow-up interviews were completed
with 56% (481 of 859) of the lawyers who had completed the initial interview.
Survey Questionnaires and Procedures
Participants were screened for eligibility on the basis of whether they had a
case that was expected to go to trial within a specified time period (6 to 12
months). Two questionnaires were used: one for telephone interviews and one for
in-person contacts.
138 GOODMAN-DELAHUNTY, GRANHAG, HARTWIG, AND LOFTUS

First, attorneys were asked to provide a code name (e.g., Explosion, Drug-
bust) for a case in litigation to help them identify that case at a later date and
enable them to protect client confidentiality. Second, they were asked to provide
a brief description of the type of case, the party represented, and to answer the
following question: “What would be a win situation in terms of your minimum
goal for the outcome of this case?” Then, lawyers indicated their degree of
certainty that they would achieve the stated minimum goal, or something better,
by answering the following question: “From 0 to 100%, what is the probability
that you will achieve this outcome or something better?”
As mentioned, we included an experimental manipulation and randomly
assigned participants to one of two conditions. In the reasons condition, partici-
pants were asked to generate reasons why they might fail to achieve their
minimum goal before indicating how likely it was that this goal would be
achieved. In the no-reasons condition, participants generated reasons after the
estimation task.
However, lawyers in the no-reasons groups at CLEs were not invited to
consider reasons why they might not achieve their minimum goal. This procedure
was followed because when the survey questionnaires were distributed in person,
there was no way to control the order in which the participants elected to read
and/or respond to the written questions. For analysis, data from the telephone and
in-person reasons groups who generated reasons before estimating the probability
of success were collapsed to form a single “reasons” group. Similarly, data from
the telephone and in-person interview groups who did not produce reasons before
making their predictions were collapsed to form a single “no-reasons” group.
Call sheets were prepared for follow-up interviews scheduled after the date
the case was expected to come to trial. The follow-up interviews were conducted
by telephone with a structured questionnaire. A combination of open- and closed-
ended questions elicited factual information about how the case was resolved, the
negotiation process, and the final case outcome or resolution. Other questions
sought subjective information, such as the attorneys’ perception of the outcome,
any changes the participants would make if they had the opportunity to retry the
case, and what factors affected the outcome.
Results
Cases Included in the Study
We present data from 481 cases. More information was gathered about civil
cases (70% of the total sample) than criminal cases (30%), as shown in Table 1.
Table 1
Lawyers in the Final Study Sample by Party Represented
Type of counsel Frequency %
Civil plaintiff 182 38
Civil defendant 155 32
Criminal prosecutor 79 16
Criminal defendant 65 14
Total 481 100
139LAWYERS’ ABILITY TO PREDICT CASE OUTCOMES

The 337 civil cases fell into several broad categories, the most common of which
were tort cases (personal injury/negligence, motor vehicle accidents, product
liability, malpractice, misrepresentation, etc.) constituting 57% of the civil cases.
Contract cases (construction, real estate, etc.) accounted for 19% of the civil
cases. Approximately 9% of the civil cases involved workplace disputes (discrim-
ination, worker’s compensation, etc.) and approximately 7% consisted of family
dissolutions. The balance of the civil cases (8%) involved miscellaneous contro-
versies over matters such as wills and trusts; copyright; trademark and patent law;
and bankruptcy, taxation, or administrative matters.
Of the 144 criminal cases studied, the most frequently encountered crimes
were those against persons (homicide, rape, robbery, assault and battery) com-
prising 64% of the criminal cases. Victimless crimes and crimes against property
accounted for the remaining 36% of the criminal cases studied (controlled
substances, misdemeanors, etc.).
Method of Case Resolution
Not surprisingly, cases were sometimes resolved by trial and sometimes by
settlement. The amount of time devoted to negotiation in civil cases exceeded that
in criminal cases by a factor of four. A high proportion of the lawyers (80%)
reported that settlement negotiations had taken place, and approximately three
fifths of the cases (59%) were resolved by some form of out-of-court settlement.
Thirty-one percent of the cases were tried in court. An additional 10% of the cases
did not proceed to trial but were resolved through dismissal, summary judgment,
or some other legal procedural mechanism.
Minimum Goals
Responses to questions about lawyers’ minimum goals varied considerably
depending on the type of case under consideration and whether the attorney
represented a party initiating or defending against the claims. In civil cases, the
stated minimum goals were coded into two major categories. By far, the majority
of the lawyers (84%) were seeking monetary awards on behalf of their clients.
This category included cases in which the attorney represented a civil defendant
and the goal was a defense verdict. In 16% of the civil cases, the goals were
nonmonetary, such as injunctive relief, specific performance of contractual obli-
gations, a mistrial, or dismissal of the claims before trial.
In criminal cases, the goals of the lawyers were more diverse: Forty-nine
percent sought convictions of the crimes charged (the prosecution), and 25% of
the lawyers sought acquittals (the defense bar). A further 10% sought convictions
of lesser included offenses; 9% sought mistrials; and the remaining 7% sought
miscellaneous goals, such as an out-of-court resolution.
Confidence in Achieving Minimum Goals
The majority of the participants expressed confidence estimates that exceeded
50% (mean and median estimates of 64% and 70%, respectively). Reported case
outcomes were initially classified in one of three groups, depending on whether
the lawyers failed to meet their minimum goals, achieved their goals, or exceeded
their goals. In 32% of the cases, the final case outcome matched the goal set by
140 GOODMAN-DELAHUNTY, GRANHAG, HARTWIG, AND LOFTUS

the lawyers. Among the remaining cases, 24% of the outcomes exceeded the
lawyers’ minimum goals, and 44% of the outcomes were less satisfactory than
the minimum goals set by the lawyers. A number of lawyers who did not achieve
the predicted goal erred on the side of underconfidence, because their minimum
goals were too modest. However, a larger proportion of the prediction deficits
came from lawyers who erred in the direction of overconfidence.
To investigate calibration (i.e., whether lawyers who provided high confi-
dence estimates were more or less accurate in their judgment than lawyers who
provided low or moderate confidence estimates), we analyzed the data within
different confidence intervals. The following six confidence intervals were used:
0–45%, 46–55%, 56–65%, 66–75%, 76–85% and 86–100%. The number of
participant lawyers producing confidence estimates in each of the six confidence
intervals is displayed in Figure 1.
As is noted in Figure 2, lawyers who provided a low confidence estimate
(0–45%) underpredicted their chances of success. The mean estimated confidence
of these 65 individuals was nearly 27%, but their success rate was significantly
higher, close to 48%, t(64) ⫽3.31, p⬍.01. By comparison, lawyers whose initial
confidence estimates were in the range of 46–55% and 56–65% tended to be
relatively well calibrated, t(96)⫽⫺.09, p⫽.92; and t(64) ⫽⫺1.15, p⫽.26,
respectively.
For the three confidence intervals above 65%, lawyers were biased in the
direction of overconfidence (see Figures 2): For confidence intervals 66–75%,
t(118) ⫽⫺2.49, p⬍.05; for confidence intervals 76–85%, t(79) ⫽⫺4.26, p⬍
.001; and for confidence intervals 86–100%, t(54) ⫽⫺3.75, p⬍.001. In
summary, far more lawyers were susceptible to the overconfidence bias than to the
underconfidence bias. In general, the higher the expressed level of confidence, the
greater the overconfidence.
Female Versus Male Attorneys
In the total survey sample, men outnumbered women by four to one (79% vs.
21%). The 361 male participants who reported years of experience were more
Figure 1. Number of participants with estimates of success in each of six confi-
dence intervals.
141LAWYERS’ ABILITY TO PREDICT CASE OUTCOMES

experienced on average (mean years of legal experience ⫽14.8, SD ⫽9.9) than
were the 90 female participants (mean years of legal experience ⫽7.1, SD ⫽4.8),
t(449) ⫽⫺7.13, p⬍.001. A comparison of initial confidence estimates by gender
revealed no differences (mean confidence was 65% for female lawyers and 64%
for male lawyers), t(479) ⫽0.56, p⫽.58.
As illustrated in Figure 3, consideration of the case outcomes showed a
different pattern for male versus female attorneys. Proportionately, although
Figure 2. Mean confidence and achievement of minimum goal, by confidence
intervals (percentage). Objective success reflects proportion meeting or exceeding
minimum goals within respective confidence intervals. Asterisks denote significant
differences between confidence expressed and goal achievement.
ⴱ
p⬍.05.
ⴱⴱ
p⬍
.01.
ⴱⴱⴱ
p⬍.001.
Figure 3. Calibration curves for male lawyers (n⫽382) and female lawyers (n⫽99).
142 GOODMAN-DELAHUNTY, GRANHAG, HARTWIG, AND LOFTUS

nonsignificantly, more female attorneys achieved their minimum goals than did
male attorneys (64% vs. 55%), ␹
2
(1, N⫽481) ⫽2.40, p⫽.12.
Female attorneys whose confidence estimates were below 65% had a nonsig-
nificant tendency toward underconfidence, t(39) ⫽1.69, p⫽.10. Female lawyers
whose confidence estimates ranged from 66% to 75% were well calibrated, as
indicated by the fact that the objective mean for achieved goals was 72%, t(28) ⫽
⫺0.14, p⫽89. Judgments above 75% by female lawyers were biased in the
direction of overconfidence, t(30) ⫽⫺2.44, p⬍.05. By comparison, male
lawyers were significantly underconfident for confidence estimates below 45%,
t(49) ⫽3.18, p⬍.01; rather well calibrated for estimates between 46% and 55%,
t(79) ⫽⫺0.28, p⫽.78; and rather well calibrated for estimates between 56% and
65%, t(57) ⫽⫺1.43, p⫽.16. Judgments made by male attorneys whose
confidence estimates exceeded 66% were systematically biased in the direc-
tion of overconfidence: For confidence estimates between 66% and 75%,
t(89) ⫽⫺2.74, p⬍.01; for confidence estimates between 75% and 85%,
t(61) ⫽⫺4.15, p⬍.001; and for confidence estimates between 86% and
100%, t(41) ⫽⫺3.29, p⬍.01.
Thus, a major source of the gender differences was that female lawyers were
susceptible to the overconfidence bias only when the predicted likelihood of
success was high. Compared with their female colleagues, male lawyers were less
able to discriminate between cases in which they had a moderate probability of
success (65–75%) and those in which they had a high probability of success (over
75%).
Effects of Experience
The measure of experience used in this study was number of years in legal
practice. This number was used as a proxy for general expertise, because no
inquiries were made as to how many cases similar to the study case had the lawyer
handled previously or whether they were specialized in any area. Thus, if a lawyer
participating in the survey had several years of legal experience but selected as the
case for prediction one that was out of the ordinary in his or her practice, this
measure could be deceptive. On the other hand, expertise acquired in one
substantive legal domain may arguably transfer to another substantive domain.
The range of experience among lawyers included in the study was consider-
able, from 0 to 45 years of practice, with a mean of 13.3 years in practice (SD ⫽
9.7). Years of experience varied somewhat, depending on whether the responding
attorneys practiced civil or criminal law and the parties whom they represented.
For example, there was a clear difference in years of experience between lawyers
handling civil cases (M⫽14.9 years, SD ⫽9.9) and lawyers handling criminal
cases (M⫽8.8 years, SD ⫽7.4), t(449) ⫽6.23, p⬍.001. There were no
significant differences between the years of experience of attorneys representing
civil plaintiffs (M⫽15.7 years, SD ⫽10.2) versus those representing civil
defendants (M⫽14.0 years, SD ⫽9.4), t(327) ⫽1.63, p⫽.10. Attorneys
prosecuting criminal cases (M⫽6.8 years, SD ⫽6.4) had significantly less
experience than did criminal defense lawyers (M⫽11.1 years, SD ⫽8.0),
t(120) ⫽⫺3.33, p⬍.001.
143LAWYERS’ ABILITY TO PREDICT CASE OUTCOMES

If low confidence or uncertainty about case outcomes were correlated with
lack of legal expertise, one might expect less experienced lawyers to express less
confidence on average than the experienced lawyers. To investigate this hypoth-
esis, we categorized lawyers in two groups: one consisting of lawyers with 10 or
fewer years of legal practice (n⫽220) and one consisting of lawyers with more
than 10 years of legal practice (n⫽231). The mean estimated confidence was
65% for those lawyers with 10 or fewer years of practice (SD ⫽20.5%) and 63%
for those with over 10 years of practice (SD ⫽20.3%), t(449) ⫽1.04, p⫽30. The
proportion achieving their goals did not differ between the groups (M⫽56.8% for
less experienced lawyers and M⫽55.4% for more experienced lawyers), ␹
2
(1,
N⫽451) ⫽0.09, p⫽.76. The full calibration curves based on differential
experience are displayed in Figure 4.
It is interesting that, for confidence levels above 60%, both lawyers with less
experience (M⫽76.5%, SD ⫽10.0%) and lawyers with more experience (M⫽
75.4%, SD ⫽10.7%) were clearly overconfident: t(151) ⫽⫺3.86, p⬍.001, and
t(146) ⫽⫺4.46, p⬍.001, respectively. The pattern was replicated for high
confidence estimates (over 80%): Less experienced lawyers (M⫽85.9%, SD ⫽
6.4%) were prone to an overconfidence bias, t(62) ⫽3.88, p⬍.001; as were more
experienced lawyers (M⫽86.0%, SD ⫽6.6%), t(57) ⫽⫺4.43, p⬍.001.
In conclusion, the data provided no support for the hypothesis that lawyers
with more practical experience are better calibrated than lawyers with less
experience. Years of practice did not count for much, at least as far as calibration
goes.
Figure 4. Calibration curves for lawyers with more than 10 years of experience
(n⫽231) and lawyers with 10 or fewer years of experience (n⫽220).
144 GOODMAN-DELAHUNTY, GRANHAG, HARTWIG, AND LOFTUS

Civil Versus Criminal Cases
Data from lawyers handling 337 civil cases were examined separately to
determine whether those lawyers were better calibrated in predicting their success
than the overall sample. The mean confidence estimate from this group was 65.0%
(SD ⫽18.0%), and the percentage that achieved their goals was 56.7%.
As a group, civil lawyers exhibited the same trend toward over- and under-
confidence, as did the entire group of participating lawyers. For example, as can
be seen in Figure 5, lawyers who were highly confident in achieving the goals they
set (estimates exceeding 75%, mean estimate ⫽80.7%, SD ⫽6.6%), were
significantly overconfident as they achieved their goals in only 62.6% of the cases,
t(146) ⫽⫺4.52, p⬍.001.
Lawyers handling 144 criminal cases had a mean confidence estimate of
62.6% (SD ⫽25.2%), and the percentage that achieved their goal was 56.9%.
There was no difference in confidence estimates between civil and criminal
lawyers, t(479) ⫽1.16, p⫽.25. There was no difference in success rate between
civil and criminal lawyers, ␹
2
(1, N⫽481) ⫽0.00, p⫽.96.
Confidence as a Function of Temporal Distance to Trial
To determine whether lawyers’ confidence levels differed as a function of the
distance in time to trial, we correlated the estimated probability of success with
the number of months elapsed between prediction and trial date. Contrary to our
expectations, the correlation was nonsignificant, r(427) ⫽.04, p⫽.40, indicating
no relation between confidence levels and temporal distance to trial.
Figure 5. Calibration curves for civil cases (n⫽337) and criminal cases (n⫽144).
145LAWYERS’ ABILITY TO PREDICT CASE OUTCOMES

Calibration by Type of Case
Civil cases. Analyses of calibration by case type were possible to a limited
extent, depending on the number of cases of a similar nature included in the study.
Several categories of the tort cases in which defendant’s negligence was at issue
were aggregated into a general negligence case group (n⫽101). Approximately
50% of the lawyers responsible for these cases achieved their litigation goals,
although the mean confidence estimate of the lawyers was 65%. Again, judgment
errors were concentrated on the upper end of the scale.
With respect to contract cases (n⫽65) the same trend toward underconfi-
dence at the lower end of the scale and overconfidence at the upper end of the
scale was present.
Criminal cases. Criminal cases were divided into two major categories:
crimes against property or victimless crimes (burglary, forgery, arson, possession
of controlled substances, DWIs, etc.) and crimes against persons (homicide,
assault, sexual offenses, etc.).
When crimes against property or victimless crimes were in issue, lawyers
were realistic in their estimates. Mean confidence estimates were 59.1% (SD ⫽
29.3%), and the success rate was 47.9%, which did not differ significantly,
t(47) ⫽⫺1.54, p⫽.13. When focusing on crimes against persons, the mean
confidence estimate was 63.6% (SD ⫽23.1%). In this group, 61.1% achieved
their goal, t(89) ⫽⫺0.48, p⫽.63 suggesting general calibration.
Subjective Versus Objective Success and Attorney Reactions to
Case Outcomes
In the follow-up phase after the case resolved, lawyers described the case
outcome and whether they had achieved their minimum goal. They were not
explicitly reminded of their earlier stated goal. Occasionally they claimed they
had achieved their goal, even when an explicit comparison proved they had not.
To explore these mismatches, we devised two measures of success. Objective
success was measured by comparing the reported outcome of the case with the
originally stated goal. The measure was coded as a binary variable for which 1 ⫽
less satisfactory than the original goal (“a loss”) and 2 ⫽equal/better than
original goal (“a win”). A subjective success measure was obtained from the
lawyers’ self-reports as to whether they achieved their minimum goals. As is
reflected in Table 2, the objective success measure confirmed the tendency of the
lawyers toward overconfidence. Lawyers’ estimated probability for success (i.e.,
their a priori estimate) was 64.3% (SD ⫽20.4%), but the objective success rate
was 56.8%, which was significantly different, t(480) ⫽3.34, p⬍.001. The mean
subjective measure of success was 66.1%, and when confidence estimates were
compared with this subjective measure of success, the overconfidence bias was
replaced by a tendency toward underconfidence, t(480) ⫽68.36, p⬍.001.
Furthermore, percentages of subjective success were significantly higher in all
categories than percentages of objective success (see Table 2).
In the follow-up interviews, lawyers rated their feelings about the case
outcomes on a scale ranging from 1 to 5, on which 1 ⫽very disappointed,2⫽
somewhat disappointed,3⫽neutral,4⫽somewhat pleased, and 5 ⫽very
pleased. Results showed that approximately two thirds (64%) of all lawyers were
146 GOODMAN-DELAHUNTY, GRANHAG, HARTWIG, AND LOFTUS

pleased or very pleased with the outcome of the case. It is interesting to note that
this number matched perfectly the estimated mean confidence in achieving the
minimal goal (64%), although lawyers achieved their goals in only 57% of those
cases. Fewer than one fifth of the lawyers (18%) were very disappointed or
somewhat disappointed with the case outcome, although more than twice as many
(43%) actually failed to achieve their stated minimum goal. Of the group of
lawyers who did not achieve their minimum goal (n⫽208), two thirds reported
that they were somewhat pleased or very pleased with the actual outcome.
Effects of the Debiasing Technique
We predicted that lawyers who generated reasons why they might not achieve
their litigation goals would be better calibrated. We expected to find that gener-
ating these reasons would reduce the overconfidence observed in lawyers making
relatively high estimates of success. To examine this hypothesis, we compared the
212 lawyers who were asked for reasons before they provided their confidence
estimate with the 269 lawyers who were not first asked for reasons. Most lawyers
(57%, n⫽274) provided one or more reasons why they might not achieve their
litigation goals. The nature of the reasons varied widely. Some pointed to
contextual factors, such as an opponent with extensive trial experience or legal
skill, a client who was particularly unattractive, or the emotional nature of the case
itself.
Of the group who provided reasons, 41.6% provided a single reason; others
listed two or more reasons. Accordingly, the first reason provided by lawyers was
coded and classified into one of six major categories. The largest category, “case
facts” (37.5%), included references to the weakness of the case, the emotional or
controversial nature of the case, and evidentiary problems. The next largest
category was “judge and jury issues” (24.8%) such as unpredictable fact finders.
The next distinct category was “witness factors” (14.2%) such as percipient or
expert witness problems. “Client factors” constituted a smaller group (12.5%) and
included references to unattractiveness and culpability. The next category, “court
context” (8%), comprised references to case precedent, jurisdiction, venue, court,
and geographic location. Surprisingly, the reasons least frequently cited were
Table 2
Mean Prediction, Objective Success, and Subjective Success, by
Party Represented (%)
Variable All Civil
plaintiff Civil
defense Criminal
prosecution Criminal
defense
Mean estimate 64.3 65.1 65.1 72.8 50.1
Objective success 57.0 51.6 62.6 67.1 43.7
Subjective success 66.2
a
60.0
b
70.3
c
77.2
d
60.1
e
N481 182 155 79 66
Note. Superscripts denote significant differences between objective and subjective suc-
cess (p⬍.001).
a
␹
2
(1, N⫽481) ⫽233.1.
b
␹
2
(1, N⫽182) ⫽104.0.
c
␹
2
(1, N⫽155) ⫽
62.7.
d
␹
2
(1, N⫽79) ⫽40.0.
e
␹
2
(1, N⫽65) ⫽23.9.
147LAWYERS’ ABILITY TO PREDICT CASE OUTCOMES

“attorney factors,” comprising only 3.2% of the reasons stated and referring to the
skill, preparedness, or experience of the participant or opposing counsel.
Notably, 18 lawyers in the reasons group declined to generate even a single
reason why they might not achieve their minimum goal. For purposes of analysis,
we included these lawyers in the reasons group. Perhaps they contemplated some
reasons in response to our prompting. Nonetheless, no support emerged for the
hypothesis that generating reasons in advance boosted calibration. A comparison
of confidence estimates in the reasons (M⫽64.3, SD ⫽19.6) and no-reasons
(M⫽64.3, SD ⫽21.1) groups revealed no significant differences, t(479) ⫽
⫺0.03, p⫽.98.
Figure 6 presents the calibration curves for the two groups of lawyers.
With one exception, the calibration curves for the reasons and no-reasons
lawyers are markedly similar. When initial confidence estimates exceeded
60%, lawyers in both the reasons (M⫽75.7, SD ⫽10.2) and no-reasons (M⫽
76.6, SD ⫽10.7) groups both exhibited overconfidence, t(140) ⫽⫺3.89, p⬍
.001; and t(177) ⫽⫺4.34, p⬍.001, respectively. Calibration differed for one
confidence interval; when initial confidence estimates were low from the
beginning (under 45%). In this range, lawyers in the no-reasons group (M⫽
26.4, SD ⫽13.2) displayed a great deal of underconfidence, t(37) ⫽3.53, p⬍
.001. In contrast, lawyers in the reasons group (M⫽27.8, SD ⫽12.2) were
well calibrated, t(26) ⫽0.98, p⫽.34.
Overall, generating reasons did not accomplish what we predicted, namely
that overconfidence would be reduced.
Figure 6. Calibration curves for the reasons group (n⫽212) and the no-reasons
group (n⫽269).
148 GOODMAN-DELAHUNTY, GRANHAG, HARTWIG, AND LOFTUS

Discussion
To our knowledge, this study of the ability of lawyers to predict the outcome
of their cases is the largest study of American legal practitioners of its kind
involving data drawn from actual cases. The findings extend previous research on
overconfidence in defense lawyers (Loftus & Wagenaar, 1988; Malsch, 1990), by
establishing that similar biases arise in predictions by criminal prosecutors and by
counsel for both plaintiffs and defendants in civil cases. Lawyers frequently made
substantial judgmental errors, showing a proclivity to overoptimism. The most
biased estimates were expressed with very high initial confidence: In these
instances, lawyers were extremely overconfident. These findings are consistent
with a large body of literature documenting overconfidence in a range of judg-
ments (theoretical explanations of miscalibration of confidence are discussed in
Gigerenzer, Hoffrage, & Kleinbolting, 1991; Kahneman, Slovic, & Tversky,
1982; Moore & Healy, 2008).
With respect to the correlates of the overconfidence bias, certain results were
somewhat counterintuitive, such as the finding that lawyers with more experience
were not better calibrated than less experienced lawyers. By contrast, expertise
predicted calibration in Dutch criminal lawyers whose specialization was mea-
sured by the number of cases handled per year in a certain field (Malsch, 1990).
Experience alone did not serve to calibrate either the Dutch or the North American
lawyers, although they plausibly received some form of contextual outcome
feedback that could have enhanced their calibration. Three potential explanations
to account for this finding are posited. First, more experienced lawyers may
handle cases of higher complexity and ambiguity than their less experienced
colleagues. Support for the notion that overconfidence is more extreme in tasks of
greater difficulty has emerged in several studies (Brenner, Koehler, Liberman, &
Tversky, 1996; Rehm & Gadenne, 1990). Second, a lawyer’s need for a highly
confident professional persona (Loftus & Wagenaar, 1988) may perpetuate a
tendency toward overconfidence over time rather than diminish it. Third, the
contextual effects of inquisitorial versus adversarial justice systems and the
differential feedback that lawyers receive over time may influence their predic-
tions.
Studies of forecasting in other professional domains showed that when the
ambiguity of the problem was reduced, forecasters relied on more objective and
fewer subjective factors in making their predictions (Braun & Yaniv, 1992).
Similarly, the finding that lawyers responsible for criminal cases involving vic-
timless crimes or crimes against property were better calibrated, compared with
lawyers responsible for cases involving crimes against persons, may be a function
of the relative ambiguity of the two types of cases.
With regard to gender, we replicated results obtained by Malsch (1990) that
female lawyers were better calibrated than their male colleagues. Male practitio-
ners were more overconfident than female practitioners. These findings are in line
with gender differences observed in research on metacognition (Pallier, 2003).
Calibration did not increase as a function of time to trial. However, lawyers
selected cases that they expected to go to trial within 6–12 months. The variability
in temporal distance to trial may have been insufficient to detect the predicted
influence of proximity to trial.
149LAWYERS’ ABILITY TO PREDICT CASE OUTCOMES

We applied two measures of success to determine the extent to which the
lawyers were well calibrated. One measure was the degree of correspondence
between the lawyers’ a priori goals and the reported actual case outcomes. This
was designated the objective measure of success. The subjective measure of
success was the lawyers’ self-report as to whether the a priori minimum goal was
achieved. The overconfidence effect disappeared and was replaced by undercon-
fidence when comparing lawyers’ a priori estimates to their subjective reports of
success. This is likely due to the fact that lawyers’ own perception of success after
the case was resolved was significantly higher than the objective measure of
success. The same self-serving bias (e.g., Babcock & Loewenstein, 1997) that
causes overconfidence in prospective judgments may also skew retrospective
judgments of success.
Improving Calibration
We failed to find support for the prediction that a cognitive manipulation
(asking participants to generate reasons against achieving their minimum goal)
was related to more realistic estimates. This might be due to the anchoring and
adjustment heuristic (Tversky & Kahneman, 1974): Lawyers choose a desirable
outcome, the anchor, and thereafter make insufficient adjustments for uncertainty
even when asked to generate reasons against their initial goal. Results in other
studies have also failed to show an influence on calibration of arguments gener-
ated against the chosen alternative (Allwood & Granhag, 1996) or a reduction of
unwarranted optimism in personal predictions by using pessimistic-scenario gen-
eration (Newby-Clark, Ross, Buehler, Koehler, & Griffin, 2000).
Lawyers are exposed to several additional factors that may influence the
predictive validity of the specific information they have regarding an upcoming
case. Compared with professional weather forecasters, lawyers are different in
two crucial respects. First, meteorologists cannot in any way influence the
outcome of their predictions. Nothing they do can make it rain. Lawyers, on the
other hand, can behave in ways that influence the case outcome. Because they
have this opportunity, they may overestimate their own capacity and neglect the
importance of factors beyond their control (Dollinger, 2008).
Second, lawyers have a much keener interest in the goals of their predictions than
do meteorologists. Because of this, lawyers might be susceptible to overoptimism and
wishful thinking. In several studies, researchers have demonstrated that merely
imagining an event can increase subjective probability estimates that the event will
occur (Carroll, 1978; Sherman, Cialdini, Schwartzman, & Reynolds, 1985). Thus,
even before the questions about confidence estimates and the reasons why lawyers
might not achieve their goals were posed, lawyers might have been biased in the
direction of overoptimism. This explanation is consistent with findings in previous
studies in which judgments about probability and likelihood were significantly
influenced by whatever side of the issue individuals thought about first (Hoch,
1984). Generation of a particular mental scenario may have hindered the later
generation of alternative, incompatible scenarios. People can enjoy the experience
of wishful thinking “as long as they are willing to pay the price of painful
disappointment when reality does not unfold as expected” (Babad, Hill, &
150 GOODMAN-DELAHUNTY, GRANHAG, HARTWIG, AND LOFTUS

Driscoll, 1992, p. 471). In the context of litigation, we agree with Griffin and
Tversky (1992):
It can be argued that people’s willingness to engage in military, legal, and other
costly battles would be reduced if they had a more realistic assessment of their
chances of success. We doubt that the benefits of overconfidence outweigh its
costs. (p. 433)
How can metacognitive realism in lawyers be improved? We failed to find
improved calibration as a function of a cognitive debiasing intervention. Perhaps
unrealistically optimistic predictions are too closely intertwined with the profes-
sional motivation of lawyers to be overcome by a cognitive intervention. These
lawyers may also be susceptible to a general human tendency to downplay
unfavorable aspects about oneself (Fiske & Taylor, 2008) and a tendency to
interpret the past in an unrealistically favorable light (Gilbert, 2006). A lawyer’s
commitment to the client and the case may induce a self-serving bias. A self-
serving bias is the expectation of one party that exceeds the beliefs of the
opposing party (Babcock, Loewenstein, & Issacharoff, 1997). Self-serving biases
in litigation were strong predictors of nonsettlement before proceeding to court
(Loewenstein, Issacharoff, Camerer, & Babcock, 1993). In previous research on
self-serving biases, encouraging participants to consider the weakness in their
case was effective in debiasing participants (e.g., Babcock & Loewenstein, 1997).
The debiasing manipulations in these studies were similar (generating reasons
why the goal might not be achieved versus thinking about the weaknesses in one’s
own case). The discrepancy between our finding and the previous finding may be
due to the fact that the participants in our study made judgments about their own
cases rather than simulated cases assigned to them in an experimental setting. A
more profound investment in the outcome in real-life cases may increase resis-
tance to debiasing interventions.
Policy Implications
The results of this study have practice and policy implications for lawyer–
client relationships, case management strategies, court efficiency, and lawyer
training and education. These policy issues can be addressed both formally and
informally.
Lawyer–client relations. Clients of lawyers who are susceptible to an
overconfidence bias may detrimentally rely on the advice of these legal profes-
sionals when making decisions about whether to litigate, they may choose
litigation over settlement, or they may allocate valuable resources without secur-
ing anticipated objectives. These clients are likely to experience disappointment,
frustration, or anger if the outcomes they were led to expect are not achieved.
Client dissatisfaction may increase public antipathy toward lawyers and diminish
public confidence in the legal system. Lawyers who have had several experiences
with dissatisfied clients are likely to adopt a risk-averse strategy in managing
client relationships to avoid a negative outcome by deliberately lowering all client
expectations so that subsequent disappointment is avoided. Lawyers who are
better calibrated can avoid dissatisfied clients by obviating the need to reduce
client expectations to preserve their relationship. Lawyer–client relationships,
151LAWYERS’ ABILITY TO PREDICT CASE OUTCOMES

client satisfaction, general public opinions of lawyers, and confidence in the legal
system may all be enhanced by well-calibrated lawyers. Clients are not well
served if their legal representatives cannot appropriately assess litigation risks or
if lawyers exercise poor judgment in advising them or deliberately lower all
client expectations.
Case management strategies. One implication of the present findings is
that lawyer performance can be improved by implementing case management
strategies that take into account the potential overconfidence biases of the litiga-
tors. Case consultations with legal peers can take place informally. For example,
in many legal firms, regular meetings are held where cases are periodically
reviewed so that the partners can manage the caseload efficiently and ethically.
These meetings provide ideal opportunities to obtain objective opinions from
other legal professionals in the form of third-party feedback about the strengths
and weaknesses of a case and the likelihood that the stated goals can be achieved.
Many of the most overconfident lawyers will be the senior partners who may not
typically obtain third-party review or feedback in the course of their practice. Law
firms should take affirmative steps to incorporate third-party feedback for the
more experienced or senior litigators in their case management systems. Legal
practitioners should also consider regularly obtaining customer feedback by
sending their clients anonymous survey questionnaires at the close of every case;
these should include questions that target the issues surrounding the management
of client expectations about the achievement of goals in a particular case.
In addition, lawyers should consider obtaining more formal external third-
party views on the likelihood of achieving their litigation goals on behalf of their
clients. It is in the early stages of litigation that patterns and expectations for the
case are established and where an intervention can have the most beneficial
effects. For example, in an evaluation of the Northern California Early Neutral
Evaluation (ENE) Program (Brazil, Kahn, Newman, & Gold, 1986), attorneys
reported changing their initial assessments of the merits of their case on the basis
of a better understanding of the case’s legal and factual issues gained during the
ENE. Furthermore, participants reported greater satisfaction with the outcome of
the case and perceptions of procedural fairness regardless of the eventual reso-
lution. However, these successful ENE programs have not been widely imple-
mented (Levine, 1987; Wissler, 2002). Alternatively, lawyers can seek the input
of an experienced third-party neutral, such as an alternative dispute resolution
practitioner, mediator, or arbitrator. These professionals are often consulted
jointly by the parties in mediation, but they can also be consulted independent of
the opposing party specifically to obtain evaluative feedback of this nature.
Court efficiency. Overconfidence in difficult or ambiguous cases could be
a barrier to alternative forms of dispute resolution; lawyers may pursue litigation
in cases even though the litigation objective cannot be achieved. This practice
may clog the courts with cases that lack merit and with unnecessary or protracted
litigation in cases that might be better suited to dismissal or settlement. To address
this issue, courts should consider requiring parties to appear before a third-party
neutral, such as a magistrate, judge, or mediator, for an assessment of the merits
of the case. A party who does not have an interest in a particular outcome might
be less prone to metacognitive biases that color the litigator’s decision making. In
many areas of law, trial may proceed only if mediation has been attempted and
152 GOODMAN-DELAHUNTY, GRANHAG, HARTWIG, AND LOFTUS

failed. The advantage of policies of this nature is that the mediation may provide
third-party neutral feedback to the lawyers that reduces overoptimism. Thus, these
policies should be further supported. Other policies in the form of interventions
that may be effective in inhibiting unnecessary litigation should be considered
(e.g., more early neutral evaluation options and expanded use of pretrial confer-
ences in criminal as well as civil cases).
Lawyer training and education. Finally, the findings that many lawyers
are susceptible to an overconfidence bias have implications for legal training and
education. Information about the outcomes of this and similar studies should be
included in the relevant curricula in clinical practice courses and in professional
and ethical responsibility classes both at law school and bar admission. Better
informed lawyers will be more likely to seek third-party feedback to avoid
subjective overconfidence.
The key in all policy recommendations is the provision of feedback to lawyers
about the link between their professional judgment and their performance. Out-
come feedback is central to developing well-calibrated judgments (Hogarth et al.,
1991; Keren, 1987; Murphy & Winkler, 1971; Stone & Opel, 2000). To enhance
calibration precision, one needs to determine the accuracy of previous predictions.
Lawyers should be encouraged to monitor their predictions in multiple ways. Case
outcomes in the form of judicial opinions or jury verdicts are dependent on many
factors outside the control of the lawyer and provide only limited information
(win/lose); these are, thus, very blunt instruments to rely on for enhancing
calibration. More sensitive, fine-grained feedback may be available from oppos-
ing counsel, judges, or an analysis of audio-visual recordings. Several avenues to
help lawyers better calibrate their metacognitive judgments are advised. Policies
requiring lawyers to consult external third parties to obtain more realistic evalu-
ations of cases and their likely outcomes may benefit clients, lawyers, the justice
system, and the public interest.
Conclusion
Future research should examine factors that enhance lawyer calibration, such
as case-related aspects and performance feedback. If case ambiguity and case
complexity are controlled, a lawyer’s experience may prove to be a better
predictor of calibration. To date, no research has thoroughly examined this issue.
Similarly, the extent to which input from an external party improves lawyers’
realism in making case predictions and achieving their objectives needs further
investigation. However, the literature on motivated social cognition implies that
the motivation to achieve a certain case outcome may lead lawyers to discount the
assessment of the external party if it does not fit with their preferred belief
(Kunda, 1990).
This study extended previous research on the judgment, decision making, and
metacognitive realism of legal professionals. Our large sample of U.S. lawyers
showed clear evidence of unrealistic litigation goals, and our experimental ma-
nipulation with the purpose of debiasing optimistic tendencies did not have the
desired effect. Because accurate legal decision making might be in jeopardy as a
function of lawyers’ optimistic tendencies, further research ought to explore ways
to enhance calibration to reach more realistic metacognitive judgments.
153LAWYERS’ ABILITY TO PREDICT CASE OUTCOMES

References
Allwood, C. M., & Granhag, P. A. (1996). The effects of arguments on realism in
confidence judgments. Acta Psychologica, 91, 99–119.
Allwood, C. M., & Granhag, P. A. (1999). Feelings of confidence and the realism of
confidence judgments in everyday life. In P. Juslin & H. Montgomery (Eds.),
Judgment and decision making: Neo-Brunswikian and process-tracing approaches
(pp. 123–146). Mahwah, NJ: Erlbaum.
Babad, E., Hills, M., & O’Driscoll, M. (1992). Factors influencing wishful thinking and
predictions of election outcomes. Basic and Applied Social Psychology, 13, 461–476.
Babcock, L., Loewenstein, G., & Issacharoff, S. (1997). Creating convergence: Debiasing
biased litigants. Law & Social Inquiry, 22, 913–925.
Babcock, L., & Loewenstein, G. (1997). Explaining bargaining impasse: The role of
self-serving biases. Journal of Economic Perspectives, 11, 109–126.
Braun, P. A., & Yaniv, L. (1992). A case study of expert judgment: Economists’
probabilities versus base-rate model forecasts. Journal of Behavioral Decision Mak-
ing, 5, 217–231.
Brazil, W. D. (1985, December). What lawyers want from judges in the settlement arena.
Federal Rules, 85–101.
Brazil, W. D., Kahn, M. A., Newman, J. P., & Gold, J. Z. (1986). Early neutral evaluation:
An experimental effort to expedite dispute resolution, Judicature, 69, 279–285.
Brehm, J. W. (1956). Post-decision changes in desirability of alternatives. Journal of
Abnormal and Social Psychology, 52, 348–359.
Brenner, L. A., Koehler, D. J., Liberman, V., & Tversky, A. (1996). Overconfidence in
probability and frequency judgments: A critical examination. Organizational Behav-
ior and Human Decision Processes, 65, 212–219.
Carroll, J. S. (1978). The effect of imagining an event on expectations for the event: An
interpretation in terms of the availability heuristic. Journal of Experimental Psychol-
ogy, 14, 88–96.
Dollinger, S. (2008). The illusion of control. In L. T. Benjamin, Jr. (Ed.), Favorite
activities for the teaching of psychology (pp. 201–202). Washington, DC: American
Psychological Association.
Dougherty, M. R. P. (2001). Integration of the ecological and error models of overcon-
fidence using a multiple trace memory model. Journal of Experimental Psychology:
General, 130, 579–599.
Dunning, D., Griffin, D. W., Milojkovic, J. D., & Ross, L. (1990). The overconfidence
effect in social predictions. Journal of Personality and Social Psychology, 58,
568–581.
English, P., & Sales, B. (2005). More than the law: Behavioral and social facts in legal
decision making. Washington, DC: American Psychological Association.
Fiske, S. T., & Taylor, S. E. (2008). Social cognition, from brains to culture. Columbus,
OH: McGraw-Hill Higher Education.
Flavell, J. (2004). Theory-of-mind development: Retrospect and prospect. Merrill-Palmer
Quarterly, 50, 274–290.
Gigerenzer, G., Hoffrage, U., & Kleinbolting, H. (1991). Probabilistic mental models: A
Brunswikian theory of confidence. Psychological Review, 98, 506–528.
Gilbert, D. (2006). Stumbling on happiness. New York: Alfred A. Knopf.
Gilovich, T., Kerr, M., & Husted Medvec, V. (1993). Effect of temporal perspective on
subjective confidence. Journal of Personality and Social Psychology, 4, 552–560.
Gollwitzer, P., & Schaal, B. (1998). Metacognition in action: The importance of imple-
mentation intentions. Personality and Social Psychology Review, 2, 124–136.
154 GOODMAN-DELAHUNTY, GRANHAG, HARTWIG, AND LOFTUS

Gonza´lez-Vallejo, C., & Bonham, A. (2007). Aligning confidence with accuracy: Revis-
iting the role of feedback. Acta Psychologica, 125, 221–239.
Griffin, D., & Tversky, A. (1992). The weighing of evidence and the determinants of
confidence. Cognitive Psychology, 24, 411–435.
Harley, E. (2007). Hindsight bias in legal decision making. Social Cognition, 25, 48–63.
Hoch, J. S. (1984). Availability and interference in predictive judgment. Journal of
Experimental Psychology: Learning, Memory, & Cognition, 10, 649–662.
Hogarth, R. M. (2001). Educating intuition. Chicago: The University of Chicago Press.
Hogarth, R. M., Gibbs, B., McKenzie, C., & Marquis, M. (1991). Learning from feedback:
Exactingness and incentives. Journal of Experimental Psychology: Learning, Mem-
ory, and Cognition, 17, 734–752.
Hogarth, R. M., & Makridakis, S. (1981). Forecasting and planning: An evaluation.
Management Science, 27, 115–138.
Horowitz, I. A., & Willging, T. E. (1984). The psychology of law: Integrations and
applications. Boston: Little, Brown.
Jay, S. (1989). The dilemmas of attorney contingent fees. The Georgetown Journal of
Legal Ethics, 2, 813–884.
Kahneman, D. (1973). On the psychology of prediction. Psychological Review, 80,
237–251.
Kahneman, D., Slovic, P., & Tversky, A. (1982). Judgments under uncertainty: Heuristics
and biases. Cambridge, United Kingdom: Cambridge University Press.
Keren, G. (1987). Facing uncertainty in the game of bridge: A calibration study. Orga-
nizational Behaviour and Human Decision Processes, 39, 98–114.
Keren, G., & Raaijmakers, J. G. W. (1988). On between-subjects versus within-subjects
comparisons in testing utility theory. Organizational Behavior and Human Decision
Processes, 41, 233–247.
Koriat, A. (2002). Metacognition research: An interim report. Applied metacognition (pp.
261–286). New York: Cambridge University Press.
Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980). Reasons for confidence. Journal of
Experimental Psychology: Human Learning and Memory, 6, 107–118.
Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108, 480–
498.
Langens, T. (2007). Regulatory focus and illusions of control. Personality and Social-
Psychology Bulletin, 33, 226–237.
Langer, E. J. (1975). The illusion of control. Journal of Personality and Social psychol-
ogy, 32, 311–328.
Levine, D. (1987). Early neutral evaluation: A follow-up report. Judicature, 70, 236–240.
Lichtenstein, S., & Fischhoff, B. (1977). Do those who know more also know more about
how much they know? Organizational Behavior and Human Performance, 20,
159–183.
Loewenstein, G., Issacharoff, A., Camerer, C., & Babcock, L. (1993). Self-serving
assessments of fairness and pretrial bargaining. The Journal of Legal Studies, 22,
135–159.
Loftus, E. F., & Wagenaar, W. A. (1988). Lawyers’ predictions of success. Jurimetrics
Journal, 28, 437–453.
Lord, C. G., Lepper, M. R., & Preston, E. (1984). Considering the opposite: A corrective
strategy for social judgment. Journal and Personality and Social Psychology, 47,
1231–1243.
Malsch, M. (1989). Voorspellingen van rechterlijke beslissingen door advocaten. Neder-
landse samenvatting van “Lawyers’ predictions of judicial decisions: A study on
calibration of experts.” Unpublished doctoral dissertation, Rijksuniversiteit Leiden,
Leiden, the Netherlands.
155LAWYERS’ ABILITY TO PREDICT CASE OUTCOMES

Malsch, M. (1990). Advocaten voorspellen de uitkomst van hun zaken [Lawyers’ predic-
tions of judicial decisions: A study on calibration of experts]. Lisse, the Netherlands:
Swets & Zeitlinger.
Meeks, J., Hicks, J., & Marsh, R. (2007). Metacognitive awareness of event-based
prospective memory. Consciousness and Cognition: An International Journal, 16,
997–1004.
Milburn, M. A. (1978). Sources of bias in the prediction of future events. Organizational
Behavior and Human Performance, 21, 17–26.
Moore, D., & Healy, P. (2008). The trouble with overconfidence. Psychological Review,
115, 502–517.
Moore, D. A., & Cain, D. M. (2007). Overconfidence and unconfidence: When and why
people underestimate (and overestimate) the competition. Organizational Behavior
and Human Decision Processes, 103, 197–213.
Murphy, A. H., & Winkler, R. L. (1971). Forecasters and probability forecasts: Some
current problems. Bulletin of American Meteorological Society, 52, 239–247.
Newby-Clark, I. R., Ross, M., Buehler, R., Koehler, D. J., & Griffin, D. (2000). People
focus on optimistic scenarios and disregard pessimistic scenarios while predicting
task completion times. Journal of Experimental Psychology: Applied, 3, 171–182.
Paese, P. W., & Feuer, M. A. (1991). Decisions, actions, and the appropriateness of
confidence in knowledge. Journal of Behavioral Decision Making, 4, 1–16.
Pallier, G. (2003). Gender differences in the self-assessment of accuracy on cognitive
tasks. Sex Roles, 48, 265–276.
Perfect, T., & Schwartz, B. (2002). Applied metacognition. New York: Cambridge
University Press.
Rehm, J. T., & Gadenne, V. (1990). Intuitive predictions and professional forecasts:
Cognitive processes and social consequences. Oxford, England: Pergamon Press.
Ross, H. L. (1970). Settled out of court. Chicago: Aldine.
Schuck, P. H. (1986). The role of judges in settling complex cases: The Agent Orange
example. The University of Chicago Law Review, 53, 337–365.
Sherman, J. S., Cialdini, R. B., Schwartzman, D. F., & Reynolds, K. D. (1985). Imagining
an event can heighten or lower the perceived likelihood of contracting a disease: The
mediating effect of ease of imagery. Personality and Social Psychology Bulletin, 11,
118–127.
Simon, W. H. (1988). Ethical discretion in lawyering. Harvard Law Review, 101,
1083–1145.
Stone, E. R., & Opel, R. B. (2000). Training to improve calibration and discrimination:
The effects of performance and environmental feedback. Organizational Behavior
and Human Decision Processes, 83(2), 282–309.
Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases.
Science, 185, 1124–1131.
Vallone, R. P., Griffin, D. W., Lin, S., & Ross, L. (1990). Overconfident predictions of
future actions and outcomes by self and others. Journal of Personality and Social
Psychology, 58, 582–592.
Von Winterfeldt, D., & Edwards, W. (1986). Egocentrism as a source of unrealistic
optimism. Journal of Personality and Social Psychology, 53, 421–430.
Vreugdenhil, H. (1993). Confidence in knowledge: Studies on the calibration of subjective
probabilities. Unpublished doctoral dissertation, University of Amsterdam, Amster-
dam, the Netherlands.
Weinstein, N. D. (1980). Unrealistic optimism about future life events. Journal of
Personality and Social Psychology, 39, 806–820.
Weinstein, N. D. (1982). Unrealistic optimism about susceptibility for health problems.
Journal of Behavioral Medicine, 5, 441–460.
156 GOODMAN-DELAHUNTY, GRANHAG, HARTWIG, AND LOFTUS

Weinstein, N. D. (1983). Reducing unrealistic optimism about illness susceptibility.
Health Psychology, 2, 11–20.
Wissler, R. L. (2002). When does familiarity breed content? A study of the role of
different forms of ADR education and experience in attorneys’ ADR recommenda-
tions. Pepperdine Dispute Resolution Law Journal, 2, 199–217.
Wittman, D. (1988). Dispute resolution, bargaining, and the selection of cases for trial: A
study of biased and unbiased data. The Journal of Legal Studies, XVII, 313–352.
Received September 14, 2009
Revision received November 23, 2009
Accepted November 30, 2009 䡲
y
157LAWYERS’ ABILITY TO PREDICT CASE OUTCOMES