Improving project outcomes through operational reliability: A conceptual model

Abstract

The work reported here is part of an on-going research effort concerned with developing a methodology to improve the outcomes of projects involving complex information systems. To facilitate the study, previous research concerned with high reliability organisations (HRO) is examined. An analysis of this literature is used to develop a conceptual model that explains how highly reliable operational capabilities are achieved and maintained. Cases of catastrophic failures are evaluated using this paradigm to illustrate its utility as an analytical research tool and its potential as a systemic framework for evaluating the operational capability of organisations prior to project initiation.

Full text

Page 1

Improving project outcomes through operational reliability:
A conceptual model
John Sullivana, Roger Beachb,*
aSchool of Business, Charleston Southern University, 9200 University Boulevard, Charleston, SC 29406, USA
bUniversity of Bradford, School of Management, Emm Lane, Bradford, West Yorkshire, BD9 4JL, UK
Received 5 November 2008; received in revised form 6 February 2009; accepted 12 February 2009
Abstract
The work reported here is part of an on-going research effort concerned with developing a methodology to improve the outcomes of
projects involving complex information systems. To facilitate the study, previous research concerned with high reliability organisations
(HRO) is examined. An analysis of this literature is used to develop a conceptual model that explains how highly reliable operational
capabilities are achieved and maintained. Cases of catastrophic failures are evaluated using this paradigm to illustrate its utility as an
analytical research tool and its potential as a systemic framework for evaluating the operational capability of organisations prior to pro-
ject initiation.
? 2009 Elsevier Ltd and IPMA. All rights reserved.
Keywords: Managing projects; Competences; Operations; System reliability
1. Introduction
Management information system (IS) projects have been
studied for many years, and numerous recommendations
have been made to improve performance. Success factors
[1,2], failure factors [3,4], models [5] and frameworks [6]
have been developed but with little beneficial impact [7].
Moreover, the difficulties appear to intensify with system
complexity [8,9]. ERP systems are highly integrated, com-
plex, cross-functional systems, which support business pro-
cesses across an entire organisation. Typically, these
projects involve extended periods of implementation and
require significant resources, management commitment
and sound project management to succeed. Yet despite this
understanding, troubled implementations and unmet oper-
ational expectations are reported in more than half of all
ERP systems [10].
Some organisations, however, are very successful in
implementing and operating complex systems, e.g., nuclear
power stations, chemical processors, the military, medical
care providers, etc. [11]. These, so called, high reliability
organisations (HRO) operate in environments where the
potential for disaster is high, the pressure to meet expecta-
tions is persistent and the consequences for failure are pub-
licly visible. That these organisations differ structurally
from other organisations in some critical way is apparent.
Adopting the contingency perspective to organisational
design and viewing structures as products of internal and
external forces (power, technology, the environment, etc.)
explains these apparent differences [12] and provides a
point of departure for this research; that projects can also
be considered as ‘‘particular kinds of organisations” (ibid)
reinforces our view that these ‘project-based organisations’
might benefit from an improved understanding of how high
operational reliability is achieved.
In this paper, the first phase in a multi-phase research
design concerned with improving the ability of organisa-
tions to implement and manage complex systems is
presented. The aims of this phase were to: (a) provide the
theoretical foundations for researching the factors that con-
tribute to operational reliability in IS project environments;
0263-7863/$36.00 ? 2009 Elsevier Ltd and IPMA. All rights reserved.
doi:10.1016/j.ijproman.2009.02.006
*Corresponding author. Tel.: +44 1274 234310; fax: +44 1274 236786.
E-mail address: r.beach@bradford.ac.uk (R. Beach).
www.elsevier.com/locate/ijproman
Available online at www.sciencedirect.com
International Journal of Project Management 27 (2009) 765–775

Page 2

(b) develop a framework to facilitate a comparative analysis
of organisational activities and the synthesis of empirical
data gathered in subsequent stages of the research, and (c)
extend knowledge of those organisational characteristics
that contribute to successful project outcomes/reliable
operational performance.
In the following sections a synthesis of previous HRO
research and the challenges of managing projects and oper-
ating complex systems are presented. To the best of our
knowledge very little work has been done in this area in
recent years and in particular within the context of pro-
jects. A conceptual model derived from this analysis is
developed. Two published cases of catastrophic failures
are evaluated to illustrate the construct’s utility as a means
of explaining how highly reliable organisations function
and to demonstrate its suitability as a framework for ana-
lysing qualitative empirical data in IS project research. The
paper concludes by proposing that the model has practical
value as a capability building tool.
2. High reliability organisations
Some organisations have been very successful in the
implementation of complex systems [11]. The HRO is capa-
ble of developing and/or applying sophisticated technolo-
gies with high degrees of success, sometimes in the face
of apparently impossible odds. They develop techniques
and systems to manage high risk situations. This is not to
suggest that failures do not occur, the HRO can experience
failure, sometimes catastrophically. But as we shall see, it is
the way that these organisations deal with failure and
recovery that is particularly significant.
Some of the early studies on the activities of the HRO
suggested that the tight coupling of tasks and system com-
plexity create an environment where disasters are inevitable
[8]. However, this view has not been universally accepted.
Roberts and Bea [11], whose work is based on Perrow’s,
assert that high reliability is possible and that organisations
possess attributes that can enhance reliability. Specifically,
they aver that the HRO will aggressively seek to know
what they do not know, design reward systems to recognize
the costs of failure as well as the benefits of reliability and
consistently communicate the organisational objectives and
how the individual contributes to the achievement of these.
Consider the case of a US Navy nuclear submarine. In this
environment the reliable performance of each individual
and the performance of the whole can be a matter of life
and death, even in routine situations [13]. The commanding
officer (CO) has absolute authority and responsibility for
everything that occurs and sanctions imposed by the CO
carry a high degree of visibility and notices of disciplinary
action are displayed publicly (ibid). Visibility and account-
ability are methods used to effectively communicate that
one crew member’s poor performance is a reflection of
the entire crew’s performance. In this way each member
of the crew is made aware of their contribution to the
whole and the organisation’s objectives.
The HRO also differs from other organisations in that it
places a higher priority on reliability than any other objec-
tive,includingprofit[14].Itplacesahighpriorityoneffective
performance and avoiding disasters through processes of
collective learning, on developing a culture of reliability
andoneffectivecommunication(ibid).Itmaintainsaculture
that is able to overcome resistance to change as part of the
learning process and represents a ‘‘body of shared knowl-
edge built up through learning” [13]. Indeed, Bierly and
Spender [13] have suggested that it is this characteristic that
was absent in Perrow’s earlier considerations of the HRO.
Rochlin [9], LaPorte [15], Roe et al. [16] and others have
similarly contributed to the characterization of these
organisations, noting that they exhibit: high technical com-
petence, high performance and oversight, a propensity to
constantly search for improvements and a willingness to
embrace highly complex activities; they operate under high
pressure, use incentives and share expectations to enhance
reliability; they maintain hazard-driven operational flexi-
bility to ensure safety; they promote a culture of reliability
and, avoid cost effective trade-offs and trial and error learn-
ing [16].
In the absence of a defining set of characteristics, an
assertion that any organisation is an HRO is open to some
interpretation. To address this issue, Roberts [17] proposed
using a two dimensional matrix. The first dimension, tech-
nological risk represented the affect the level of advanced
technology has on the reliability of a system. The second
dimension, reliability, represented an organisation’s ability
to consistently meet its objectives. Using these dimensions
organisations were categorized into four quadrants. For
example, a television weather reporter might be plotted in
quadrant 1, low reliability and low technological risk;
whilst it may prove difficult to accurately predict the
weather, there is little technological risk associated with
generating the forecasts. A metropolitan water supplier
might be classified into quadrant 3, high reliability and
low technological risk; water delivery involves the applica-
tion of proven technology and failure is often quickly rec-
tified with minimal risk to service. Similarly an HRO such
as the United States Military could be plotted at the right
side of quadrant 3 and in quadrant 4 because, although
rare, system failure typically results in highly consequential
outcomes. Conversely, the Chernobyl and Three Mile
Island nuclear power plants during their respective crises
could be positioned in quadrant 2, low reliability and high
technological risk, i.e. in immediate danger and likely to
experience disaster [17].
There are a number of limitations apparent with this
framework. It cannot accommodate the full gamut of
organisations or the components that contribute to reliabil-
ity or technological risk. For example, it is reasonable to
assume that profit-seeking organisations might place a dif-
ferent emphasis on reliability compared to non-profit
organisations, whilst service organisations (high consumer
contact) may avoid technological risk in preference for pro-
ven technologies compared to manufacturers. Addition-
766
J. Sullivan, R. Beach/International Journal of Project Management 27 (2009) 765–775

Page 3

ally, it does not reflect the consequences an organisation
might face for being unreliable and how this will impact
on future activities or the level of accountability to which
it may be held. A distinction must also be made between
capability and reliability [18,19]. Capability is a potential
measure of performance, whilst reliability refers to how
effectively that capability is applied and how repeatable
the outcomes are. Whilst NASA has unquestionably
advanced capability, it has experienced periods of unreli-
ability. Hence, whilst NASA would normally be plotted
in quadrant 4 (high, high), at other times quadrant 2 (high,
low) is more appropriate. In other words, the framework
does not recognize periods of unreliability or recovery.
There are also risks other than technology, e.g. financial
and market related risks, suggesting that a more general
interpretation of this dimension is required. The following
sections develop these areas further.
3. Risk
The primary risks faced by the HRO are those that may
result in significant property damage or loss of life [20–
22,16] and those associated with failure to meet the expec-
tations of stakeholders for safe and effective performance
[23–25]. We refer to these as operational and stakeholder
risks.
3.1. Operational risk factors
Grabowski and Roberts [26] have suggested that HRO
risk factors involve ‘‘high consequence.” In extreme cases,
they include uncertain or hazardous operating environ-
ments, ‘‘a situation that invokes a sense of dread” [8]. Risk
factors exist in other forms such as introducing new tech-
nology [27] where failure to properly manage the technol-
ogy might threaten the value of the company stock [28]
or drain resources from more profitable ventures [29]. Risk
factors can include human activities such as false assump-
tions about reality or the effects of political activity
[18,30]. Risk factors can arise as effects of a ‘‘disaster incu-
bation period”, when small anomalies evolve into signifi-
cant problems [18]. Additionally, when members of a
group become overly concerned with reaching a consensus
(a condition known as ‘group think’) issues with potentially
important consequences can be overlooked; a condition
more likely to occur when the group is constrained by time
[31].
Organisations such as the military, power utilities and
chemical plants appear to operate with a heavy risk compo-
nent; an omni-present awareness of the consequences of
failure [20,32–35] whilst manufacturers of electronic com-
ponents may not. Risk factors can be categorized into three
groups: known risk (the organisation has experience of a
particular factor, e.g. a plane crash, chemical spill); imag-
ined risk (a risk that has not materialized but has been iden-
tified as a potential future hazard, e.g. groups of senior
executives flying on the same plane), and; unknown risk
(risks that emerge in a unique and unaccounted for situa-
tion, e.g. the loss of hydraulic power aboard United Airlines
Flight 232 in 1989). The existence of these risk factors and
their potential influence on system reliability is likely to be
more obvious in some organisations than in others.
3.2. Stakeholder expectations
An organisation’s stakeholders include its managers,
employees, regulatory and political entities and, increas-
ingly, the general public and consumers. Stakeholder influ-
ence on project outcomes is well known and strategies to
influence the behavioural characteristics of individuals to
mitigate the effects in ‘‘sensitive projects” have been pre-
sented [36]. However, a ‘‘very low error rate has become
the norm of social expectation; no measurable increase in
accidents or other failures is likely to be accepted, even if
operational levels are markedly increased” [37]. Failure to
meet these expectations
consequences.
The investigations of the Presidential Commission [38]
and the US House of Representatives, Committee on Sci-
ence and Technology [39] into the causes of the space shut-
tle challenger disaster echoed public sentiments that the
shuttle must fly reliably – all the time. The loss of the chal-
lenger could have, and should have, been prevented [40,13].
The HRO also requires a ‘‘high level of goal congruence
and a shared sense of duty to the collective purpose” [41].
Aboard US Navy aircraft carriers, each individual is
expected to perform assigned tasks reliably as well as mon-
itoring the larger picture [20,13,11]. In these organisations,
the expectations of individuals are so high that an organi-
sational culture develops surrounding those expectations
[17,13,25]. This is in stark contrast to the non-HRO where
failure to meet expectations rarely invokes the same degree
of anxiety from stakeholders. Therefore, we conclude that
the consequences of failure are related to stakeholder
expectations in that failure to meet minor expectations will
result in minimal consequences whilst major failures will
result in more severe consequences. Sometimes, even the
threat of such a calamity is enough to jeopardize public
confidence, and erode public support. In 1979, the Three
Mile Island nuclear power plant in Pennsylvania nearly
experienced a meltdown of the reactor core [42]. In situa-
tions of the highest risk, public fears of what could have
happened were enough to invoke consequences for the
organisation, even though disaster was avoided.
can resultin significant
4. Capability
Capability can be defined in terms of resources and com-
petences;thetoolsnecessarytoaccomplishcertaintasks and
the ability to use those tools effectively. All organisations
requireresourcestofunction,i.e.,time,money,people,tech-
nology, and information. However, the HRO places a
higher priority on the adequacy of these resources [22,24].
Competencesinclude proprietaryprocessknowledge,
J. Sullivan, R. Beach/International Journal of Project Management 27 (2009) 765–775
767

Page 4

unique skills and experiences. The HRO must, by necessity,
have significant reserves of these competences to adequately
address the hazards of their environment [16,43].
4.1. Resources
The HRO considers reliability to be equal or greater in
importanceto efficiency
resources is a prerequisite of reliability. Grabowski and
Roberts [22] suggest that over time large scale systems
can become degraded, ‘‘disordered and disorganized” and
that ‘‘preventing this entropy requires careful system plan-
ning and maintenance, as well as significant resources.”
Thus, carrying higher operational costs than conventional
organisations appears to be a price that must be paid for
exceptional system reliability [24]. A point supported by
Perrow [8] who earlier had noted that drives toward effi-
ciency (cost reduction or increased profitability) increased
the danger in complex systems. Roberts [17] considered this
to be especially true in large scale HRO environments
where resource requirements can be significant. The HRO
also has a heightened awareness of investments in safety
[43,44]. Many organisations understand that ‘‘expenditure
on safety is a good investment” [45] but the HRO is char-
acterized by an operational ethos in which reliability rivals
efficiency; that the cost of an accident far outweighs the
cost of preventive measures [43].
and maintainingadequate
4.2. Competence factors
Competences include the skills, procedures, knowledge
and experience of an organisation’s members. Thus the
ability to accomplish a specified objective is heavily reliant
on an organisation’s people. These factors are difficult to
quantify and tend to fluctuate over time as an organisa-
tion’s membership changes. In times of crisis, exceptional
competence is often credited with averting disaster or the
lack of it as the cause of one [11].
To attain ‘high reliability’ high levels of competence are
necessary but even organisations with plentiful resources
will occasionally experience system failure. Further, organ-
isations heading for failure cannot be turned around by
throwing money at the problem. In essence, it is the com-
petence factors that prevent disaster, not resources alone.
Moreover, how these competences are nurtured and
deployed within the organisation will impact on the effec-
tiveness of the system. The matrix structure, frequently
associated with project-based organisations, can be partic-
ularly useful in overcoming problems that are inter-func-
tional in nature [46]. They also provide competence and
confidence building opportunities for individuals that can
lead to the creation of sapiential authority (ibid). Sapiential
authority, i.e. the authority invested in someone by virtue
of their knowledge and expertise or maturity (life knowl-
edge) rather than hierarchical position (structural or posi-
tional authority), is more likely to occur in rapidly
changing knowledge based environments where knowledge
is embedded in the individual [47]. Sapiential authority
within team structures is said to be particularly useful when
events become critical (ibid).
As previously mentioned, there is some disagreement as
to whether all failures can be completely prevented in com-
plex systems [8,48]. Perrow’s [8] ‘Normal Accident Theory’
proposes that accidents in complex technological systems
must be expected, that eventually enough circumstances
will align themselves and failure will follow. Whilst, others
have averred that achieving the failure free operation of a
high risk system is idealistic and can lead managers to
develop unrealistic expectations of the system [48] with
the result that ‘‘the naı ¨ve, perhaps wistful or short-sighted,
views of decision-makers who, due to institutional pres-
sure, desperation or arrogance, are tempted to make unre-
alistic assumptions about the systems they direct” [49].
The above withstanding, there appears to be a consensus
that competent organisational practices can improve reli-
ability and reduce the effects of system failure [14,17,50–
52,43]. For example, in 1989 United Airlines Flight 232
from Denver to Chicago suffered an explosion in one of
the aircraft’s engines causing a loss of hydraulic power.
The manufacturer (McDonnell-Douglas) had not antici-
pated the possibility of such a failure and as a consequence
advice on how to proceed was not available. The pilot, a
passenger (also a pilot), the second officer and engineer
working with ground personnel eventually crash landed
the plane. Hundred and eleven passengers died but 185 sur-
vived [43], demonstrating that high performance is not a
product of luck but a characteristic of the industry [53].
5. Capability to mitigate risk
Whilst some organisations must carry excess capability
to mitigate exceptional operational risks, commensurately
lower capability is required by those whose perceived risk
and consequences of failure are lower. There are, however,
organisations whose capabilities are not sufficient to miti-
gate the risk they face. The Chernobyl nuclear power plant
incident demonstrated that the organisation did not have
the capabilities to mitigate the risk involved with operating
a nuclear power plant [33]. In other words, an imbalance
between risk and capability causes systems to fail.
As risk increases, the organisation’s ability to mitigate
that risk will be continually tested [34]. Roberts and Rous-
seau [23] have averred that for the HRO, ‘‘substandard per-
formance or deviations from standard procedures meet
with severe and adverse consequences.” Systems such as
air traffic control will incur high social and political conse-
quences for failure [37]. Thus, the HRO is highly motivated
to prevent system failures. ‘‘Attention is paid to minor inci-
dents not only because of financial or safety consequences,
but because of the potential consequences if the causes of
small incidents combine (‘when the holes line up’) to enable
a major problem” [34]. Roberts [17] observed that in cases
of catastrophic system failure, such as the Union Carbide
plant at Bhopal or the Challenger disaster, the public
768
J. Sullivan, R. Beach/International Journal of Project Management 27 (2009) 765–775

Page 5

demand change and that the HRO must ‘‘avoid errors or
failure because the potential cost is unacceptable to soci-
ety” [17]. In other words, an awareness of the consequence
of failure can influence overall system performance.
6. The conceptual model
Organisations that reliably manage complex systems
maintain a balance between those factors that represent a
threat to the system and those that mitigate them. The
HRO is able to consistently achieve this balance over long
periods of time. Considering the inherent risk their operat-
ing environment may pose this balance can be delicate. A
conceptual model of the elemental parts and governing
dynamics that appear to facilitate high operational reliabil-
ity is shown in Fig. 1.
In this construct operational reliability is portrayed as a
scale held in equilibrium by five interacting forces: expecta-
tions, risk factors, resources, competence factors, and con-
sequences. If the weight of capability is sufficient to
counteract the weight of risk, the scale remains in balance,
and the organisation’s systems continue to operate reliably.
However, if the risks exceed the organisation’s capabilities,
the scale will become unbalanced and a system failure
result. Thus, the dynamic relationships are explained in
the following sections.
6.1. Resource and expectations
As previously noted, adequate resources are essential to
support complex systems in the HRO [8,17,22,24]. For
organisations with sufficient resources, expectations will
be high. This may be especially true of safety [40,13]. Con-
versely, cutting back resources and emphasising efficiency
is detrimental to reliability [8,17]. Therefore, to maintain
system equilibrium a reduction in resources must be accom-
panied by a reduction in performance related expectations,
whilst an increase in resources is usually accompanied by an
increase in expectations. The bidirectional nature of this
relationship is represented in the model as dotted lines
whilst the variable relative strength (or weakness) of these
components is represented as triangles (Fig. 2).
6.2. Expectations and consequences
Naturally, those responsible for the reliability of the sys-
tem object when resources are reduced. In situations where
external political influences shape the behaviour of the
operating environment objections can include public state-
ments of impending disaster and system failure [37]. Fears
of the consequences of not meeting expectations are clearly
strong motivating factors in these situations. A relationship
can, therefore, be seen to exist between expectations and
consequences. However, not all expectations are equal.
Delaying the launch of the space shuttle may incur addi-
tional costs and the system will fall short of expectations
but the expectation that the operation should be safe is
much higher and failure in this regard has greater conse-
quences. Therefore, as the weight of expectations increase
the consequences for not meeting those expectations will
also increase. Similar to the relationship between expecta-
tions and resources a triangular representation is used to
illustrate the variable relative strength (or weakness) of
these components of the model.
6.3. Consequences and competence factors
A relationship can also be detected between conse-
quences and competence factors. When an imbalance
occurs, the consequences for the system’s failure will result
in initiatives to develop mitigating competences. The
organisational learning processes that the HRO will use
in these instances will often lead to new procedures,
improved safety measures and technological developments.
Thus, an HRO will possess highly developed learning
mechanisms that enable them to learn from mistakes and
avoid repetitions. However, this relationship appears unidi-
rectional, since competence development is unlikely to
affect consequences.
A defining characteristic of the HRO is this persistent
desire to learn as an organisation. In cases where organisations
Resources
Risk Factors Competence
Factors
Reliability
Expectations
Risk
Capability
Increasing
severity
Increasing
level of
investment
Increasing
weight of
expectations
Bidirectional relationship
Consequences
Unidirectional relationship
Fig. 1. Conceptual model for operational reliability
Weight of
expectations
Investment in
resources
Res-1
Res-2
.
.
.
.
Res-n
Ex-1
Ex-2
.
.
.
.
Ex-n
Expectations
Resources
Bidirectional relationship
Fig. 2. Relationship between expectations and resources.
J. Sullivan, R. Beach/International Journal of Project Management 27 (2009) 765–775
769