A Comparison of Design Decisions made Early and Late in Development

Abstract

The occurrence rates and cost impact of design changes made early and later in the design process were studied, to test and quantify the 80-20 rule of design cost impacts, that early design decisions account for the majority of costs in a development program. Cost and schedule impact of decisions made throughout the development process was carried out at a large aerospace firm on two programs covering 7 years of development with 275 person-years effort. The underlying data used was the rate and cost of design changes made. We found no significant difference in the rate of occurrence of design change decisions made, but we found a significant difference in the cost impact of the design changes. Overall, early design change decisions cost 5 times more than later design change decisions. This difference is primarily due to the inability to determine if an early design decision is correct until later in development during testing.

Full text

ICED17
Paper ID: 142 (Draft, Accepted for publication)
A Comparison of Design Decisions made Early and Late in Development
James Tan2, Kevin Otto1, Kristin Wood2
1Aalto University, Finland; 2Singapore University of Technology and Design
ICED17: 21st International Conference on Engineering Design
University of British Columbia, Vancouver, Canada
Aug. 21-25, 2017
Full Paper Submission - DesProc
Topics: Design organisation and management, Design processes
Keywords: Requirements, Project management, Process modelling, Early design phases
Abstract:
The occurrence rates and cost impact of design changes made early and later in the design process
were studied, to test and quantify the 80-20 rule of design cost impacts, that early design decisions
account for the majority of costs in a development program. Cost and schedule impact of decisions
made throughout the development process was carried out at a large aerospace firm on two programs
covering 7 years of development with 275 person-years effort. The underlying data used was the rate
and cost of design changes made. We found no significant difference in the rate of occurrence of
design change decisions made, but we found a significant difference in the cost impact of the design
changes. Overall, early design change decisions cost 5 times more than later design change decisions.
This difference is primarily due to the inability to determine if an early design decision is correct until
later in development during testing.

ICED17
1 INTRODUCTION
The 80-20 rule implies design decisions account for 80% of the product cost, as described in several
works and as studied and reviewed by Ulrich and Scott (1993). Similarly Ullman (2015) concurred
and found that 75% of the product cost is committed during design concept generation and further the
freedom to change the design reduces as the development progresses. Studies within industry practice
have shown that the opportunities for life cycle cost savings reduces as the design matures (Aerospace
Industry Association, 2009). In this paper, we seek to refine the underlying understanding of these
assertions by looking at past development programs within a large aerospace firm to establish the cost
impact differences of design decisions and the occurrence rates of incorrect decisions, or design
defects, over the development process. In combination, this analysis can show the relative impact of
decisions made early versus late in development. We found a slightly higher occurrence rate of design
defects in the early program phases than later phases. More significantly, we found that changes in
design decisions made early require significantly more rework activities, nominally thirteen times
more rework on average. Taken together, these observations indicate early concept design decisions
have significantly higher cost impact. In the projects analysed, early design phase decisions accounted
for 86% of the total potential rework cost. Therefore, the analysis confirmed the 80-20 rule with
statistical significance.
2 RELATED RESEARCH AND STUDY SCOPE
In the literature, the 80-20 rule has been considered and supported at different industrial scales. In the
defense industry, Mark et al (2006) considered entire weapons systems in general, whereas Obaid and
David (2003) analyzed several fighter programs. Elsewhere, Tassey (2002) found similar design
impact trends in the software community. While all these works report the high impact nature of
design, there remains work to highlight where in design this high sensitivity exists. We seek here to
provide a refined understanding of occurrence and sensitivity, how often correct or incorrect decisions
are made at different points in design and their relative cost implications, given the inherent uncertain
nature of understanding of outcomes in development.
Using the 80-20 heuristic as motivation, several authors have sought improvements to the design
process. Ulrich and Scott (1998) studied design process improvements for coffee makers based on the
80-20 rule. Geoffrey et.al. (1994) used the 80-20 heuristic as motivation for adopting design for
manufacturing and assembly (DFMA) methods. These works seek to understand in quantifiable detail
the mechanics of design activities on the product development process.
Phase-gate design processes are often used in practice (Cooper, 1990, 2011), where the gate review
process provides an opportunity to review and consider early potentially non-complying requirements.
The gate review also allocates resources to the various stages of the developmental process (Cooper,
1998). Gate reviews are documented activities where decisions are discussed, critiqued and possible
additional refined work activities defined. Therefore, gate review documentation provides data to
study the impact of design decisions made before and after each gate review. Of the several gate
reviews in a development process, if the 80-20 rule is correct, then the early concept freeze gate
review can be considered a highly sensitive milestone. It is when the early design decisions are
completed and subsequent activities are approved to instantiate the chosen design. Therefore, to study
the 80-20 rule, we choose to study the impact of decisions made before and after the concept freeze.
In summary, between decisions made before and after the concept freeze, we seek to understand the
difference in cost impact of design decisions made.
Cost impact can be measured in terms of finances such as costs to develop, build and use the new
design. Design process costs can be measured as work activities or time and materials measurements.
In our study, cost is determined as the number of work activities and the additional cost is a result of
rework activities.
An entirely separate consideration is the value of design decisions, whether design decisions made
contribute to added future revenues. This is outside the scope of our work, which, as the works above,
restrict to considering cost impacts of design decisions made.

ICED17
Costs of design decisions can then be thought of in terms of differences between correct and incorrect
decisions made, where a correct means the decision enables lower total cost to the firm. For our
purposes, a design defect is the consequence of an incorrect decision made and this is recorded by
missed requirements. We seek to understand the rate of occurrence of incorrect decisions made in the
activities both before and after the conceptual design freeze. We also seek to understand the
difference in cost sensitivity of decisions made both before and after the conceptual design freeze.
Both of these factors contribute to the support of whether design accounts for 80% of cost.
To analyze this, we first review a typical corporate stage gate design process and clarify the relevant
gates. Then, we consider a set of past projects case studies, and clarify design defects in the generated
outcome designs, both those reworked and those defects that were never fixed. These sources of data
provide the inputs for the analytical process as shown in Figure 1. Next, such outcome design defects
will be traced to its root cause decision made. Further, the actual and ‘what-if’ rework activities are
then analyzed to form a relationship and cost impact assessment. Finally, we will discuss the impact of
our findings and provide opportunities for future work.
Figure 1. Data Sources and Analysis Flow
3 CORPORATE DESIGN PROCESS
We map our typical corporate design processes (Cooper 2001, Anderson, 2004, Hales 2011) to be
consistent with the open and publicly documented DOD 5000-2P standard (Department of Defense,
2001, 2015). It has 4 major phases and 3 major milestones as shown in Figure 2. Milestone A is the
end of the requirements analysis, while Milestone B marks the concept freeze and provides the go
ahead for development. Milestone C freezes the design and checks for production feasibility. Our
scope of study begins with Milestone A and ends after the initial deployment of the product.
Figure 2. The DOD 5000-2P Process

ICED17
Using the DOD 5000-P2 process, developmental stages are delimited with design technical reviews as
gates (Department of Defense, 2001). The design reviews we study are in the following order; systems
requirements review (SRR), preliminary design review (PDR), critical design review (CDR), and
production go ahead (PGA). The periods between the gates we denote as phases: the activities before
the concept is frozen (Before Concept Freeze, BCF), after the concept is frozen (After Concept Freeze,
ACF), after the design is frozen (After Design Freeze, ADF) and after the production go ahead (After
Production Go Ahead, APGA). These phases scope the extent of our analysis and is shown in Figure
3.
Figure 3. Corporate Design Process
4 CASE STUDIES IN THE LITERATURE
Before studying our industrial firm, we first studied the literature for other case studies on design
defects. The case studies originate from various industries. We examined the overrun cost and
schedule together with the root cause of the design defect as found in the literature and industry
sources. Initial projected costs / time were used as the baseline, and reported additional cost / time
were factored in normalised. This derived the percentage of total program cost / schedule slip
attributed to the issue identified. The design case studies are summarized in Table 2, and the cost and
schedule impact are presented in Figures 4 and 5.
Table 2. Case Studies of Design Defects
Case Study
Issue
Root Cause
Reference
Millennium
Bridge
Resonance
Lateral vibration mode not
considered in the design.
BBC, 2000
Chinook Mk 3
upgrade
Avionics uncertifiable
Unverifiable software
operating parameters.
Burr, 2008
A400M
Performance requirements
mismatch
Ill-defined work share
program.
Brothers, 2014
A380
Wiring connection post
assembly
Incompatible Design Tools.
Wong, 2006
F-22
Unstable industrial base
Development concept of
the system architecture was
not reviewed adequately.
Obaid, 2003
SH-2G(A)
Unable to operate in low
light conditions
Inadequate requirements
analysis.
Tom, 2005
S-80 Submarine
Unable to surface after
submerging
Calculation error.
Govan, 2013
Citi Corp Center
Building could not
withstand quarterly winds
Inadequate calculation
verification process.
Vadaro, 2013
Boeing 787
Thermal runaway in
batteries
Outdated design
verification process.
Willard, 2013
Type 45 Destroyer
Failure of power
generation system
System design with a single
point of failure.
Batchelor, 2016

ICED17
Figure 4. Cost Overrun Impact of Missed Requirements
Figure 5. Schedule Impact of Missed Requirements
As can be seen in Figures 4 and 5, from the data presented in the literature the root causes of design
defects originating from before concept freeze exhibit higher impact on cost and schedule than those
of after concept freeze. Such literature case studies are anecdotal and provides motivation to
investigate further. We do so here by examining a company’s design process and past results in detail.
5 DATA GATHERING AND ANALYSIS
We considered the development history of a large aerospace firm over the past 7 years. We select this
timeframe as it covered two multi-year development programs of new unmanned aerial vehicles
(UAVs). The programs were executed over 7 years of continuous development effort, representing
275 equivalent person-years of internal corporate effort. Both programs exhibited different forms of
missed requirements late in the testing phase. A design defect was identified as an item entered in the
Engineering Change Request (ECR) system. For each ECR, we conducted root cause analyses with
the practicing engineers (via interviews with documentation reviews) to trace each design defect from
onset detection through the ECR system, the Design Review documentation, the Failure Reports
System (FRACAS) and gate reviews, to thereby identify the root cause design decision. We captured
the activity and phase of the root cause design decision. An inter-rater-reliability assessment between
analysts within the company was carried out to ensure the categorization of the program phase for
each design defect. The Cohen’s kappa was 0.810, indicating good agreement.
In the programs studied, 264 ECRs, FRACAs, gate review and design reports were reviewed for a
total of 211 requirements. From this, 58 design defects occurred resulting in missed requirements, and
thus relevant to our study scope. The resulting corrective actions were tracked through the design

ICED17
process. Each design defect translated from one of our data sources. We analysed this data set of
design changes for point of occurrence and for cost impact of rework activities between program
segment via root cause analysis of the work activities. The insights generated from the analysis will
then provide basis for improvement of standard work flows.
To do this, the necessary rework activities to fix each identified design defects (58 in total) were
determined. To describe the various types of problems involved, the design defects are grouped by
system type in Table 3. As shown, most design defects were associated with changes to the fuselage
and avionics, though all major UAV systems experienced at least one design defect.
Table 3. Design Defect Identification by System Type
Design Defect ID
Design Defect (System)
4,5,6,34,39,40,41,44,
47,48,49,54,55,56,58
Airframe
9, 22
Electrical Power
13,14
Electrical System Installation
38,43,45,46
Engine
35,36
Exhaust
20,21
Fuel
1,2,12,23,24,25,
26,28,29,30,31,37
Fuselage
7,8,15,16,18,19,42,
50,51,52,53,57
Modular Avionics
10,33
Landing Gear
11,17
Navigation
3,27,32
Stabilizer
To study each design defect, the development process activities were analysed from the activity where
the onset detection of the design defect occurred, and then traced back through the project schedule to
the precedent activities. This is continued until the root cause activity was identified. The count of
activities impacted were then summed. For example, if a design defect was detected in the after
concept freeze (ACF) phase and the defect root cause was traced to the before concept freeze (BCF)
phase, then the total number of activities required to address the design defect will include those
within the program segments BCF and ACF. At the other extreme, if the root cause of the design
defect was discovered at the point of detection, then rework activities are confined to the single
activity only.
The root causes of design defects by development process tasks is shown in Figure 6. As can be seen,
a large fraction of design defects were ‘Requirements Definition’ changes; the requirements defined
turned out to be inconsistent. For example, a chosen payload was in excess of that possible given the
chosen range.

ICED17
Figure 6: Design defects root causes by development process task.
Using this methodology, the required rework for all major design defects that arose is shown in
Figures 7 and 8, for defects with root cause before concept freeze and after concept freeze
respectively. First, each figure depicts the number of actual rework activities completed when the
design defect was discovered. For comparison, each figure also depicts the number of rework activities
that would have been required to fix the defect if it had been discovered immediately at the root
program segment.
Figure 7. Rework activities required for the root causes that arose in BCF
Figure 8. Rework activities required for the root causes that arose in ACF

ICED17
From the figures, we can see that rework activities that arose from BCF root causes were significantly
more than those of ACF root causes. The differences from before vs after concept freeze can be
quantified. The average rework required difference from before vs after concept freeze can be
computed from Figures 7 and 8.
The results as shown in Figures 7 and 8 indicate earlier decisions are costlier to change. From the
overview illustrated in Figure 9, the average rate of increase of BCF multiplier rework activities
between the BCF and ACF program is 2.9, while for BCF and ADF the multiplier is 15.1. For the
multiplier between BCF and APGA, the multiplier value is 12.0 and finally the average multiplier
across all program segments is 13.0.
Figure 9. Increase of actual rework activities using BCF multipliers
A further observation from Figure 6 is that many early phase decisions could not be completely
determined correct until late in the program, during the build and test phases. Fundamentally, this is
the reason of the large rework activity levels. Few of the root causes before concept freeze could be
detected before concept freeze, and so were incorrectly carried forward into the late phase testing.
This resulted in high cost sensitivity of these early design decisions. As indicated in Figure 10, we
observe the following from our data; 9 of design defects were not resolved (permanent non-
compliance due to the high cost involved), 13 of the 49 design defects required re-certification on top
of drawing changes.
Figure 10: Design Defects Resolution Outcomes
In addition to cost impact, however, the occurrence rate of decision changes with relation to work
activities need to be considered as shown in Table 4. Each program segment had a different rate of
occurrence in poor design decisions that resulted in design defects. The BCF, ACF and ADF program
segments had 59%, 28% and 22% chance of poor design decisions respectively, clearly showing early
design phase decisions also have a higher error rate and are prone to being changed.

ICED17
Table 4: Design Defect Occurrence Rate
Program
Segment
Activities
Activities
with
Defects
Defect
Rate
BCF
17
10
59%
ACF
75
21
28%
ADF
27
6
22%
The cost impact of changing any decision is the occurrence probability times the rework required. The
analysis indicates design decision have different likelihood of error based on program segment.
Further, with the ability to detect an incorrect early decision delayed until the later testing phase, the
results showed that rework levels are 13 times higher on average for early phase decisions.
Theoretically, the cost of changing all design decisions made over the development process is the sum
over all decisions of their change occurrence probabilities multiplied by cost impacts. Since the
occurrence probability rates vary, the relative cost of design decisions before-versus after-concept-
freeze is significant. We determine the design decision defect costs by summing up the total work
activities across the program segments with the rework multiplier as tabulated in Table 5. As can be
seen, the rework cost attributed from before the design freeze was 86%.
Table 5: Rework Activities Cost Contribution
Segment
Activities
Rework
Multiplier
Probability
of rework
Cost
Contribution
Percent
Contribution
BCF
17
13.0
0.59
147.0
33.5%
ACF
75
7.4
0.28
231.2
52.7%
ADF
27
2.9
0.22
44.1
10.1%
APGA
16
1.0
0.00
16.0
3.7%
6 CONCLUSION
At a major aerospace firm, the cost of changing design decisions made in the conceptual design phase
were found to be on average 13 times larger than the cost of changing decisions made after the concept
was fixed. Fundamentally, this difference was due to the inability to detect the need to change early
design decisions until late in the program during system testing. We also found that occurrence rates
of revised design decisions was higher earlier in the development process.
Considering the set of all design decisions made over the development process, the expected cost
impact of changing any decision is the occurrence probability times the rework required. With the
ability to detect an incorrect decision delayed until the testing phase, we determined that early design
decisions account for 86% cost impact of all design decisions. This supports the 80-20 rule in terms of
pre-and post-design freeze activities.
Future work would include repeating this study across different industries and scales, such as a meta
study on various industrial companies. Interventive measures are also needed to reduce the occurrence
rate of conceptual design changes. Particularly, large benefits would accrue when reducing the time
needed to identify early phase design errors, rather than being unable to detect them until the latter
testing phases of development.
ACKNOWLEDGEMENTS
This work has been funded and supported by the Economic Development Board of Singapore under
the Industrial Partnership Programme. The authors would also like to thank the SUTD-MIT

ICED17
International Design Centre (IDC, idc.sutd/edu/sg) for financial and intellectual support. Any
opinions, findings, or recommendations are those of the authors and do not necessarily reflect the
views of the supporters.
REFERENCES
Aerospace Industries Association S3000L International procedure specification for Logistics Support Analysis
(LSA), 2009.
Anderson Jr, John David. Fundamentals of aerodynamics. Tata McGraw-Hill Education, 2010.
Batchelor, Tom. “Home.” UK (Express.co.uk), July 22, 2016. http://www.express.co.uk/news/uk/692341/Type-
45-destroyer-engine-problems-analyst-claims-MoD-failed-properly-test-engines.
BBC. “Millions to Repair Swaying Bridge.” BBC UK (BBC News), November 16, 2000.
http://news.bbc.co.uk/2/hi/uk_news/1026224.stm
Brothers Caroline. “Germany and France Delay Decision on Airbus Military Transport.” Global
Business (International Herald Tribune), November 8, 2014.
Burr. T. “Chinook MK 3 Helicopters (Rep. No. HC-512).” London: National Audit Office (2008).
Cooper, Robert G. "Stage-gate systems: a new tool for managing new products." Business horizons 33, no. 3
(1990): 44-54.
Cooper, Robert G. Winning at new products: Creating value through innovation. Basic Books, 2011.
Cooper, Robert G., and Elko J. Kleinschmidt. "Resource allocation in the new product process." Industrial
Marketing Management 17, no. 3 (1988): 249-262.
Department of Defense “Department of Defense Instruction, Number 5000.02” Department of Defense (2015).
Department of Defense, System Management College “Systems Engineering Fundamentals” Department of
Defense (2001).
Geoffrey B. “Product Design for Manufacturing and Assembly” Computer Aided Design Volume 26 (1994).
Govan, Fiona. “£2 Billion Spanish Navy Submarine Will Sink to Bottom of Sea.” The
Telegraph(Telegraph.co.uk), May 22, 2013.
http://www.telegraph.co.uk/news/worldnews/europe/spain/10073951/2-billion-Spanish-navy-submarine-
will-sink-to-bottom-of-sea.html.
Hales, Crispin, and Shayne Gooch. Managing engineering design. Springer Science & Business Media, 2011.
Mark V.A., Robert S.L., Shelia E.M., Obaid Y, “Historical Cost Growth of Completed Weapon System
Programs.” RAND Corporation, RAND (2006).
Obaid Y, David E.S, Mark A.L., Frances M.L., “Lessons learnt from the F/A 22 and F-18 E/F developmental
programs” RAND Corporation (2003).
Tassey, Gregory. "The economic impacts of inadequate infrastructure for software testing." National Institute of
Standards and Technology, RTI Project7007, no. 011 (2002).
Tom Allard “Navy’s $100m Chopper Can’t Fly in Bad Light - National - www.smh.com.au.” March 19, 2005.
Accessed December 3, 2016. http://www.smh.com.au/news/National/Navys-100m-chopper-cant-fly-in-
bad-light/2005/03/18/1111086017635.html
Ullman. David G. The mechanical design process” McGraw-Hill Education, 2015.
Ulrich, Karl T., and Scott Alan Pearson. "Does product design really determine 80% of manufacturing cost?."
(1993).
Ulrich, Karl T., and Scott Pearson. "Assessing the importance of design through product
archaeology." Management Science 44, no. 3 (1998): 352-369.
Vadaro, M.J “LeMessurier Stands Tall: A Case Study in Professional Ethics” The AIA Trust (2013).
Williard, Nicholas, Wei He, Christopher Hendricks, and Michael Pecht. "Lessons learned from the 787
Dreamliner issue on lithium-ion battery reliability." Energies 6, no. 9 (2013): 4682-4695.
Wong, Kenneth “What Grounded the Airbus A380?” 2006. Accessed December 3, 2016.
http://www.cadalyst.com/management/what-grounded-airbus-a380-5955.