Towards Efficient Supercomputing: A Quest for the Right Metric.

Full text

Page 1

Towards Efficient Supercomputing:
A Quest for the Right Metric∗
Chung-Hsing Hsu, Wu-chun Feng, Jeremy S. Archuleta
Los Alamos National Laboratory
{chunghsu, feng, jsarch}@lanl.gov
Abstract
Over the past decade, we have been building less and
less efficient supercomputers, resulting in the construction
of substantially larger machine rooms and even new build-
ings. In addition, because of the thermal power envelope
of these supercomputers, a small fortune must be spent to
cool them. These infrastructure costs coupled with the ad-
ditional costs of administering and maintaining such (un-
reliable) supercomputers dramatically increases their total
cost of ownership. As a result, there has been substantialin-
terest in recent years to produce more reliable and more ef-
ficient supercomputers that are easy to maintain and use.
But how does one quantify efficient supercomputing? That
is, what metric should be used to evaluate how efficiently a
supercomputer delivers answers?
We argue that existing efficiency metrics such as the
performance-power ratio are insufficient and motivate the
need for a new type of efficiency metric, one that incorpo-
rates notions of reliability, availability, productivity, and to-
tal cost of ownership (TCO), for instance. In doing so, how-
ever, this paper raises more questions than it answers with
respect to efficiency. And in the end, we still return to the
performance-power ratio as an efficiency metric with re-
spect to power and use it to evaluate a menagerie of pro-
cessor platforms in order to provide a set of reference data
points for the high-performance computing community.
1. Motivation
In[6],Pattersonnotedthatthecomputerscienceresearch
community has focused on performance, performance, and
more performance for the past few decades. This observa-
tion is even more evident in the high-performance com-
puting (HPC) community, where the Top500 Supercom-
puter List (http://www.top500.org) and the Gordon Bell
∗
This work was supported by the DOE ASC Program through Los
Alamos National Laboratory contract W-7405-ENG-36.
Awards for Performance and Price/Performance take cen-
ter stage. Unfortunately, this focus on performance (and to
a lesser degree, price/performance, where price only cap-
tures the acquisition cost) has led the HPC community to
build less and less efficient supercomputers with lower reli-
ability, reduced availability, lower productivity, and signifi-
cantly higher total cost of ownership (TCO).
For example, while the performance of our n-body code
has increased by 2000-fold since the Cray C90 of the early
1990s, the performance per watt has only increased 300-
fold and the performance per square foot by only 65-fold.
This trend in inefficiency has resulted in the construction of
substantially larger and more complex machine rooms and
even new buildings. Further, because of the thermal enve-
lope of such supercomputers, a small fortune must be spent
to simply cool them, e.g., $6M/yearat Lawrence Livermore
National Laboratory (LLNL) [10]. The primary reason for
this less efficient use of space (as well as power) has been
the exponentially increasing power requirements of com-
pute nodes, i.e., Moore’s Law for Power Consumption, as
shown in Figure 1. When nodes consume more power, they
must be spaced out and aggressively cooled, e.g., LLNL re-
1.5µ µ
1 1µ µ
0.7µ µ
0.5µ µ
0.35µ µ
0.25µ µ
0.18µ µ
0.13µ µ
0.1µ µ
0.07µ µ
Chip Maximum
Power in watts/cm2 2
Figure 1. Moore’s Law for Power Consump-
tion.
IEEE Workshop on High-Performance, Power-Aware Computing (HP-PAC).
in conjunction with the IEEE Parallel & Distributed Processing Symposium
Denver, Colorado, April 2005, LA-UR 05-0936.

Page 2

quires an additional 0.7 watts of cooling for every 1.0 watt
of power consumed by their supercomputers [10].
Without exotic cooling facilities, traditional (inefficient)
supercomputers would be so unreliable that they would of-
tentimes be unavailable for use by the application scientist.
For instance, the empirical data from our Little Blue Pen-
guin cluster (2000-2001) as well as unpublished empirical
data from a leading vendor corroborates that the failure rate
of a compute node approximately doubles with every 10◦C
(18◦F) increase in temperature, as per Arrhenius’ equation
when applied to microelectronics.Given that temperatureis
directlyproportionaltopowerconsumption,keepinga com-
pute node cool is obviouslycritical. Unfortunately,building
such exotic cooling facilities can cost as much as the super-
computer itself, e.g., the building for the ASCI Q computer
cost nearly $100M. Operating and maintaining such facili-
ties costs even more and is still no guarantee that the super-
computer will not suffer failures, as shown in Table 1 [9].
In sum, all of the above contributes to an astronomical to-
tal cost of ownership (TCO).
To address the above problems, we took a systems de-
sign and integration approach towards building an effi-
cient, reliable, and relatively high-performance supercom-
puter dubbed Green Destiny in April 2002.1Green Destiny
is a 240-processor supercomputer that fits in a telephone
booth (i.e., a footprint of five square feet) and sips only 3.2
kW of power (when booted diskless) while sitting in an 85-
90◦F warehouse at 7,400 feet above sea level. Despite its
harsh operating environment, the reliability of Green Des-
tiny is arguably second to none: no unscheduled failures in
its 24 months of existence.
Since the debut of Green Destiny, and most notably af-
ter a New York Times article in June 2002 that compared
and contrasted the ASCI Q and Green Destiny supercom-
puters [4], we have observeda dramatic shift away from the
“Moore’s Law for Power Consumption” forecast made at
IEEE/ACMMICRO-32in1999[8].Specifically,AMD,and
more recently Intel, have temporarily veered off of “Moore
s Law for Power Consumption” and have stopped explicitly
labelingmicroprocessorsbased on clock speed, which is di-
rectly proportional to power consumption. They are instead
focusing on more efficient microprocessordesign. More di-
rectly,IBM used the issues of space and coolingto help cre-
ate a renaissance for its Blue Gene project in late 2002. In
sum, what we are really observing is a quest for more effi-
cient horsepower.
Despite the (over-exposed) success of Green Destiny,
diehard HPC system researchers reacted with incredulity.
1 Complementary approaches include the recently unveiled IBM Blue
Gene/L, which focuses on hardware design & manufacturing to de-
liver “five 9s” reliability (i.e., 99.999%) and Google’s server farm,
which focuses on delivering reliability at the software level while as-
suming hardware unreliability underneath.
Why would anyone sacrifice so much performance (i.e.,
1.5x to 2x worse) to achieve better reliability and avail-
ability (i.e., “infinitely” better) as well as lower TCO (i.e.,
2x better)? On the other hand, HPC application researchers
from biological, pharmaceutical, and application software
companies have embraced the idea enough that a start-up
company based on Green Destiny was recently launched
— Orion Multisystems (http://www.orionmulti.com).Thus,
the question that begs to be asked in the quest for more effi-
cient horsepower is as follows:
What metric should be used to evaluate how efficiently
a given system delivers answers?
This paper presents an initial attempt at answering the
above question — finding the right metric for efficient su-
percomputing. For now, we only focus on efficiency in
terms of system performance and power consumption as
well as their subsequent effects. System performance and
power consumption are easily quantifiable measures; they
are also first-class design constraints for supercomputers.
We start by examining the use of existing metrics for effi-
cient supercomputing,in particular, the performance-power
ratio commonly seen in the field of low-power circuit de-
sign. We then argue that the use of such an existing metric
is insufficient to address several key issues in efficient su-
percomputing. In other words, we need a new type of effi-
ciency metric to address the issues of reliability, availabil-
ity, productivity,and total cost of ownership (TCO).
2. Background
One place that we can borrow an existing metric for effi-
cient supercomputing is the low-power circuit design com-
munity where researchers face a similar dilemma. These re-
searchers must tackle the challenging problem of design-
ing a circuit that simultaneouslyoptimizesperformanceand
power. Naivelyfocusingon performanceultimately leads to
a designthat dissipates toomuchpowerwhereas optimizing
merely for the most power-efficient design rarely achieves
the needed performance.As a consequence, the community
has proposed metrics that combine both performance and
power measures into a single index.
2.1. The Use of An Existing Metric: EDn
The most commonly used metric in the low-power cir-
cuit design community is in the form of EDn[7] where E
is the energy, D is the circuit delay, and n is a nonnegative
integer. The power-delay product (PDP), the energy-delay
product (EDP) [3], and the energy-delay-squared product
(ED2P) [5] are all special cases of EDnwith n = 0, 1, and
2, respectively. Intuitively, EDncaptures the “energy usage
per operation.” A lower EDnvalue indicates that power is
more efficiently translated into the speed of operation. The

Page 3

System
ASCI
Q
ASCI
White
NERSC
Seaborg
#CPUs
8,192
Reliability
MTBI: 6.5 hours. 114 unplanned outages/month.
HW outage sources: sotrage, CPU , memory.
MTBF: 5 hours (2001) and 40 hours (2003).
HW outage sources: storage, CPU, 3rd-party HW.
MTBI: 14 days. MTTR: 3.3 hours.
SW is the main outage source.
8,192
6,566
Table 1. Reliability and Availability of Large-Scale HPC Systems [9].
(MTBI: Mean Time Between Interrupts, MTBF: Mean Time Between Failures, MTTR: Mean Time To Restore)
parameter n implies that a 1% reduction in circuit delay is
worth paying an n% increase in energy usage; thus, differ-
ent n values represent varying degrees of emphasis on de-
liverable performance over power consumption.
Sometimes, a variant of EDn— the number of opera-
tions per second per watt — is used. In the circuit design
community, the use of performancen/power with a higher
n value is encouragedfor performance-optimized,high-end
products. For example, using SPEC3/W to evaluate server-
class processors is considered the fairest when compared
to the use of SPEC/W or SPEC2/W [1]. It seems natural
to adopt this metric to define the efficiency of a supercom-
puter and set n ≥ 2 to put more weight on the performance
side. However, we argue that there is a danger in using this
metric for efficient supercomputing. Specifically, the met-
ric biases itself toward massively parallel supercomputing
systems.
2.2. The Biased Effect of EDn
Using EDnwith n ≥ 2 as an efficiency metric to com-
pare two supercomputerdesigns has a biased effect towards
a massively parallel HPC architecture. A large n value not
only emphasizes the performance aspect of a HPC system,
but it also exaggerates the performance gained from the
massiveparallelism. More specifically,the EDnmetric with
n ≥ 2 increases exponentially with respect to the number
of processors in a supercomputer.
Let us consider a few current and future supercomput-
ers, as listed in Figure 2. Here, for the sake of discus-
sion, we set the efficiency metric as FLOPSn/W. In order
to better present the efficiency comparisons, we normal-
ize FLOPSn/W with respect to Blue Gene/L because Blue
Gene/L has the best FLOPSn/W efficiency for all n’s and
for all the listed supercomputers.
Based solely on the FLOPS2/W metric in Figure 2, Blue
Gene/L possesses a far superior design with respect to ef-
ficiency. The second most-efficient supercomputer is ASCI
Purple, but it lags behind by several orders of magnitude.
Blue Gene/L and ASCI Purple have totally different sys-
tem architectures. Blue Gene/L uses many (131,072) em-
bedded low-performance (0.7 GHz) processors to achieve
performance-power efficiency; in contrast, ASCI Pur-
ple uses significantly fewer (12,288) but more powerful
(2.0 GHz) processors. Thus, Blue Gene/L takes advan-
tage of FLOPSn/W by aggregating a gaggle of low-power
profile embedded processors to achieve significantly bet-
ter power efficiency. The performance loss that is sustained
by using these embedded processors is more than off-
set by the additional order-of-magnitude increase in the
number of processors.
Interestingly, GreenDestiny,
puter that is based on an “efficient design” philosophy
like Blue Gene/L, falls behind all the listed supercomput-
ers by several orders of magnitude in terms of FLOPSn/W,
where n ≥ 2. The reason for the gargantuan differ-
ence in FLOPSn/W is due to the metric’s bias towards su-
percomputers with larger numbers of processors. Because
Green Destiny has a miniscule processor count when com-
pared to all the other listed supercomputers, particularly
Blue Gene/L, the merit of its system architecture with re-
spect to efficiency is marginalized. Therefore, we conclude
that using FLOPSn/W, where n ≥ 2, as an efficiency met-
ric is highly misleading given that two systems with similar
“efficient design” philosophies have two completely differ-
ent efficiency measures.
More formally, consider a supercomputer that has s pro-
cessors and each of the processors can deliver F flops at P
watts. The FLOPSn/W metric can then be written in terms
of s, F, and P, as follows.
another supercom-
FLOPSn/W =(s · F)n
s · P
= sn−1·Fn
P
(1)
Equation (1) establishes a positive correlation between
FLOPSn/W and the processor count s. That is, the met-
ric FLOPSn/W is biased toward HPC systems that con-
tain a larger number of processors (s) as n gets larger.
This partly explains why Green Destiny has a very low
FLOPSn/W — all the other supercomputershave 10x-500x
more processors than Green Destiny. As a result, Equa-

Page 4

SystemProcessor
Count
131,072
10,368
3,132
12,288
10,240
5,120
4,536
12,288
Theo. Perf.
(TFlops)
367.00
41.47
25.06
98.30
61.44
40.96
39.92
30.72
Max. Power
(MW)Name
Blue Gene/L (BG/L)
Red Storm (RS)
MACH 5 (M5)
ASCI Purple (Purple)
Columbia (Col)
Earth Simulator (ES)
MareNostrum (Mare)
ASCI Q (Q)
Green Destiny (GD)
1.2
1.7
0.4
4.7
2.0
8.0
0.6
3.0
2400.22 0.0052
0
20
40
60
80
100
BG/L RSM5PurpleCol ESMareQ GD
Normalized Metric Value (%)
System Name
The FLOPS^2/W Metric
Figure 2. The Efficiency of Several Supercomputers in terms of FLOPS2/W normalized with respect
to Blue Gene/L (BG/L).
tion (1) suggests that setting n = 1 can eliminate such a
bias.
Insum,usingEDnwitha largernvalue,so astoempha-
size performance, produces a bias toward supercomputers
with massivelyparallelHPC architectures.To eliminatethis
bias, n should be set to 1, i.e., the performance-powerratio
is a better metric for efficient supercomputing, at least rela-
tive to power consumption. Yet efficiency is not just about
how efficiently a supercomputer uses power. As discussed
earlier, we need a new efficiency metric that addresses the
issues of reliability, availability, productivity, and total cost
of ownership (TCO). Though the aforementioned discus-
sion influences these issues, it did not address any of them
directly.
2.3. Reliability, Productivity, and TCO
The FLOPSn/W metric is really a derivative of EDn,
which is borrowed from low-power circuit design and tai-
lored for use in efficient supercomputing. Another metric
that can be “borrowed” from the circuit-design community
is reliability. Reliability refers to a system’s ability to op-
erate continuously without failure and to maintain data in-
tegrity; it is usually measured in terms of mean time to fail-
ure (MTTF). In addition, there exist new metrics that are
specifically designed for HPC systems such as availability
and serviceability. System availability refers to the avail-
ability of a system irrespective of its hardware and software
reliability, e.g., though Google server farms assume unreli-
ability in their hardware, Google software provides a relia-
bility layer that ensures near 100% availability. Serviceabil-
ity refers to the ease with which a system can be brought
back to an operational state after a failure. Availability and
serviceability contribute to the productivity of a system.
While a supercomputer may be capable of computing at
petaflop speeds, it does not necessarily mean that it will be
a productive machine. For example, in one unpublished in-
stance, the expected boot time of a planned supercomputer
exceeded the forecasted MTTF. In this case, the MTTF
would havebeen so short that the productivityof the system
would be zero. Even when the MTTF moderately exceeds
the boot time, the question becomes how available and ser-
viceable will the supercomputerbe for productiveuse by an
end user.
Productivity is oftentimes quantified as the ratio of the
number of jobs that a system has run over a long period of
time to the total amount of resources that went into buy-
ing and runningthe system. Basing decisions on this defini-
tion of productivity avoids the embarrassment of procuring
”cheap” piles of hardware that never quite run applications
properly and end up cluttering the data-center floor; it also
avoids procuring fast, expensive systems solely for the pur-
pose of securing Top500 bragging rights. In short, produc-
tivity shifts the focus towards producing meaningful com-
putationaloutputgivenfixedresources,i.e.,gettingthemost
out of what you have.
The power dissipation of a supercomputer has a signifi-
cant impact on both reliability and productivity, and conse-
quently TCO. Consider the following scenario. A scientist
submitsa paralleljobthat requiresan entiredaytocomplete
on a large-scale supercomputing system. Partway through
its execution, the application encounters an unscheduled
failure, which causes the fault-management mechanism of
the supercomputing system to shut the system down. Even
if the scientist performed application-level checkpointing,
(s)he still has to wait until the system is repaired and back

Page 5

on-line before (s)he can re-submit the job to the system to
complete. (In the case where no checkpointing was done,
the scientist re-submits the job from the start and hopes that
it does not fail the next time around.) The above scenario
briefly illustrates why it is important for large-scale super-
computing systems to be highly available in delivering reli-
able and efficient computational cycles to scientists.
2.4. Where Art Thou Efficiency Metric?
In sum, we intuitively know that productivity is influ-
enced by a system’s reliability and that system reliability
is sensitive to the amount of heat dissipation from the sys-
tem. However, we are not yet aware of any efficiency met-
ric that models this cascading effect. One of the primary
obstacles in having such a model is the lack of substantive
empirical data that supports a quantitative relationship be-
tween reliability and heat stress in a large-scale supercom-
puting system. In contrast, the relationship between relia-
bility and heat stress has been well established at the cir-
cuit level. For example, Arrhenius’ equation exponentially
linksthefailurerate withan increaseintemperatureofa cir-
cuit.
While our quantitative understanding of the relationship
between reliability and heat stress may be tenuous at best,
we have some qualitative understanding that reducing the
power draw of a supercomputer will improve both relia-
bility and productivity. This is because the high tempera-
tureof a supercomputerwill morelikely induceoverheating
problems, and the increase in temperature is largely caused
by the transformation of electrical energy into heat energy.
In other words, if a supercomputer draws a reasonably low
amount of power, the damages caused by heat stress to the
hardwareparts can be dramatically reduced,thus increasing
the system’s reliability, reducing downtime, and improving
productivity.
3. Case Study: FLOPS/W Efficiency
The above discussion on identifying an efficiency met-
ric raises more questions than it answers. What we are still
leftwithistheimperfectFLOPS/W metric(andcorrespond-
ingly, the imperfect LINPACK FLOPS metric). With that in
mind, we resign ourselves to present FLOPS/W efficiency
results with the LINPACK benchmark across a gamut of
four-processor parallel-computing systems as well as a few
larger-scalecluster systems whose “natural”sizes are larger
than four processors, thus providing a set of reference data
points for the high-performance, low-power/power-aware
computingcommunity.Themicroprocessorsoftheparallel-
computing platforms that are evaluated span eight differ-
ent architectures from the low-end 1.0-GHz VIA C3 (Ne-
hemiah M10000), through the 1.133-GHz Intel PIII-M,
Wall Outlet
Profiling
ComputerComputer
Profiling
Digital
Power MeterPower Meter
Digital
Computer or
Cluster System Cluster System
Computer or
Power StripPower Strip
Figure 3. Experimental Set-Up for Benchmark
Tests
0.933-GHz Transmeta TM5800, 1.2-GHz Transmeta Ef-
ficeon, 2.0-GHz AMD Athlon64, 2.2-GHz Intel Xeon, and
1.0-GHzIntelItanium2(Deerfield),andtothehigh-end2.0-
GHz AMD Opteron.
Figure3showstheexperimentalset-upforallofourtests
with the LINPACK benchmark (HPL). The computing sys-
tem, shown in the bottom right of Figure 3 and denoted
as “Computer or Cluster System,” runs HPL over Linux
2.4.x and reports on its performance. In order to measure
the system’s power consumption, we use a Yokogawa digi-
tal power meter that is plugged into the same power strip as
the system. The power meter continuously samples the in-
stantaneous wattage at a rate of 50 kHz (i.e., every 20 s)
and delivers the readings to the profiling computing, shown
in the upper left of Figure 3.
We first performed our benchmark tests on four-
processor parallel-computing systems whose configura-
tions ranged from four single-processor nodes to two
dual-processor nodes to a single quad-processor node, as
shown in Table 2. Each system had an aggregate mem-
ory of one or more gigabytes (GB) with each processor
typically having an average of 0.5 to 1.0-GB mem-
ory. This set-up, however, put the Nexcom HiServer 318,
0.1 of Green Destiny (Figure 4), and the DT-12 (Fig-
ure 5) systems at a disadvantage, particularly the latter
system. In all of these systems, the chassis for the com-
pute nodes supported more than four processors: 18 for the
HiServer 318, 24 for 0.1 of Green Destiny (i.e., one chas-
sis), and 12 for the DT-12. Thus, even though only
four processors were powered-on in each of these sys-
tems, the power consumption of the surrounding chas-
sis was for 18, 24, and 12 processors, respectively. So,
instead of the chassis’ power consumption being amor-
tized over 18, 24, and 12 processors, respectively,they were
amortized only over four processors. In addition, the lat-
ter two systems contain embedded Ethernet backplane
networks that must be powered-on,irrespective of the num-
ber of compute nodes that are powered-on in the chassis.