Green Supercomputing in a Desktop Box

Abstract

The advent of the Beowulf cluster in 1994 provided dedicated compute cycles, i.e., supercomputing for the masses, as a cost-effective alternative to large supercomputers, i.e., supercomputing for the few. However as the cluster movement matured, these clusters became like their large-scale supercomputing brethren - a shared (and power-hungry) datacenter resource that must reside in a actively-cooled machine room in order to operate properly. The above observation, coupled with the increasing performance gap between the PC and supercomputer, provides the motivation for a "green supercomputer" in a desktop box. Thus, this paper presents and evaluates such an architectural solution: a 12-node personal desktop supercomputer that offers an interactive environment for developing parallel codes and achieves 14 Gflops on Linpack but sips only 185 watts of power at load - all this in the approximate form factor of a Sun SPARCstation 1 pizza box.

Full text

Page 1

Green Supercomputing in a Desktop Box∗
Wu-chun Feng∗, Avery Ching†, and Chung-Hsing Hsu‡
∗Virginia Tech
Dept. of Computer Science
Blacksburg, VA 24061 USA
feng@cs.vt.edu
†Northwestern University
Dept. of EECS
Evanston, IL 60208 USA
aching@ece.northwestern.edu
‡Los Alamos National Laboratory
Advanced Computing Lab
Los Alamos, NM 87545 USA
chunghsu@lanl.gov
Abstract
The advent of the Beowulf cluster in 1994 provided ded-
icated compute cycles, i.e., supercomputing for the masses,
as a cost-effective alternative to large supercomputers, i.e.,
supercomputing for the few. However, as the cluster move-
ment matured, these clusters became like their large-scale
supercomputing brethren — a shared (and power-hungry)
datacenter resource that must reside in a actively-cooled
machine room in order to operate properly. The above ob-
servation, coupledwith the increasing performancegap be-
tween the PC and supercomputer, provides the motivation
for a “green supercomputer” in a desktop box. Thus, this
paper presents and evaluates such an architectural solu-
tion: a 12-node personal desktop supercomputer that offers
an interactive environment for developing parallel codes
and achieves 14 Gflops on Linpack but sips only 185 watts
of power at load — all this in the approximate form factor
of a Sun SPARCstation 1 pizza box.
1 Introduction
Sun Microsystems introduced the first workstation to
the scientific computing community in 1982. By the late
1980s, Sun had become the undisputed leader of the work-
station market when they introduced the Sun SPARCsta-
∗This paper is also available as Los Alamos Technical Report LA-UR-
07-0360.
1-4244-0910-1/07/$20.00 c ?2007 IEEE.
tion 1, rated at 12.5 MIPS and 1.4 Mflops while running
at 20 MHz.The workstation’s features were so tightly
integrated that they fit in a 16” x 16” x 3” enclosure —
the first “pizza box” workstation.
duced the first multiprocessing desktop workstation, the
SunSPARCstation 10with dual60-MHzSuperSPARC pro-
cessors; but rather than continue to scale-up the number of
processors in the SPARCstation 10, Sun instead delivered
the 64-bit UltraSPARC in the mid-1990s even though its
price-performanceratio was much worse than PCs. This ar-
guablyled to the demise of the computerworkstation, when
coupled with the emergence of PCs as cost-effective alter-
natives to workstations.
Concurrent to the emergence of the PC was the open-
source Linux operating system (OS). This confluence of
technologies ultimately led to the Beowulf commodity-
clustering movement [1], a movement that dramatically
lowered the entry costs into high-performance computing
for computational scientists and provideddedicated “super-
computing for the rest of us.” However, as this movement
matured through the late 1990s and early 2000s, commod-
ity clusters became the very thing that they were purported
to be an alternative to, i.e., an expensive, expansive, and
power-hungry resource that resides in a specially-cooled
datacenter whose shared use is arbitrated by a batch sched-
uler such as LSF or PBS.
With the notion of “supercomputing for the rest of us”
now effectively obsolete, how does an application scientist
develop a parallel code on the desktop? A dual-processor
SMP platform like the Dell PowerEdge 2650 may neither
be enough to debug a parallel code nor to test its scalability.
By 1992, Sun intro-

Page 2

On the other hand, using a shared datacenter HPC resource
like a large-scale cluster can result in scheduling conflicts
or long queues, resulting in longer turnaround times for the
program developer. This makes debugging a parallel ap-
plication more of a batch process than an interactive one.
Thus, to address the above, this paper presents the origin,
architecture, and performance evaluation of a green super-
computerinadesktopbox,i.e.,a12-processordesktopclus-
ter in an oversized pizza box and with a peak power enve-
lope of only 185 watts when running Linpack.1
The remainder of the paper is organizedas follows. Sec-
tion 2 briefly explains the origin of the aforementioned
green supercomputer in a desktop box, specifically the
Orion Multisystems DT-12. Section 3 provides an architec-
tural overview of the low-power DT-12. Section 4 presents
an initial performance evaluation of the DT-12 (circa 2004)
versus a typical SMP “desktop” server (also circa 2004),
specifically, a Dell PowerEdge 2650 server. Finally, Sec-
tion 5 concludes our work.
2 Background
TherootsoftheOrionMultisystemsDT-12canbetraced
back to the energy-efficientGreen Destinycluster [5, 14, 3],
which leveraged Beowulf cluster technology (such as com-
modity hardware, Linux, and MPI) while being energycon-
scious (i.e., power awareness at design time) in order to im-
prove the reliability and availability of compute cycles. The
basic idea behind the DT-12 was to deliver the above ad-
vantages of Green Destiny but in the form factor of a “pizza
box,” thus filling the widening performance gap between
supercomputers and traditional PC workstations.
Back in 2001, we observed that supercomputers were
becoming less efficient with respect to both power and
space consumption. For example, though the performance
on our n-body code that simulates galaxy formation in-
creased 2000-fold from the early 1990s to the early 2000s,
the performance-per-watt and performance-per-square-foot
only improved by 300-fold and 60-fold, respectively. This
has resulted in the construction of massive datacenters with
exotic cooling facilities (and even, entirely new buildings)
to house these supercomputers, thus leading to an extraor-
dinarily high total cost of ownership.
The main reason for this inefficiency has been the expo-
nentially increasing power requirements of compute nodes,
i.e., Moore’s Law for Power Consumption [5, 3, 4, 7].
When nodes draw more power, they must be spaced out
and aggressively cooled.2Our own empirical data as well
1For reference, a Dell PowerEdge 2650 desktop server with dual 2.2-
GHz Intel Xeons consumes nearly 220 watts when running Linpack.
2Perhaps a more insidious problem to the above inefficiency is that the
reliability of these systems continues to decrease as traditional supercom-
puters continue to aggregate more processors together.
as unpublished empirical data from a leading vendor indi-
cates that the failure rate of a compute node doubles with
every 10◦C (18◦F) increase in temperature above 75◦F, and
temperature is proportional to power density. Thus, the su-
percomputers in the TOP500 List require exotic cooling
facilities; otherwise, they would be so unreliable (due to
overheating-induced failures) that they would be unavail-
able for use by the application scientist.
To address the above, we started the Supercomputing
in Small Spaces (http://sss.lanl.gov/) project in late 2001
and identified low-power building blocks to construct our
energy-efficient Green Destiny [5, 14, 3], a 240-processor
Beowulf cluster that fit in a telephone booth (i.e., a foot-
print of five square feet) and sipped only 3.2 kilowatts when
running diskless, i.e., two hairdryers, but with performance
slightly better than a 172-processor Cray T3E 900 (circa
11/2001 TOP500 List) when running Linpack. Green Des-
tiny provided reliable compute cycles with no unscheduled
downtime from April 2002 to April 2004, all while sitting
in an 85◦F dusty warehouse at 7,400 feet above sea level —
thus illustrating its ability to be moved out of the datacenter
andintoan“officespace”thatdidnothaveanyspecial cool-
ing facilities. This transformationfrom datacentercluster to
office cluster ultimately led to a subsequent transformation
into a desktop cluster, as embodied by the Orion Multisys-
tems DT-12.
3Architectural Overview
The Orion Multisystems DT-12, as shown in Figure 1,
is a personal desktop cluster workstation that contains 12
individual x86 compute nodes in a 24” x 18” x 4” (or one
cubic foot) pizza-box enclosure. Collectively, these nodes
provide the horsepower of a small supercomputer, the ad-
ministrative ease of a single-processor computer,3and the
low noise, heat, and power draw of a conventional desktop.
Figure 1. The DT-12 Personal Desktop Cluster
Workstation.
Each compute node contains a Transmeta Efficeon pro-
cessor runningthe Linux operating system and Transmeta’s
3Booting the DT-12 amounts to depressing a single power switch, and
in just over a minute, the entire single-system image is then up and ready
to run a parallel job.
2

Page 3

power-aware LongRun2 software, its own memory and Gi-
gabit Ethernet interface, and optionally, its own hard disk
drive. The nodes share a power supply, cooling system,
and external 10-Gigabit Ethernet network connection. (The
head node provides an interface to the end user.)
In short, the DT-12 arguably exports the utmost in sim-
plicity with only one power plug, one power switch, one
monitorconnection,one keyboard,andone mouse. At load,
it achieves 14 Gflops on Linpack while drawing only 185
watts of power, i.e., less than two 100-watt light bulbs.
3.1Hardware
The DT-12 is a cluster of 12 x86-compatible nodes
linked by a switched Gigabit Ethernet fabric.4The cluster
operates as a single computer with a single power switch
and a single-system image rapid-boot sequence, which al-
lows the entire system to comeon-linein just overa minute.
In the DT-12, one node functions as the head node, pro-
viding an interface to the end user and controlling jobs
throughout the cluster. The other nodes, referred to as com-
pute nodes, are available to the head node for parallel com-
puting. When idle, the head node can also act as a compute
node.
TheDT-12plugsdirectlyintoa standard15-Aoffice out-
let with no special cooling or power requirements. The in-
cluded I/O board provides video, keyboard and mouse, se-
rial port, USB, and fan control. The DT-12 also provides
a DVD/CD-RW and one 3.5” hard drive on the head node.
The board can accommodate one 2.5” hard-disk drive per
each of the other nodes althoughdisk drives on the compute
nodes are optional.
3.2Software
With the single-system image rapid-boot sequence from
Orion Multisystems, the DT-12 appears as monolithic as
possible to the end user, e.g., logging into individual com-
pute nodes is typically unnecessary. The system software
accomplishes this by providing the same Linux kernel on
all nodes via a single OS installation on the head node that
is shared among all compute nodes. Each compute node
then locally runs commonly used cluster tools such as rsh,
Sun Grid Engine, and MPICH2.
4Experimental Results
In this section, we present an initial performance eval-
uation of the DT-12 from 2004 and compare it to a high-
enddevelopmentplatforminuse from2004,specificallythe
4Linking DT-12 systems together can then be achieved via the external
10-Gigabit Ethernet interface.
Platform
SPECint2000
SPECfp2000
DT-12
526
358
PowerEdge 2650
792
726
Table 2. SPEC CPU2000 Results
Dell PowerEdge 2650 SMP. We chose this comparison for
severalreasons. First, with Beowulfclustershavingbecome
shared datacenter resources, we argue that application pro-
grammers have turned to high-end desktop server platforms
like the Dell PowerEdge 2650 in order to develop their par-
allel codes. Such a desktop SMP machine can fit on one’s
desk without the need for special wall outlets or cooling
facilities, just like the similarly sized Orion Multisystems
DT-12. Second, in addition to the DT-12 having similar di-
mensionsto the Dell PowerEdge2650, is also uses a similar
amount of power. Third, the price of each system was four
digits, i.e., approximately $9,000 for the DT-12 and $5,000
for the PowerEdge 2650.
OurDell PowerEdge2650isa dual-processor2Uchassis
machine, configured with dual 2.2-GHz Xeon processors,
1.5-GB memory, and a Fujitsu 18.4-GB 15000 RPM U160
SCSI drive. Each Xeon processor used hyper-threading for
a total of 4 virtual processors. A quick comparison of the
DT-12 and the PowerEdge 2650 in Table 1 illustrates some
of the key hardware differences between the two develop-
mental platforms. The dimensions of the two platforms are
nearly identical, and the power draws at load (i.e., Linpack)
are comparable.
We conductedmany experiments to show that a personal
desktop cluster workstation that is energy efficient, such as
the Orion Multisystems DT-12, can provide a more suitable
development platform for parallel codes versus the typical
SMP workstation. More importantly, we demonstrate that
the DT-12 can deliver green supercomputing in a desktop
box. OurbenchmarksincludedSPECCPU2000,HPL,NAS
MPI, STREAMS MPI, ttcp, bonnie, and mpi-io-test.
4.1Processor Performance
As an initial performance characterization of the Ef-
ficeon processor used in the DT-12, we began our experi-
ments with the SPEC CPU2000 benchmarks [11] using an
Intel compiler (version 8.1) and with the baseline optimiza-
tion options -O3 -xW -ipo. Table 2 shows our base-
line results. The DT-12 achieved 526 for SPECint2000 and
358 for SPECfp2000 while the numbers for the PowerEdge
2650 were 792 and 726, respectively. If we had run the
same customized (but proprietary) high-performance code-
morphing software on the DT-12 as we did on Green Des-
tiny, the floating-point performance would have improved
by about 50% to the neighborhoodof 535 for SPECfp2000.
3

Page 4

Platform
Dimensions
Orion Multisystems DT-12
24(W) x 18(D) x 4(H) (in)
1 cubic foot
185.3 watts
Twelve 1.2-GHz Efficeon CPUs
1 GB/Processor
Gigabit Ethernet NIC/Processor
Dell PowerEdge 2650
19(W) x 26(D) x 3.5(H) (in)
∼ 1.02 cubic feet
217.0 watts
Two 2.2-GHz Xeon CPUs
1.5 GB
Dual Gigabit Ethernet NIC
Power at Load
Processors
Memory
Network
Interfaces
Storage80-GB 5400 RPM IDE (Head)
20-GB 4200 RPM IDE (Other)
18.4-GB 15000 RPM SCSI drive
Table 1. Experimental Hardware Configurations
According to the SPEC measurements, the integer per-
formanceof a 1.2-GHzEfficeonprocessoris roughlyequiv-
alent to a 1.5-GHz Pentium 4 (i.e., between 515 and 534).
However, the floating-point performance of the Efficeon
processor does not keep up with the 1.5-GHz Pentium 4
(i.e., between 543 and 549, compared to the Efficeon’s
358) although it likely would have if Transmeta had used
the prototypical high-performancecode-morphingsoftware
that was originally developed for Green Destiny and tai-
lored towards iterative scientific codes.
Our next test, HPL [6], is a freely available software
package that implements the Linpack benchmark. It solves
a (random)denselinearsystemin doubleprecision(64-bits)
arithmetic on distributed-memory computers. Using HPL
requires an MPI 1.1 compliant implementation and either
the Basic Linear Algebra Subprograms (BLAS) or the Vec-
tor Signal Image Processing Library (VSIPL). Generally,
HPL is considered scalable with respect to the number of
processors used duringtesting since its overall performance
is mostly attributed to the system’s CPU.
For HPL, the DT-12 delivers 14.17 Gflops while the
Dell PowerEdge 2560 achieves 5.1 Gflops, or roughly three
times slower than the DT-12. Since the DT-12 has twelve
processors compared to the two processors in the Pow-
erEdge 2650, we infer that a 2.2-GHz Xeon processor per-
formsslightly morethantwice as fast as a 1.2-GHzEfficeon
in this application.
In our final CPU test, we ran experimentsusing the NAS
Parallel Benchmarks (NPB) [9]. Collectively, they mimic
the computation and data-movement characteristics of ap-
plications in computational fluid dynamics (CFD). NPB is
based on Fortran and MPI. These implementations, which
are intended to be run with little or no tuning, approxi-
mate the performance that a typical user can expect from
a portable parallel program in a distributed-memory com-
puting system.
Table 3andTable 4 showthe results fromthe class A and
class B workload, respectively. Due to process restrictions
in some tests, we only obtained results for certain numbers
of processes. For example,SP and BT requirethat the num-
ber of processors be a square of an integer, therefore we
have results from 1, 4, and 9 processors. In Table 4, FT was
not able to complete for the PowerEdge 2650 (even after
waiting numerous minutes). We attribute this to insufficient
memory and heavy disk swapping.
In general, as the numberof processes increases, the DT-
12 scales nearly linearly in the LU, BT, EP, and CG tests.
The FT, MG, SP, and IS tests do scale to some degree on
the DT-12, although not linearly. However, because some
of these tests are dependent on system components besides
theprocessor,theyarenotexpectedtoscalelinearly. Incon-
trast, the PowerEdge 2650 does not achieve linear scalabil-
ity on any of the 8 NPB tests and would provide poor feed-
back on demonstrating whether a parallel code is achiev-
ing an expected linear speedup. Running more processes
than the number of actual processors on the dual-processor
PowerEdge 2650 does not allow for any useful scalability
testing.
4.2 Memory Performance
The STREAM benchmark[12] measures the sustainable
memorybandwidthand the correspondingcomputationrate
for simple vector kernels. The STREAM benchmark has
four different components which are summarized in Ta-
ble 5.
We used the MPI version of STREAM to test the scal-
ability of the memory subsystem. Each test was compiled
with MPICH2 [8] andrun three times with the best runused
in our results. We chose to use the best run for this bench-
mark since memory testing is particularly sensitive to any
OS interrupts. Figure 2a shows that the DT-12 achieves
linear scalability while the PowerEdge 2650 in Figure 2b
struggles to achieve any speedup. While it is a two-way
SMP and each processor has hyper-threading(thereby hav-
ing four virtual processors), we hardly see any speedup be-
tween 1 to 4 processors on the PowerEdge 2650.
4

Page 5

Class A Workload
DT-12
8
543.37
1180.40
PowerEdge 2650
8
361.20 347.42
273.49313.86
213.08
419.88419.46
810.68
30.7528.93
17.18 17.61
249.97 221.15
149 12149 12
FT
MG
SP
LU
BT
IS
EP
CG
143.29
97.42
98.95
179.12
268.56
11.23
3.46
70.27
287.84
359.49
319.74
779.28
946.81
17.41
13.80
260.61
286.14
324.11
188.12
325.47
540.76
24.34
4.87
283.42
458.26194.59
1521.10
1472.29841.85
19.10
27.52
526.59
31.2341.1117.6117.47
Table 3. NAS Parallel Benchmarks – Class A
Class B Workload
DT-12 PowerEdge 2650
8
N/A
346.66
489 1249 12
N/AFT
MG
SP
LU
BT
IS
EP
CG
354.70
605.56
365.94
681.74
1029.67
21.15
14.72
172.96
649.08
1203.56
N/A
283.09
208.64
374.46
814.64
29.45
17.22
72.13
N/A
703.20214.50
1466.66417.98
2018.45 858.07
32.97
29.52
357.51
28.33
17.28
191.24
33.2644.2917.5317.78
Table 4. NAS Parallel Benchmarks – Class B
4.3I/O Performance
In order to test sequential I/O performance, we used
bonnie [2]. Bonnie performs a series of tests on a file of
unknown size. It does sequential output using the putc()
stdio macro, writes blocks using the write() command, and
rewrites blocks of size 16 KB (reads the blocks into mem-
ory, dirties them, and writes them back). It does sequential
input using getc() and the read() command. The test results
can be significantly affected by the amount of system mem-
ory available since writes are bufferedon most file systems.
Both of our test platforms used ext3, which certainly
buffers write requests whenever memory is available. We
also used the largest file size possible on the local file sys-
tem (2047 MB) in our tests. Since this is a sequential I/O
test, the DT-12 can only use a single processor and a single
memory (1 GB) for file system buffering. The PowerEdge
2650 has both its processors available and 1.5 GB of mem-
ory for I/O buffering. Because the file size is 2047 MB,
moreofthe file I/O canbe bufferedonthe PowerEdge2650,
resulting in higher I/O performance (because it is mostly
writing to memory and not to the actual hard disk). For the
DT-12, we also tested I/O access to a node’s local storage
through ext3 and to remote storage on another node using
NFS. On the PowerEdge 2650, testing NFS performance
does not make sense as both processors have access to their
local disks.
The results for sequential output and sequential input are
shown in Figure 3. As expected, there are certain cases, for
instance, the block write and the block read, that are much
better for the PowerEdge 2650 due to the increased buffer-
ing capabilities from having more memory per processor.
When bonnie rewrites data through NFS on the DT-12, we
see that its performance really suffers due to fetching data
over the network and rewriting it.
We took our parallel I/O test, mpi-io-test, from the
PVFS2 test suite. mpi-io-test simply writes 16 MB per
processor into a shared file and reads it back through the
MPICH2 ROMIO [13] interface. We set-up the PVFS2 file
system [10], a next-generationparallel file system forLinux
clusters, on the nodes to attain the high possible bandwidth.
For the DT-12 platform, we configured all 12 nodes as I/O
servers, with one node doubling as the metadata server. For
thePowerEdge2650platform,weconfiguredtwoprocesses
as I/O servers, with one doubling as the metadata server.
5