Runahead Threads to improve SMT performance.

Full text

Page 1

Runahead Threads to Improve SMT Performance
Tanaus´ u Ram´ ırez1, Alex Pajuelo1, Oliverio J. Santana2, Mateo Valero1,3
1Universitat Polit` ecnica de Catalunya, Spain. {tramirez,mpajuelo,mateo}@ac.upc.edu.
2Universidad de Las Palmas de Gran Canaria, Spain. ojsantana@dis.ulpgc.es
3Barcelona Supercomputing Center, Spain.
Abstract
In this paper, we propose Runahead Threads (RaT)
as a valuable solution for both reducing resource con-
tention and exploiting memory-level parallelism in Si-
multaneous Multithreaded (SMT) processors.
technique converts a resource intensive memory-bound
thread to a speculative light thread under long-latency
blocking memory operations. These speculative threads
prefetch data and instructions with minimal resources,
reducing critical resource conflicts between threads.
We compare an SMT architecture using RaT to both
state-of-the-art static fetch policies and dynamic re-
source control policies.In terms of throughput and
fairness, our results show that RaT performs better
than any other policy. The proposed mechanism im-
proves average throughput by 37% regarding previous
static fetch policies and by 28% compared to previous
dynamic resource scheduling mechanisms.
improves fairness by 36% and 30% respectively. In ad-
dition, the proposed mechanism permits register file size
reduction of up to 60% in a SMT processor without per-
formance degradation.
Our
RaT also
1. Introduction
Simultaneous Multithreading (SMT) [18][21] is a
modern architectural design based on executing multi-
ple instructions from multiple threads at the same time.
This paradigm focuses on sharing different processor
resources to overlap execution of different threads to
enhance performance. However, in SMT environments,
threads not only share important resources, but also
compete for them in processor core. The different char-
acteristics and requirements of every thread can unbal-
ance resource allocation. Some threads will consume
more resources than others, degrading overall perfor-
mance seriously and hindering the benefit of multi-
threaded execution.
To overcome this situation, different fetch policies
and resource schedulers have been proposed. A fetch
policy decides which threads can feed the processor
with new instructions to exploit available resources.
A resource scheduler controls the resource allocation
among threads, trying to avoid resource monopoliza-
tion.These resource control policies stall or flush
threads under determined conditions to prevent re-
source overuse.
The worst case happens when threads with poor
cache behavior (memory-bound threads) are executed.
A memory-bound thread can block the reorder buffer,
because following instructions can neither issue nor
commit while waiting for long-latency memory oper-
ations. Besides, a lot of critical resources may have
already been assigned to this thread, starving other
threads of required resources and preventing their for-
ward progress. This effect would likely lead to global
performance degradation in SMT processors.
However, if we stall threads during memory inten-
sive periods, they will advance too slowly.
quently, the available memory-level parallelism is not
exploited, failing to reach the potential performance
achievable.Current resource policies do control ac-
tions, either stalling or flushing threads, which harm
performance opportunities of memory-bound threads
to unfairly benefit fast threads. Also, these policies can
sometimes produce a resource under-utilization situa-
tion, preventing a stalled thread from using resources
that no other thread requires.
In this paper, we propose Runahead Threads
(RaT) to exploit this memory-level parallelism while
reducing resource contention in SMT processors.
Runahead[5][11] execution is a mechanism whose goal
is to bring speculative data and instructions into the
caches.We propose a new utilization of the Runa-
head mechanism on SMT processors as a different, fair
memory-aware fetch policy to improve performance of
memory-bound threads without harming ILP threads.
Our technique applies Runahead execution to any
running thread when a long-latency load is pending.
Thus, when a thread undergoes a long-latency load,
it turns into a runahead thread.
into a speculative light mode.
head thread, this thread uses the different resources
Conse-
That is, it enters
While being a runa-

Page 2

during short time without limiting the available re-
sources for other threads. At the same time, the is-
sued prefetches increase the memory-level parallelism,
improving its own performance. In other words, our
proposal transforms an eager resource thread into a
light-consumer thread with fast instruction stream ex-
ecution. With this ability, RaT allows memory-bound
threads to advance speculatively, instead of stalling the
thread, doing beneficial work without disturbing the
other threads.
Our evaluation shows that RaT improves through-
put for different kinds of workloads with respect to
previous I-fetch polices and dynamic resource schedul-
ing techniques. This improvement is especially high
for memory-bound workloads, which performs, on av-
erage, 83% and 65% better respectively. In addition,
this throughput improvement is achieved with a signif-
icant overall balance between performance and fairness
(70% and 56%) in a power-efficient way (better ED2).
Finally, we show that the presented mechanism allows
to reduce the size of the register file (up to 60%) of an
SMT processor without penalizing performance.
The rest of this paper is organized as follows. Firstly,
Section 2 discusses previous related work. We describe
Runahead Threads in detail in Section 3. Section 4
describes our experimental environment and simulation
tool. We evaluate RaT with regard to state-of-the art
fetch and resource allocation policies in Section 5. In
Section 6 we analyze the sources of improvement and
the benefits of RaT. Finally, concluding remarks are
given in Section 7.
2. Related Work
In the context of SMT processors, several specula-
tive multithreading approaches try to exploit thread
level parallelism by speculatively spawning threads:
TME [20], SSMT [2] and DDMT [12]. Most of these
pre-computation or pre-execution techniques directly
execute a subset of the original program on separate
processor contexts to speed up the main computation
thread. These additional threads are commonly called
helper or assisted threads. Recent proposals [4][22] dy-
namically construct code slices (p-slices) via hardware
to execute in helper threads (p-threads) which perform
pre-computation and prefetching. These techniques re-
quire the construction of efficient p-slices and the in-
sertion of special instructions in the code.
Runahead Threads share some aspects with these
pre-execution techniques. However, they use a differ-
ent approach. They try to accelerate single running
threads by spawning helper threads to pre-execute a
shortened version of the program. Then, they employ
several contexts and processor resources to improve
the performance of that single main thread. Unfortu-
nately, spawning separate threads and communication
between them with the main thread can be complex
and detrimental for performance.
On the contrary, our mechanism does not spawn
new threads nor insert specific instructions. Our ap-
proach uses the same thread context to take benefit of
long-latency useless memory access periods. In these
periods, we apply Runahead Threads (RaT) to both
exploit the memory-level parallelism and alleviate re-
source contention among threads. That is, we switch
the critical thread into a light speculative thread that
uses as few resources as possible to improve its perfor-
mance.
SMT processors in literature include fetch policies
and resource allocation schedulers that work together
to alleviate resource contention. Initial fetch policies,
like Round Robin [18] and ICOUNT [18], only deter-
mine which threads feed the processor pipeline and
which are left out, assigning different priority to each
thread. Several techniques built on top of ICOUNT
were proposed later to prevent threads blocked by
memory operations from monopolize the shared re-
sources. STALL [17] detects that a thread has a pend-
ing L2 miss and stops fetching further instructions.
The allocated resources are held until the L2 miss is
solved. FLUSH [17] minimizes this problem by flush-
ing the instructions of a thread during a long-latency
memory operation. The victim thread de-allocates all
of its resources, making them available to other execut-
ing threads at the cost of increasing its re-start latency.
These static policies never control the per-thread re-
source utilization. They only make decisions based on
static criteria or events to release resources. Therefore,
even if they try to prevent resource monopolization,
they can sometimes cause resource under-utilization if
other threads do not require the deallocated resource.
Recently, some dynamic resource control policies have
been proposed. DCRA [1] directly monitors the us-
age of resources by each thread, trying to guarantee
that all threads get a fair amount of the critical shared
resources.
Hill Climbing [3], instead of monitoring
the resource indicators, varies the resource allocation
of multiple threads by using the gradient descent al-
gorithm to improve throughput. In both techniques,
when a thread exceeds its assigned resource sharing, it
is stalled until that utilization decreases.
The MLP-aware fetch policy presented in [15] is a re-
cent proposal that is related to our work. However, un-
like our technique, it executes only some extra instruc-
tions indicated by a memory-level parallelism (MLP)
predictor. After that, it stalls or flushes the thread.
The number of speculative instructions executed is lim-
ited by the hardware setup of the MLP predictor (e.g.
the long-latency shift register size). This limitation re-
duces the opportunities to improve performance, since
not all distant MLP can be exploited.

Page 3

Finally, a recent work [13] in the SMT context pro-
poses a mechanism to release physical registers early
belonging to those instructions that are independent of
an L2 cache miss. The basis of this mechanism consists
of traversing the ROB several times to identify which
registers can be early deallocated (those that are not
dependent on the L2-miss load). However, even if the
idea is simple, the hardware overhead of the proposed
mechanism is not when compared to our proposal.
3. Introducing Runahead Threads
Runahead execution is a well-known mechanism
whose goal is to bring speculative data and instruc-
tions into the caches. It was first proposed for in-order
processors [5] to improve the data cache performance.
It was later extended for out-of-order processor as a
simple alternative to large instruction windows [11]. In
this sense, Runahead consists of avoiding the blockage
of the instruction window due to long-latency opera-
tions (eg. a load that misses in the L2-cache). In-
stead, the processor continues executing instructions
speculatively, trying to follow the most likely program
path until the load that triggered the runahead mode
is resolved. The runahead benefit comes from the pre-
execution of these speculative instructions which im-
proves the data and instruction cache efficiency.
3.1. Runahead Operation
Runahead execution prevents the reorder buffer
(ROB) from stalling on long-latency memory opera-
tions by executing speculative instructions. When a
memory operation that missed in the L2 cache reaches
the head of the ROB, a checkpoint of the architectural
state is taken. After that, the processor assigns an in-
valid or bogus value to the destination register of the
memory instruction that caused the L2 miss and enters
in runahead mode. During runahead mode, the proces-
sor continues speculatively executing instructions and
pseudo-retiring them out of the instruction window.
All the instructions that operate over the invalid value
will be considered invalid. The propagation of this in-
valid state is made using an invalid bit (INV) associ-
ated with each physical register. These invalid instruc-
tions are folded (not executed) once they are detected
as invalid. The instructions that do not depend on an
invalid value are executed as normal, except that they
do not update the architectural registers and memory
state.
Once the memory access that started runahead
mode is resolved, the processor rolls back to the ini-
tial checkpoint and resumes normal execution. As a
consequence, all the speculative work done by the pro-
cessor is discarded. Nevertheless, this execution is not
completely useless since Runahead execution will gen-
erate useful data and instruction prefetches, improving
the behavior of the memory hierarchy during real exe-
cution.
3.2. Runahead Threads
Resource conflicts are an important drawback for
SMT processor performance. Our idea is to use Runa-
head Threads to transform a resource intensive thread
(memory-bound thread) into a light-consumer thread
with fast instruction stream execution.
threads to go forward, doing speculative prefetches
without limiting the available resources for other
threads.
When a thread is turned into a Runahead Thread,
the invalid instructions do not use processor resources
since they are pseudo-retired immediately. Other long-
latency loads are also invalidated just like the load that
started the runahead mode, performing the memory
access only as a prefetch.
structions executed in the runahead thread are usu-
ally short-latency instructions that use different re-
sources for short periods of time. Therefore, Runahead
Threads are much less aggressive than normal threads
with the valuable processor resources, allocating and
deallocating them in short periods of time. Besides,
the issued prefetches in runahead increase the memory-
level parallelism of threads.
allow the threads to do useful processing instead of
stalling for several cycles due to resource contention.
This allows
The rest of the valid in-
So, Runahead Threads
3.3. Implementation details
The adaptation of runahead to SMT scenarios is not
straightforward. Now, we describe some relevant de-
sign aspects related to Runahead Threads.
- Checkpoints. Each thread context usually has its
own architectural registers per register file. To cor-
rectly recover the architectural state, each thread only
needs to checkpoint the contents of its architectural
registers. A copy of the full physical register file is un-
necessary. Otherwise, the checkpoint would include the
register information of all threads, taking long time.
- Register control. A runahead thread also needs
the INV bit vector to identify the validity of computed
values because, in case of an invalid one, its value is
unimportant. Then, to adapt the runahead operation
to a multithreaded environment, each thread has its
own INV bit vector to track the propagation of register
invalidations. An instruction with an invalid operand
is not executed and when it reaches the commit stage,
it is pseudo-retired in program order. If it is a valid
instruction, it updates its physical destination register
and pseudo-retires. So, when a physical register is in-
valid (INV bit set to 1) this can be freed and used for
the rest of the threads.

Page 4

-Runahead cache.
the runahead cache to provide communication of data
and invalid status between runahead loads and stores.
Based on this information, some loads dependent on
stores can be identified as valid or invalid. Neverthe-
less, there are some cases in which this memory depen-
dency cannot be identified. For example, a store that
has an invalid effective address cannot save its status
or data in the runahead cache.
From the SMT point of view, using a runahead cache
can be expensive in terms of hardware. The runahead
cache needs to be large to avoid line contention among
threads. Likewise, it is necessary to include a new iden-
tification tag for each thread to distinguish the block
owner. We measure the performance with and with-
out the runahead cache to consider the need to include
it in the RaT proposal and we found that using the
runahead cache does not have significant impact on
performance in our SMT model1. Based on this result
and the fact that a runahead cache implies the use of
more area in the SMT core, we decide not to use it in
our RaT implementation. The functional difference is
that some loads dependent on previous retired stores
use stale values for speculative memory accesses, but it
just affects Runahead execution, i.e., it does not affect
correct program execution.
-Floating-point resources. Runahead Threads im-
prove the performance of the SMT processor mainly
due to the pre-execution of memory operations. Gen-
erally, the computation of the address for memory ac-
cesses involves a base register plus an offset. This is
an integer arithmetic operation, so floating-point (FP)
instructions are not needed to compute the effective ad-
dress. According to this observation, we can decrease
the resource demand of Runahead Threads by avoiding
the execution of FP instructions in Runahead Threads.
This modification was considered for runahead ex-
ecution in out-of-order processors [10]. We apply it
here again for an additional benefit in the SMT envi-
ronment. If a runahead thread does not execute FP in-
structions, it does not need the floating-point resources
of the SMT processor. So, once an instruction is de-
tected to be an FP operation in the decode stage, it
is invalidated and directly proceeds to pseudo-commit.
With this modification, FP instructions in a runahead
thread do not use any processor resources after they
are decoded. Therefore, the FP issue queue, the FP
functional units, and the FP physical register file are
not used by most FP runahead instructions. The ex-
ceptions are FP loads and stores, which are treated as
prefetch instructions because their effective addresses
are obtained in the integer pipeline.
Mutlu et al.[11] introduce
1In [11], the performance deviation without RA cache in
SPEC2000 is also very small for a single-threaded out-of-order
processor.
- Synchronization. Finally, an important issue in the
context of SMT processors is that there can be both in-
dependent and parallel programs. The latter normally
uses a scheme that allows threads to synchronize each
other within the processor. The basic mechanism re-
lies on block, acquire, and release instructions to per-
form thread synchronization. In the case that a parallel
thread switches to a runahead thread, these instruc-
tions are ignored. The instructions inside the critical
section are speculatively executed but do not modify
program state avoiding data inconsistency among par-
allel threads.
4. Experimental Framework
Our simulation environment is based on an SMT
execution-driven simulator derived from SMTSIM [16].
We have extended the simulator to support simulation
checkpoints and a more precise memory hierarchy. We
have implemented Runahead Threads in this simula-
tor, as well as other fetch and resource scheduling tech-
niques for comparison purposes.
Current SMT models use dynamic resource parti-
tioning, which allows threads to improve their perfor-
mance by allocating idle shared resources. This can-
not be done in the case of a statically partitioned or
monolithic SMT design, since each context has a fixed
resource pool assigned. Thus, statically partitioned de-
signs lead to a lack of flexibility far from the SMT ideal.
In our SMT model, we use a complete resource sharing
organization to benefit from the dynamic design ad-
vantages. The threads coexist in the different proces-
sor stages, sharing the issue queues, the reorder buffer
(ROB), the physical registers, the functional units, and
the caches. Table 1 lists the main configuration param-
eters of this simulated SMT processor.
Table 1. SMT processor baseline configuration
Processor core
Processor depth
Processor width
Reorder buffer size
INT/FP registers
INT/FP/LS issue queues
INT/FP/LdSt units
Branch predictor
Memory subsystem
Icache
Dcache
L2 Cache
Caches line size
Main memory latency
10 stages
8 way
512 shared entries
320 / 320
64 / 64 / 64
6 / 3 / 4
Perceptron
64 KB, 4-way, 1 cyc pipelined
64 KB, 4-way, 3 cyc latency
1 MB, 8-way, 20 cyc latency
64 bytes
400 cycles
As we can observe, the simulated processor uses a
shared ROB for all the hardware threads.
separate ROB per thread would probably require less
hardware complexity to implement Runahead Threads.
Using a

Page 5

Table 2. SMT simulation workload classification
MEM2ILP4
applu,art apsi,eon,fma3d,gcc
art,mcfapsi,eon,gzip,vortex
art,twolfapsi,gap,wupwise,perl
art,vprcrafty,fma3d,apsi,vortex
equake,swimfma3d,gcc,gzip,vortex
mcf,twolf gzip,bzip2,eon,gcc
parser,mcfmesa,gzip,fma3d,bzip2
swim,mcf wupwise,gcc,mgrid,galgel
swim,vpr
twolf,swim
ILP2
apsi,eon
apsi,gcc
bzip2,vortex
fma3d,gcc
fma3d,mesa
gcc,mgrid
gzip,bzip2
gzip,vortex
mgrid,galgel
wupwise,gcc
MIX2
applu,vortex
art,gzip
bzip2,mcf
equake,bzip2
galgel,equake
lucas,crafty
mcf,eon
swim,mgrid
twolf,apsi
wupwise,twolf
MIX4MEM4
ammp,applu,apsi,eon
art,gap,twolf,crafty
art,mcf,fma3d,gcc
gzip,twolf,bzip2,mcf
lucas,crafty,equake,bzip2
mcf,mesa,lucas,gzip
swim,fma3d,vpr,bzip2
swim,twolf,gzip,vortex
art,mcf,swim,twolf
art,mcf,vpr,swim
art,twolf,equake,mcf
equake,parser,mcf,lucas
equake,vpr,applu,twolf
mcf,twolf,vpr,parser
parser,applu,swim,twolf
swim,applu,art,mcf
However, we have chosen the shared ROB design to
expose the mechanism to any possible critical resource
contention. The additional complexity mainly lies in
selectively squashing the speculative instructions once
the runahead mode finishes. Nevertheless, this proce-
dure is already implemented in an SMT processor with
a shared ROB to recover from branch mispredictions.
The experiments were performed with workloads
created from the SPEC 2000 benchmark suite.
benchmarks were compiled on an Alpha AXP-21264
using the Compaq C/C++ compiler with the -O3 opti-
mization level to obtain Alpha standard binaries. For
each benchmark, we select an interval of 300 million
instructions representative of entire program execution
using the reference input set. To identify the most rep-
resentative simulation point, we have analyzed the dis-
tribution of basic block execution using SimPoint[14].
Measurements are then taken using the FAME[19] eval-
uation methodology. FAME re-executes all traces in a
multithreaded workload until all of them are fairly rep-
resented in the final measurements.
To create the multithreaded workloads, we consider
only workloads composed of 2 or 4 threads. Several
studies [6][8] have shown that SMT performance sat-
urates or even degrades for workloads with more than
4 threads. We characterize the benchmarks based on
the L2 cache miss rate of each program simulated in a
single-threaded processor. Next, we group them into
three types of workloads: high instruction-level paral-
lelism threads (ILP), memory-bound threads (MEM),
and a mixture of both (MIX). Table 2 shows our sim-
ulation workloads identified by the number of threads
they contain and the thread types. Note that we choose
large groups of workloads to avoid result deviation due
to the specific behavior of a particular workload.
All
5. Comparative Evaluation
Now, we evaluate the performance and fairness of
RaT compared to previous proposals based on instruc-
tion fetch and resource control policies.
We use two metrics for the evaluation. One is the
performance (IPC) throughput, measured as the aver-
age sum of IPC of all running threads in a workload:
Throughput =
?n
represents
proposed in [9],
i=1IPCMT,i
n
(1)
Theother metricthefairness-
which isperformance balance,
the harmonic mean of IPC speedup of each thread
compared to its single thread performance:
Fairness =
n
IPCST,i
IPCMT,i
?n
i=1
(2)
with IPCMT,iand IPCST,ibeing the IPC for thread
i in multithreaded and single-threaded mode respec-
tively, and n being the number of threads.
5.1. RaT and I-Fetch Policies
An instruction fetch (I-Fetch) policy determines
which thread is going to fetch instructions in a given cy-
cle. We compare two I-fetch schemes for handling long-
latency loads, STALL and FLUSH [17], with RaT. All
of them are compared to the ICOUNT policy, which
we use as the reference baseline.
Figure 1 shows the throughput (a) and the fair-
ness (b) of these techniques for the different workloads.
From the performance point of view, FLUSH outper-
forms STALL, but RaT is clearly ahead of both. In
Figure 1(a), among the three types of workloads, RaT
has the best performance mainly for memory-bound
workloads, namely 83% and 70% better than FLUSH
for 2 and 4 threads respectively. The fact that RaT
exploits the memory-level parallelism during runahead
mode avoids the need for stalling or flushing a thread,
and thus it does not slow down the program progress.
Figure 1(b) compares the fairness of the differ-
ent static techniques evaluated here. In this Figure,
a higher bar is interpreted as better.
achieves the best results in terms of fairness.
though the performance improvement for ILP work-
loads is moderate (10% for ILP4), it is more impressive
for MEM workloads: RaT gets 55% and 63% improve-
ment over FLUSH for 2-thread and 4-thread workloads
respectively. We also observe that the fairness for both
STALL and FLUSH has little deviation, being close
to ICOUNT for all 4-thread workloads. Furthermore,
STALL loses around 10% for MEM workloads.
Again, RaT
Al-