Introducing control-flow inclusion to support pipelining in custom instruction set extensions.

Full text

Page 1

Introducing Control-Flow Inclusion to Support
Pipelining in Custom Instruction Set Extensions
Marcela Zuluaga∗, Theo Kluter†, Philip Brisk†, Nigel Topham∗, Paolo Ienne†
∗University of Edinburgh
School of Informatics, Institute for Computing Systems Architecture, Edinburgh, UK
E-mails: g.m.zuluaga@sms.ed.ac.uk, npt@inf.ed.ac.uk
†Ecole Polytechnique F´ ed´ erale de Lausanne (EPFL)
School of Computer and Communication Sciences, CH-1015 Lausanne, Switzerland
E-mails: {Theo.Kluter, Philip.Brisk, Paolo.Ienne}@epfl.ch
Abstract—Multi-cycle Instruction set extensions (ISE) can be
pipelined in order to increase their throughput; however, typical
program traces seldom contain consecutive calls to the same
ISE that would allow this temporal parallelism. Often, there are
intermittent calls to branch instructions, at a minimum, that
prevent the pipelined execution of subsequent calls to the same
ISE within a loop. What is needed is ISEs that cover an entire
loop body, which can create a stream of repeated calls to the
same ISE during program execution; this, in turn, permits the
use of hardware pipelining. To address this concern, we introduce
a new type of ISE that borrows ideas from zero-overhead loop
instructions to permit pipelined execution of loops. To further
expose instruction-level parallelism, the ISE supports loops whose
bodies form hyperblocks, which are regions of program control
flow that have multiple exits (including loop iterations and break
points within loops). These ISEs broaden the scope of instruction-
level parallelism and obtain higher speed ups compared to
traditional ISEs, primarily through pipelining, the exploitation
of spatial parallelism, and reducing the overhead of control flow
statements and branches.
I. INTRODUCTION
The market for embedded processors is driven by stringent
requirements, namely cost, time-to-market, power/energy con-
sumption, and performance. Processor architects must begin
with knowledge of the application domain in order to meet
such demands; however, design automation can further help
to meet time and cost constraints.
Numerous design methodologies for application-specific
instruction set processors (ASIPs) have been proposed. There
are two general approaches: loosely coupled external copro-
cessors, such as loop accelerators, and customizations to the
instruction set architecture (ISA), which permits application-
specific tuning of the instruction set, cache size, register file
size, branch predictors, and other microarchitectural features.
Commercial processors that feature application-specific en-
hancements include the Xtensa by Tensilica [10] and ARC-
tangent by ARC [1].
Much research in recent years has focused on ASIP design
automation, e.g., to find the best partition of an application
into hardware, either as co-processors or custom instruction
set extensions (ISEs). A co-processor can provide substantial
acceleration, but requires significant area and often has a large
communication overhead. At present, the designer is most
often responsible for partitioning the application; however,
once the partition is defined, advanced behavioral synthesis
methods can automatically generate the accelerators.
ISEs, on the other hand, offer speedups at a finer granularity.
Unlike coprocessors, which are loosely coupled, ISEs are
tightly coupled with the processor pipeline, and can exchange
scalar data with the processor’s register file every cycle. ISEs
have evolved to the point where they have their own local
memories [3], and, as a consequence, can achieve speedups
comparable to co-processors.
This paper presents an innovative method to create multi-
cycle ISEs that are executed as hardware pipelines that com-
prise complete loops, including those with irregular control
flow. This approach is justifiable because most programs spend
the bulk of their runtime in a few deeply nested loops, and
this is particularly true in embedded applications. Our ISEs for
loop acceleration are based on zero-overheadloop instructions;
to increase instruction-level parallelism, our method supports
hyperblocks, which have multiple exits. We support this fea-
ture to facilitate loops that have infrequently executed regions
that contain operations, such as function calls, that present
ISEs cannot support in hardware.
To evaluate the effectiveness of our techniques, we extend
the OpenRISC processor with our ISEs, and execute the
resulting system on an FPGA as a soft processor. This permits
us to measure, rather than estimate, the performance gain
of our approach in comparison to existing techniques that
employ smaller ISEs that cannot affect control flow. Existing
ISEs cannot modify the program counter, therefore they must
have a single entry and a single exit; they can execute the
computations within a single loop iteration, but they cannot
execute several iterations at once (let alone the entire loop);
the most important drawback here is that existing ISEs cannot
pipeline the loop.
A detailed case study of the JPEG application shows that
our method achieves a speedup of 3.1× over pure software
execution; in contrast, the most sophisticated ISEs that exist
prior to our work, which can access local memories but with-
out pipelining or support for hyperblocks, achieve a speedup
114
978-1-4244-4938-5/09/$25.00 c ?2009 IEEE

Page 2

???
???
?????????? ????????
???
?????????
???????
???????
?????????????????
??????????????????
????????
????????
?????????????????????????
?????????????????????????????????????????
???
?????????
???????
??????
???
?????????
???????
??????
???
?
?
??
?????????
???? ??????????????
?
?
?????????
???????
??????
?????????
???????
??????
?
?
???
Fig. 1.
identified using traditional techniques. This instruction takes 4 clock-cycles to complete in the AFU. Assuming that the ISE is a non-blocking instruction, i.e.
the operations following it can be issued before its completion, the computations in the AFU cannot be overlapped in order to take advantage of a pipelined
AFU. (b) All operations in the basic block are included in one ISE, and the AFU issues every iteration. Thus, operations in the AFU can overlap in time,
yielding a speedup of at least 2.9× on the loop execution by taking advantage of a pipelined AFU.
(a) A basic block extracted from the control flow graph of the DCT kernel in the JPEG application. In this basic block, a multi-cycle ISE is
of 2.2× over software.
II. PIPELINING ISES
Early ISEs were designed to interface only with a pro-
cessor’s register file; in a single cycle, all inputs were read,
the computation was performed, and all outputs were written.
These restrictions were stringent, because the typical register
file in a RISC processor has two read ports and one write port.
Limited I/O has been a major problem to overcome, and sev-
eral solutions have been presented to address this concern [3],
[7], [13], [16], [24]. Increasing the data bandwidth to and from
ISEs facilitates the creation of larger and larger ISEs, which
increases parallelism and application performance. State-of-
the-art ISEs are multi-cycle, may read their data from the
memory subsystem in addition to the register file and processor
pipeline, and execute synchronously with the main processor
clock.
These multi-cycle ISEs can be pipelined in order to in-
crease their throughput. Pipelining divides the circuit into
several execution stages, allowing it to operate concurrently
on different inputs. This permits the simultaneous exploitation
of both spatial and temporal parallelism in order to increase
throughput. To exploit a hardware pipeline, several ISEs must
be emitted in consecutive cycles; however, typical program
streams rarely contain consecutive invocations of the same ISE
that would permit this type of execution. This requires the use
of ISEs that cover entire loop bodies; prior ISE identification
methods, e.g., [23], are insufficient, as they cannot include
memory accesses (loads and stores), branch instructions that
modify the program counter of the base processor, or function
calls. Historically, only the base processor can execute these
types of operations.
Biswas et al. [3] and Kluter et al. [17] introduced methods
to facilitate architecturally visible storage. Under this scheme,
ISE data can be communicated through a coherent scratchpad
memory in addition to the register file. Direct Memory Access
(DMA) operations are required to move data between the
scratchpad memory and off-chip RAM. Load and store op-
erations that access the scratchpad can be included in the ISE.
This significantly increases I/O bandwidth and the speedups
achievable through ISEs.
The hardware realization of a specific ISE is called an
Application-specific Functional Unit (AFU). As an example,
consider the DCT kernel from JPEG encoding, as shown in
Fig. 1. The body of the main loop of the kernel is identified
as an ISE. When 8 inputs are available, the AFU requires
4 cycles. As there are no loop-carried dependencies, this
instruction can be pipelined, yielding a speedup of 2.9× for
this kernel.
Unfortunately, there is one limiting factor: several opera-
tions relating to the loop itself must be done in software. In
particular, the loop counter must be incremented, compared
with the maximum loop count, and a conditional branch that
determines whether the loop continues, must all execute in
software. This will require that the processor issues at least
three instructions per iteration to facilitate the loop; this, in
effect, cancels any benefits from the pipelined AFU.
As shown in the time table in Fig. 1(a), the computations in
the AFU cannot be overlapped in order to take advantage of
a pipelined AFU. The shaded squares account for the number
of cycles taken by the execution stage of each instruction.
Following the ISE, the processor issues the three consecutive
instructions. When the branch is taken, the processor issues
another ISE. At this point, the AFU has already finished the
2009 IEEE 7th Symposium on Application Specific Processors (SASP)
115

Page 3

computations of the previous instruction.
This example assumes that the ISE is a non-blocking
instruction, i.e. the operations following it can be issued before
its completion. This type of overlap is feasible for long-latency
ISEs, but not for single-cycle ones. If the ISE is a blocking
instruction, then the processor must wait for the AFU to
finish before it can issue another instruction, which prohibits
concurrent execution of software instructions with the ISE.
Zero Overhead Loops
To address this concern, hardware pipelined ISEs can be
implemented in a style that is similar in principle to zero-
overhead loop instructions. The three software operations
described above are now integrated into a single complex
instruction that controls the iteration of the loop. The ISE
implementing the pipelined loop body executes continuously
until the breaking condition is met. By executing these oper-
ations as a zero overhead loop, the impediments to pipelined
ISE execution, as stated above, are mostly eliminated. This
permits ISEs to issue automatically every clock cycle, which,
in turn, permits the use of every stage of the pipeline during
loop iterations. This situation is shown in Fig. 1(b), where
operations in the AFU can overlap in time, reducing the
execution time of the loop by taking advantage of a pipelined
AFU.
III. ALLOWING LOOP BODIES WITH MULTIPLE EXITS
Significant performance improvements can be obtained by
pipelining critical loops. However, loops often contain struc-
tures that cannot be included in a single ISE without intro-
ducing control dependencies. These structures include multiple
control flow paths, multiple exits, inner loops and calls to fun-
tions that cannot be inlined. In these cases, unimportant paths
with high resource usage can prohibit the optimization of the
execution of more important paths. To mitigate this problem
and further expose instruction-level parallelism, we propose
ISEs that support loops whose bodies form hyperblocks [21].
A hyperblock is a single-entry, multiple-exit region of the
control flow of a program, with no internal join points and no
loops. Hyperblocks support predicated execution, which has
already been considered in the field of custom instructions [4].
The use of predication increases the size of regions that
correspond to a single path of control flow, which, in turn,
increases the likelihood of finding instruction-levelparallelism.
Unfortunately, predication does not always remove all control
flow disruptions, and many blocks that can be predicated are
poor candidates for custom intructions. Additionally, unbal-
anced branches may be costly to predicate, as if-conversion
always executes both sides of a branch. Moreover, one side
of the branch may contain forbidden operations, such as a
function call, that cannot become part of an ISE.
To construct hyperblocks, applications are profiled to iden-
tify hot loops and determine which branches are taken most
frequently; heuristics are then applied to select which ba-
sic blocks are consolidated into a hyperblock. Furthermore,
techniques such as loop unrolling, tail duplication, and loop
???
???
????
???
????????????????? ??????
????
?
?
?
????????????
????????????
????????
????
???????????????????????
????
?
Fig. 2.
executed 17% of the time during profiling, it can be excluded. This allows
the formation of a loop ISE with multiple exit points in order to optimize the
execution of the most taken trace in the loop.
The detection of a hyperblock. As one of the branches is only
peeling can assist hyperblock formation without altering the
correctness of program execution.
As an example, Fig. 2 shows the entropy encoding kernel
from JPEG. Traditional techniques can find an ISE in the first
basic block of the control flow segment. Unfortunately, there
is a control flow disruption that prevents the complete loop
from being converted into an ISE. Profiling indicates that the
branch leading to a function call is rarely executed. Fig. 2
shows that a multi-exit hyperblock can be formed so that the
entire loop body, except for the rarely executed branch, can
be implemented in custom hardware. For the common case,
when the disruption does not occur, the entire loop body will
benefit from the speedup achieved through the ISE.
The new loop instruction will contain the operations corre-
sponding to two conditions, one to exit the loop, and another
one to execute the branch that has been excluded.
IV. IDENTIFICATION OF LOOP INSTRUCTIONS
The process for identifying loop ISEs is described as
follows:
1) The application is profiled and hyperblocks are formed.
Hyperblocks that contain operations that can always be
included in an ISE trivially become loop ISEs.
2) Hyperblocks that contain at least one operation that
is forbidden from inclusion of an ISE are deformed,
i.e., the hyperblock is discarded and replaced with the
original control flow constructs.
3) Traditional ISE identification [23] is carried out in the
remaining basic blocks and hyperblocks in the appli-
cation. Methods that can exploit architecturally visible
storage [3], [17] are used:
a) Data structures (arrays) accessed in the basic block
or hyperblock are identified using existing disam-
biguation techniques. All load/store instructions to
116
2009 IEEE 7th Symposium on Application Specific Processors (SASP)

Page 4

data structures that cannot be disambiguated are
forbidden from inclusion in an ISE.
b) DMA transfers are placed in the most profitable
positions.
4) Instances of ISEs are inserted in the program at the
appropriate locations.
V. AFU IMPLEMENTATION
Fig. 3 shows the interface between the AFU and the base
processor. The processor provides control signals to initiate
the ISE, along with read/write interfaces to its register file.
In typical RISC processors, the register file can provide two
inputs and read one output from the AFU every cycle. An AFU
controller, which receives the initialization signals, enables the
pipelined datapath execution and activates the AFU’s local
storage units.
The AFU datapath has its own architecturally visible local
memory. This memory can provide inputs to the ISE and store
its results after the loop executes. DMA transfers data from
the main memory to the local memory for ISE execution, and
copies the data back after the loop terminates, without stalling
the processor. As noted by Kluter et al. [17], this type of
memory access creates coherence problems, as data that is
modified in the AFU local memory and written back to the
memory may be different from the values of the same data that
reside in the data cache. A hardware or software coherence
mechanism needs to be used to prevent the cache to AFU
local memory coherence problem.
Additionally, a local register file is used to store loop-carried
variables. Specialized load and store instructions are included
in the set of ISEs to permit the reading and writing to this
local register file.
The AFU outputs that correspond to loop break conditions
are passed to its control unit, which transfers control back to
the processor. To facilitate correct execution in the presence of
multiple exit points, the AFU also returns to the processor the
address of the correct exit point. A loop ISE has itself as the
default destination, which is achieved by implicitly issuing the
same instruction within the AFU. The remaining destinations
are the other hyperblock exits, e.g., break instructions in
the original program’s control flow, including the eventual
completion of the loop.
The local register file also stores the possible destinations of
the ISE loops. The ISE store instruction is used to load these
values at the beginning of the execution of each loop ISE.
Although this data is all compile-time constant, the register
file would be quite large if it must store every exit address for
every loop ISE. In actuality, the maximum number of local
registers will be the sum of the number of exit point plus the
number of loop-carried variables for the ISE that uses most.
Due to the use of hyperblocks, loop ISEs can be viewed
as a complex multi-target branch instruction. Implicitly, this
suggests that the AFU would need to modify the Program
Counter (PC) in the fetch stage of the base processor pipeline.
Sidestepping this issue is beneficial, as branching out of the
???????
?????????????
??????
??? ?????? ?????
?????????????????
???????????
"!"#?$"%
&"' &" (%' !"
) *?# +? ")
$", "#?-
.?/"!"0"'*?' /"!
.?/1!2*?, /1%3
' !"&(45*6)7
/"42*?&"42*
"!"#?$(%
89,' *? ")
:?;=<
*?,#?!") >? 1,)
@BA?C
DE"?
F
?
F
???
?G?
F
???
???????????
H?I(J K?L2M?JN5KPO
H?I(JK?L2M?J N5K(Q
?
?
?
???
F
????R9S
T9U
?????
F
??V"V"???
??? ?????????
Fig. 3. System overview of an accelerated processor. The Loop Control Unit
enables the pipelined execution of the AFU. Furthermore, it detects the break
condition and sends back the exit addresses (see Fig. 2) stored in the local
register file.
hyperblocks directly would issue the result of the branch to
the branch predictor.
In a sense, the formation of a hyperblock can be viewed as a
type of profile-guided static branch prediction. Consequently,
branches that have been absorbed into the hyperblock, includ-
ing hyperblock exits, are removed from the program and are
no longer issued to the branch predictor; conflicts involving
these branches are eliminated as a consequence.
As hyperblock execution is not driven by the PC, there
is no need to predict branch target addresses for exiting
the hyperblock. Consequently, updating the PC directly on
a hyperblock exit would overwrite meaningful entries in the
predictor itself. In contrast, our approach returns the branch
target address to the processor pipeline as data, permitting the
use of a direct, rather than a conditional branch instruction,
which will not update the predictor.
Initiation Constraints
The Initiation Interval (II), in the context of pipelining,
is the number of cycles the AFU must wait before issuing
the next iteration of the loop. To initiate a new ISE every
clock cycle, the aggregate I/O bandwidth required for its
computations cannot exceed the number of available memory
and register file read and write ports. If these conditions are not
met, then the input and output operations must be scheduled
accordingly. This can be taken as the resource constraints of
the initiation interval of the loop, and can be formalized as
follows:
??inputs
in ports
In other words, ResII is the initiation interval, as determined
solely by resource constraints.
The greatest dependencydistance, in clock cycles, found be-
tween iterations is another constraining factor of the initiation
interval. This is known as recurrence constraint RecII.
ResII = max
?
,
?outputs
out ports
??
(1)
2009 IEEE 7th Symposium on Application Specific Processors (SASP)
117

Page 5

The maximum instruction throughput is the inverse of
stage delay or initiation interval of the AFU. It specifies the
number of clock cycles between the initiations of sequential
instructions into the AFU. It is calculated as follows:
II = max(ResII,RecII)
(2)
Pipeline scheduling
As-Soon-As-Possible (ASAP) scheduling [8] is applied to
the ISE so that the loop break conditions are evaluated at
the earliest possible point; however, the pipelined execution
of several loop iterations at once, technically, is speculative.
Therefore, write operations must not be scheduled before the
break conditions of the previous iterations are resolved. The
alternative is to implement the interface by which the AFU
writes data back to the processor using some form of gated
write buffer, which is beyond the scope of this work. Inputs
and outputs are then scheduled along different pipeline stages
making sure that the number of read and write accesses does
not exceed the number of ports at any time.
VI. CODE EXAMPLE
Fig. 4 shows the high-level code transformations that are
required to facilitate the use of a loop ISE for the JPEG
entropy encoding kernel shown in Fig. 2. Before the loop,
the addresses of the exit points are loaded, as discussed
above, using a dedicated instruction, ISE_STORE. The for
loop is replaced with a LOOP_ISE, effectively the high-
level language equivalent of a zero-overhead loop assembly
instruction. Unlike a traditional zero-overheadloop instruction,
LOOP_ISE returns the destination address of the exit point. A
jump instruction following the loop ISE then transfers control
to the appropriate point of continuation.
Recall that the loop ISE may contain control flow that
exits the ISE, executes some statements in software, and
then re-enters the ISE for the next iteration. To facilitate
software execution, data must be moved from the AFU back
into the processor in advance; this is accomplished with the
ISE_LOAD instruction, which moves variable r back into the
register file. If data in the AFU local memory is required,
then a DMA transfer may be initiated as well. Next, the
software code executes until it finishes. Afterwards, any values
modified by the software code that may be needed by future
iterations of the loop are written back to the AFU. In this
case, r is re-initialized to 0 and sent back to the AFU using
an ISE_STORE instruction.
If the loop terminates, the destination address provided by
the ISE corresponds to the continue_label. In this case,
there is no return to the ISE, so normal software execution
continues after the loop.
VII. EXPERIMENTAL SETUP
We used the JPEG encoding/decoding chain, taken from the
EEMBC benchmark suite [12], to evaluate loop ISEs. JPEG
compression is comprised of three kernels: Discrete Cosine
Transformation (DCT), quantization, and entropy encoding;
???
??????
??????
??????????????????????
??????????
????????????
????
?
?????
??????????????
???
?
????
???????????????
???
??????
?
?
???
????????????????
??????????????
??????????????
??????
??????????
??????????????
?????????????
????????????????
??????
??????????
????????????????
?????????????
??????????????
???????????
???????????
??????????
??????????????
?????????????
????????????????
?????????? ????
???????????????
?????????????
??????????????????????????
??????????????????????????????
???
??????
??????
?????????????????
?????????????????
??????????? ???????????????????
???????????
??????????????????????
??????????? ?????????????????
??????????????
???
?
???
???????????????
???
??????
?????????????????
?????????????????
??????????????? ???
Fig. 4.
transformations needed in order to use a loop ISE.
Code abstraction, taken from the original application, showing the
likewise, JPEG decompression is composed of entropy de-
coding, de-quantization, and Inverse DCT (IDCT). For all
of the experiments, we used a 24-bit RGB-encoded picture
with a resolution of 1024x768 pixels, which is comparable to
the image resolution found in current web-cams and mobile
phone cameras. JPEG was chosen because many of the smaller
kernels in EEMBC contain straightforward loops with no
internal control flow, and therefore do not require hyperblocks.
For the purpose of comparison, we implemented prior
methods for ISE generation that do not encompass full loops.
Our evaluation platform is an OpenRISC processor, with an
interface for custom instructions that is similar in principle to
Altera’s Nios II soft processor.
We compare with ISEs identified by Pozzi et al. [23] which
cannot access data in memory. These type of ISEs can only
interface with the processor’s register file; the register file
of our target processor has two read ports and one write
port, so ISE identification uses these constraints. For this
approach, instruction identification, C code generation and ISE
implementation in VHDL have been done by automated tools.
We also compare with ISEs identified by Biswas et al. [3],
which can access architecturally visible local memory; The
latter was reimplementated including speculative DMA trans-
fer techniques to ensure coherence between the processor’s
data cache and the AFU’s local memory, as described in [17].
For this approach, instruction identification, C code generation
and ISE implementation in VHDL have been done by hand.
The proposed approached can also access architecturally
visible local memory and was also implementated including
speculative DMA transfer techniques as described in [17].
Instruction identification, C code generation and ISE imple-
mentation in VHDL have been done by hand but we hope to
automate this processes in the future. Nevertheless, in order to
make a fair comparison with other methods, we endeavored to
keep generality in our manipulations in order to enable future
automation.
For ISEs enhanced with architecturally visible storage, we
118
2009 IEEE 7th Symposium on Application Specific Processors (SASP)