Adaptive Allocation of Software and Hardware Real-Time Tasks for FPGA-based Embedded Systems

Abstract

Operating systems for reconfigurable devices enable the development of embedded systems where software tasks, running on a CPU, can coexist with hardware tasks running on a reconfigurable hardware device (FPGA). Furthermore, in such systems relocatable tasks can be migrated from software to hardware and viceversa. The combination of high performance and predictability of hardware execution with software flexibility makes such architecture especially suitable to implement high-performance real-time embedded systems. In this work, we first discuss design and scheduling issues for relocatable tasks. We then concentrate on the on-line admission control problem. Task allocation and migration between the CPU and the reconfigurable device is discussed and sufficient feasibility tests are derived. Finally, the effectiveness of our relocation strategy is shown through a series of synthetic simulations.

Full text

Page 1

Adaptive Allocation of Software and Hardware Real-Time Tasks for
FPGA-based Embedded Systems∗
Rodolfo Pellizzoni and Marco Caccamo
Department of Computer Science, University of Illinois at Urbana-Champaign
{rpelliz2,mcaccamo}@uiuc.edu
Abstract
Operating systems for reconfigurable devices enable the
development of embedded systems where software tasks,
runningon a CPU, can coexist with hardware tasks running
onareconfigurablehardwaredevice(FPGA).Furthermore,
in such systems relocatabletasks can be migrated from soft-
ware to hardware and viceversa. The combination of high
performance and predictability of hardware execution with
software flexibility makes such architecture especially suit-
able to implement high-performance real-time embedded
systems. In this work, we first discuss design and schedul-
ing issues for relocatable tasks. We then concentrate on the
on-line admission control problem. Task allocation and mi-
gration between the CPU and the reconfigurable device is
discussedandsufficientfeasibilitytests arederived. Finally,
the effectivenessof our relocationstrategyis shown through
a series of synthetic simulations.
1Introduction
As systems-on-chips (SoCs) become more widely used
due to their improved performance, both in term of speed
and power consuption,reconfigurabledevices and in partic-
ular field-programmablegate arrays (FPGAs) are becoming
more and more popular in the development of embedded
systems where issues such as short time-to-market and up-
datecapabilitiesafterdeploymentarecritical. Recentdevel-
opments in the field of operatingsystems for reconfigurable
devices (OSRD) [18, 25, 26] enable a highly dynamic use
of partially reconfigurable FPGAs, running multiple con-
current circuits (hardware tasks) with full multitasking ca-
pabilities. Furthermore, the introduction of embedded de-
vices comprised of an FPGA and one or possibly several
CPUs permits to run both software and hardware tasks on
the same silicon device, achieving even greater flexibility.
While a lot of work has been done in the design of suit-
able operating system abstractions and in the development
of working prototypes for OSRD, much more remains to
∗This work is supported in part by the NSF grant CCR-0237884, NSF
grant CCR-0325716, and NSF CNS-0509268.
be done to obtain a feasibly usable platform. In particu-
lar, the important topic of real-time resource management
has received little attention. In this work, we first introduce
our vision for a reconfigurable platform that enables relo-
cation (i.e. task migration between software and hardware)
as a way to improve the system ability to cope with dy-
namic workloads. We then propose a novel allocation and
admission control scheme that is able to improve the usage
of system resources while preserving all timing constraints.
In particular, our main contribution is the development of a
relocation scheme with proven feasibility conditions that is
suitable for real-time applications.
The paper is organizedas follows. In Section 2 we intro-
duce oursystem abstraction,discussing its applicabilityand
practical limitations, and we further describe our resource
management scheme. In Section 3 and 4 we present our so-
lutions to the allocation and relocation problem, providing
simulation results in Section 5. Finally, in Section 6 we dis-
cuss related works and in Section 7 we provide concluding
remarks and future work.
2System Model
We consider a system comprised by a general purpose
CPU and a partially Reconfigurable Device (RD), together
with main memory and I/O devices. Modern devices, like
theXilinx Virtex-IIProandVirtex-IVfamilyofFPGA [27],
implementalloftheaboveonasingleconfigurableSoC.An
OSRD is used to manage the entire system; prototypeshave
been proposed in [18, 25, 26].
Tasks can be provided to the system in both a software
anda hardwareconfiguration. Thesoftwareconfigurationis
a traditional software program that runs on the CPU, while
the hardware configuration is implemented as a hardware
circuit on the RD. Codesign tools can be used to generate
both configurations given an initial specification in a high-
level language [10]. Since the RD is partially reconfig-
urable, it is possible to reconfigure a single hardware task
at run-time by downloading its configuration data (known
as a bitstream) without affecting the remaining hardware
configurations. Tasks are relocatable, i.e. they can migrate
from software to hardware and viceversa (relocation is im-
plemented by [18]).
1

Page 2

Tasks can be dynamically activated and terminated. Fur-
thermore,we assume that tasks are subject to real-time con-
straints, i.e. once activated they are periodically executed
and each task instance must terminate before a given dead-
line.
We believe that this architectural model suits a vari-
ety of systems, including Micro unmanned Aerial Vehicles
(MAVs) [9], wearable computing [20] and sensor networks
for complex tracking and surveillance applications [22],
which exhibit characteristics that makes solutions based on
both single or multiple CPU and on fixed hardware unsuit-
able:
Dynamic workload with high computational de-
mands: As an example, in applications based on multi-
ple unmanned aerial vehicles coordinating on a global mis-
sion each vehicle must perform multiple concurrent con-
trol tasks together with complex multi-vehicle coordina-
tion [12], wireless processing and data aggregation, sensor
processing and target tracking and localization. The work-
load is extremely dynamic, depending on both the vehicle
and the mission status. Due to multiple target tracking and
multiple surveillance objectives different tasks are contend-
ing for system resources. Wearable computing and sensor
networksfortrackingapplicationssharesimilar characteris-
tics (dynamictasks with event-basedworkloadsurges),plus
due to their long deployment time software updates are of-
ten necessary. Due to these intrinsic dynamic aspects, sta-
tic task allocation on fixed hardware is hardly possible. At
the same time, general purpose processors can not provide
the required level of performance. The proposed model can
constitute a valid solution, combining the flexibility of gen-
eral purpose systems with the performance of hardware so-
lutions; in particular, the advantages of relocation are thor-
oughly discussed in [17, 20].
Energy and cost constraints: All the proposed systems
are severely energy constrained, since they draw power
from either batteries or solar cells; the amount of energy
used for computation is a significant percentage of the to-
tal energy consumption (see [9] for details on MAVs). FP-
GAs have been proven to provide better performance and
to be more power efficient than both general purpose and
application specific processors for a variety of applications
[20]. Furthermore,all the proposedsystems are deployedin
large numbers and are therefore cost-sensitive. The use of
a single SoC including a high performance FPGA can eas-
ily replace a number of discrete chips thus lowering board
complexity and helping reducing costs. Both energy and
cost constraints imply that the available computational re-
sources must be efficiently used, i.e. by maximizing the
amount of computation (tasks) that the system can handle.
Real-time constraints: MAV applications, wearable
computing systems and sensor networks can include criti-
cal monitoring and targeting tasks for which proven delay
bounds must be guaranteed. Furthermore, flight control on
MAVs is implemented through hard-real time tasks.
Following the discussion above, the overall goal of our
resource management scheme can be stated as: maximize
the number of tasks simultaneously running in the system,
while guaranteeing all real-time constraints. Therefore, we
need to provide an admission control test: every time a task
is presented to the system, we run the test to check if the
new task can be admitted while guaranteeingall the already
running tasks. We introduce the details of our management
scheme in Section 2.3, after we discuss some key model
limitations in Section 2.1 and our task model in Section 2.2.
It is important to note that our management goal as-
sumes that, from a computational point of view, the soft-
ware and hardware configurations of a task provide equal
performance. In other applications (for example multime-
dia terminal) it can be more useful to consider hardware
configurationsprovidingbetterservice comparedto the cor-
responding software configurations. In this case the overall
system goal would be to maximize the quality of service
perceived by the user. Although we do not consider such
scenario in this work, we are currently investigating it and
our current results show that the admission control scheme
can be applied unchanged to this further case.
2.1Model Limitation
When concerned with practical implementation of the
proposed abstraction, several limitations of currently avail-
able reconfigurable devices and operating systems need to
be considered. First of all, hardware configurationsmust be
constrained in rectangular areas. Two area model are em-
ployed by current OSDR prototypes. In slotted area model,
the device is divided in a series of slots, each of which has
the same dimensions. Each task is partitioned by means of
suitable design tools in some number of slots, which can be
positioned anywhere on the device. The slotted area model
incurs internal fragmentation: some area on the device can
be wasted if the area occupied by a task is not a multiple of
the defined slot area. The slotted area model is employed
by [17, 18].
In the 1D area model, each task occupies a rectangu-
lar area on the device. The vertical dimension is fixed and
spans the height of the device, while the horizontal dimen-
sion can vary. The 1D area model incurs both internal and
external fragmentation: the total area available on the de-
vice can be greater than the area required by a task, but
placing it can be impossible if the area is divided in smaller
unconnected stripes. The 1D area model is employed by
[25]. A more complex 2D area model is further discussed
in the literature, but to the best of our knowledge, no work-
ing OSRD prototype is able to employ such model. In this
work, we will only consider the slotted area model.
Task communication is also a major issue, and differ-
ent solutions, including buses and packet-switch networks,
have been proposed [25, 15]. Communication is particu-
larly critical in the slotted model since slots pertaining to
the same tasks need strict synchronization. Real-time con-
straints for bus based systems are introduced in [6]. Since
in this work we are mainly concernedwith the management
andrelocationproblem,we will assume thatthe systempro-
videsenoughcommunicationresourcesto meettheneedsof
all tasks and reserve a more thoroughly analysis for future
work.
2

Page 3

Animportantissue regardsthereconfigurationcapability
of the RD. Each time a new task is started on the device, its
bitstream needs to be loaded inside the device’s configura-
tion SRAM using the configuration interface; while down-
loading a bitstream, the area occupied by it can clearly not
be used (other tasks can still run undisturbed). The load
time is proportional to the task area; for modern, large de-
vices,it is notnegligible,in theorderoftens ofmilliseconds
to reconfigure a task that occupies the entire device [23].
This imposes severe constraints on how hardware tasks are
managed. In particular, hardware tasks cannot be sched-
uled like periodic software tasks. Consider the slotted area
model and suppose that hardware tasks are scheduled like
periodictasks, i.e. each hardwaretask is definedby a period
andanexecutiontimeandisperiodicallyactivated. Forsim-
plicity, assume that all tasks occupy only one slot and have
thesame periodpandexecutiontime e. Let Trecbethe time
needed to reconfigure the entire device, and trec=Trec
the time needed to reconfigure a single slot, where A is the
total number of slots. If we serialize slot reconfigurations,
while a slot is reconfigured in trectime all other tasks can
keep running; therefore, we define U as the task utilization
e+trec
p
. It is then easy to see that if we want to keep the de-
vice constantly busy, we need to reconfigure the entire de-
vice
must hold: p ≥Trec
ms [23] and U =1
than 5 Hz. Therefore, in order to reduce the reconfiguration
overhead we will impose that each hardware configuration
executes for the entirety of its period, so that no reconfig-
uration is needed if no new task is activated. This is not
a major limitation, since different synthesis parameters in
toolspermita tradeoffbetweenoccupiedarea andexecution
time. This means that although the hardware configuration
executes longer than the software one, it occupies a much
smaller area that the one needed by an equivalent CPU ded-
icated to running the software configuration.
The last issue regards hardware/software relocation.
While suspending and migrating a software task between
homogeneous CPUs is relatively easy, since the state of a
software task can easily be saved, saving the state of a hard-
ware task is more complex,since it involvessaving the state
of all its internal registers. While this is not technically
impossible [21], it can nevertheless incur in a unbearable
overhead. Instead, a different approachto relocation will be
used. We assume that each task, either in software or hard-
ware configuration, eventually reaches a point in which the
execution of its next periodic instance does not depend on
the state of the task after the completion of its previous in-
stance, i.e. no internal state must be preserved between the
two successive activations. When this point is met, a task
can be relocated at the end of its period. Note, however,
that reconfiguration constraints must be taken into account:
while we can usually safely assume that starting a task on
the CPU takes zero time, this is not true for the RD. There-
fore, the OSRD must first begin loading the task bitstream
into the RD, which can possibly last for multiple task peri-
ods. When the loading operation completes and the end of
A
be
1
Utimes each p seconds, thus the following inequality
U. Supposing a typical time Trec= 50
4, we cannot achieve frequencies greater
a periodis reached,the software configurationis terminated
on the CPU and the hardware configurationis started on the
RD. If a stateless point is never reached, we can add some
additional logic to the task in order to save and restore state
between instance activations. The resulting overheadis still
much lower that permitting to save the state of the task at
any time [16].
2.2Task Model
Each relocatable task τiis defined by a period pi, a rela-
tive deadline Diand two configurations: τs
finedbyanexecutiontimeei, andτh
an area ai. We assume relative deadlines equal to periods,
i.e. ∀i,Di= pi. The execution time of a software configu-
ration can be either a worst-case parameter (for hard tasks)
or average-case parameter (for soft tasks). Furthermore, let
Ui=ei
Hardware configurations have no associated execution
time. Each periodic instance (also called a job) of a hard-
ware configuration runs for the entire period. Since hard-
ware configurationscannot be preempted,they always meet
their deadline as long as configuration changes (relocation)
are only allowed between jobs. The area parameter depends
on the area model of the RD: under the slotted area model,
we denote with A the total number of slots on the RD and
with aithe number of slots occupied by τh
We assume that communication among tasks follows a
synchronous dataflow approach, i.e. all inputs to a job are
made available by the OSRD before the job starts and all
outputs are propagated at the end of the job to subsequent
tasks in the data graph. The dataflow model has several ad-
vantages. First, it enables transparency between hardware
and software configurations since all data can be held in
buffers managed by the operating system. Second, many
commercially available languages and tools for hardware
specificationfollow the dataflowmodel[3, 11]. Third, there
is no need to account for blocking time due to critical sec-
tions during the execution of a task. Finally, the careful
placement of buffers takes care of delays in data propaga-
tion along the communication infrastructure; in particular,
precedence constraints among successive tasks can be re-
moved by buffering one full task period.
Software tasks can be scheduled on the CPU using any
real-time scheduler with proven schedulability bounds and
suitable isolation mechanisms. In this paper we will con-
sider the EDF scheduler [14] in conjunction with the well-
known Constant Bandwidth Server [1]. The CBS provides
isolation between hard and soft tasks so that all jobs of hard
tasks are proven to complete within their deadlines if a fea-
sibility condition is met. For a fixed task set TSof software
tasks, the following is a sufficient and necessary feasibility
condition provided that kernel overhead is included in task
execution times:
?
where U is known as the total software utilization.
i(software), de-
i(hardware),definedby
pibe the task’s software utilization.
i.
U =
τi∈TS
Ui≤ 1,
(1)
3

Page 4

τi
τs
i
τh
i
pi
Di
ei
Ui
ai
A
T
TS
TH
AT = {TS,TH}
UT =?
aT =?
ithtask
ithtask software configuration
ithtask hardware configuration
task period
task relative deadline
software configuration execution time
task utilization
hardware configuration area
RD area
task set
set of tasks in software configuration
set of tasks in hardware configuration
allocation for task set T
total utilization of tasks in T
total utilization of tasks in soft. config.
total area of tasks in T
Table 1. System notation
τi∈TUi
UAT= UTS
τi∈Tai
In the same way, in order to be schedulable on the RD
hardware configurations must meet placement constraints.
Definition 1 (Slotted feasible placement) For the slotted
area model, given a set TH of hardware configurations
scheduled on the RD we say that their placement is feasi-
ble iff:
?
Tasks can dynamically join and leave the system. The
activation time of a task corresponds to the activation of
its first job. The termination time of a task corresponds
to the deadline of its last job. At any time t, T (t) is the
set of currently active tasks.
the set of software tasks running on the CPU at time t
and TH(t) be the set of hardware tasks placed on the RD;
then AT(t) = {TS(t),TH(t)} is the allocation for T (t) iff
TS(t) ∪ TH(t) = T (t). Hence, the allocation of a task set
defines how tasks are partitioned between the CPU and the
RD. T is said to be feasible iff each job of tasks in T gets
executed on either the CPU or the RD and AT(t) results in
both a feasible schedule and a feasible placement.
Table 1 summarizes the notation used throughout this
work.
τi∈TH
ai≤ A.
(2)
Furthermore, let TS(t) be
2.3Management Scheme
The followingoverallmanagementstrategywill be used.
When a task or a groupof tasks arrives in the system, an ad-
mission test is run to determine if it can be admitted. If the
test succeeds, then the task is immediately activated on the
CPU; in fact, loading a hardware configuration on the RD
would delay the activation of the task. After the new task is
activated on the CPU, or whenever a task is terminated, the
system performs a relocation phase. The goal of the relo-
cation phase is to relocate tasks, including the newly admit-
ted one, in order to minimize the total software utilization,
while preservingall feasibility constraints. We feel that this
optimization objective is sensible for multiple reasons:
• Sincenewlyactivatedtasks areadmittedontheCPU to
avoid the RD configuration overhead, minimizing the
CPU utilization maximizes the probability of passing
the admission test.
• Although we only consider relocatable tasks in this
work, real systems would probably also be comprised
ofsoftware-onlytasksthatcannotbeplacedontheRD.
• Although we are only concerned with the admission
control problem in this work, we can envision situ-
ations in which hardware configurations provide ser-
vices with better performance and lower power con-
sumption compared to corresponding software config-
urations.
• The OS needs to run both the admission test and fur-
ther computations to drive the relocation phase and to
load hardware configurations. This added overhead
can be considered as an added utilization term on the
CPU.
We will split the problem as follows. In Section 3, we
discuss the problem of finding an optimal allocation given
a task set subject to the slotted area model, assuming that
no task is already running and therefore no relocation is
required. In the subsequent Section 4 we will see how a
pseudo-optimal solution can be used to drive the relocation
phase. Due to space constraints, theorem proofs are not re-
ported; they can be found in [19].
3 Allocation Problem
Given a task set T of relocatable tasks, the optimal al-
location problem consists in determining the feasible allo-
cation AT that minimizes the total software utilization on
the CPU, supposing that no task is already running in the
system.
The problem can be stated as an integer linear program-
ming optimization problem. Let us introduce for each task
τiin T two indicator variables riand ci. riis set to one if
τiis placed on the RD, while ciis set to one if the task is
scheduled on the CPU. The optimal allocation problem can
then be represented as follows:
Definition 2 (ILP ALLOC) Minimize
ject to the following constraints and the restriction that
variables ri,citakes integer values only:
?
τi∈TciUi, sub-
∀τi∈ T ,ci+ ri= 1
?
∀τi∈ T ,0 ≤ ri≤ 1
∀τi∈ T ,0 ≤ ci≤ 1
(3)
τi∈T
riai≤ A
(4)
(5)
(6)
4

Page 5

Lemma 1 ([19]) Any optimal solution to ILP ALLOC is an
optimal solution for the allocation problem under the slot-
ted area model, supposing that no task is already running.
Now note that since ∀τi∈ T ,ri+ci= 1, min?ciUi=
the ILP ALLOC problem can be restated as the following
equivalent ILP KNAP problem:
min?(1 − ri)Ui =
?Ui− max?riUi. Therefore,
Definition 3 (ILP KNAP) Maximize
the following constraints and the restriction that variables
ritake integer values only:
?
∀τi∈ T ,0 ≤ ri≤ 1
Problem ILP KNAP is in the form of the well-known 0-1
KNAPSACK problem [13], which is known to be NP-hard
in the weak sense. This means that pseudo-polynomial ex-
act algorithms exist for the problem. However, since we are
required to solve the allocation problem at run-time, even
pseudo-polynomial algorithms can be excessively costly.
Furthermore, as we will discuss in Section 4.1, using an
optimal algorithm does not lead to a significant increase in
performance.
We will therefore use the simple greedy algorithm for
0-1 KNAPSACK to obtain a pseudo-optimal solution. The
greedy algorithm works as follows, where R is used as a
helper variable:
?riUi, subject to
τi∈T
riai≤ A
(7)
(8)
• Order all tasks in decreasing order ofUi
A.
ai. Assign R ←
• Starting from the first task τiin the defined order, if
R ≥ aithen set ri← 1,R ← R − ai, else set ri← 0.
Proceed to next task.
Since we need to order the tasks, the complexity of the al-
gorithm is O(N log(N)) where N is the number of tasks
in T . In order to characterize the performance of the algo-
rithm, let LP KNAP be the linear relaxation of ILP KNAP
(obtained by removing the constraints of ri being inte-
ger),OPT(ILP KNAP), OPT(LP KNAP) be the optimal so-
lution to the ILP KNAP and LP KNAP problems respec-
tively, and GREEDY(ILP KNAP) be the greedy solution to
ILP KNAP. Furthermore, let τcbe the critical task, i.e. the
first task, in decreasing order of
the greedy solution. It can be seen that the only difference
between the OPT(LP KNAP) and GREEDY(ILP KNAP)
solutions is that while τcis partially placed on the RD in
the optimal linear solution, the greedy algorithm places it
entirely on the CPU. Therefore, since OPT(LP KNAP) ≥
OPT(ILP KNAP) ≥ GREEDY(ILP KNAP), the following
inequalities hold:
Ui
ai, such that rc = 0 in
OPT(LP KNAP) ≥ GREEDY(ILP KNAP) >
> OPT(LP KNAP) − Uc
(9)
OPT(ILP KNAP) ≥ GREEDY(ILP KNAP) >
> OPT(ILP KNAP) − Uc
Given GREEDY(ILP KNAP), the total CPU utilization
UT can be computed as?
UT ≤ 1. However, if some tasks are already running in the
system, then a relocation phase is needed to reach the new
computed allocation AT. The following section details the
relocation phase and how to combine it with the admission
test.
(10)
τi∈T(1 − ri)Ui=?
τi∈TUi−
GREEDY(ILP KNAP). The task set can then be admitted if
4 Relocation Phase
In this section we discuss how task relocation can be
performed without violating any feasibility constraints for
the slotted area model. We consider a general relocation
problem of the following type: given a task set T , relo-
catable task sets TS→S,TS→H,TH→H,TH→S, and an al-
location A?
want to relocate tasks in order to obtain a new allocation
A??
and TH→H represent the sets of tasks that are kept on the
CPU and RD respectively, while TS→H and TH→Srepre-
sent the sets of tasks that are relocated from the CPU to the
RD and from the RD to the CPU respectively.
RD constraints must be considered when performing re-
location. Consider a simple example in which TH→S =
{τi},TS→H= {τj}, τi,τjhave the same area and the slot-
ted area model is used. Also suppose that RD area is fully
occupied. Then in order to perform relocation we really
need to ”swap” the two tasks from CPU to RD and vice
versa. However, tasks can only be relocated at the begin-
ning of a job, and there may be no time instant in which
two jobs of τi,τjstart simultaneously. Furthermore, recon-
figuring the device takes time. Therefore, the only feasible
approach is as follows: first, at the beginning of some job
τs
the bitstream of τh
beginning of some job τs
Note that for some time both tasks’ software configurations
are running on the CPU. This implies that relocation incurs
an overhead in term of CPU utilization, in the sense that
in order to perform relocation in a feasible way we need to
leave some free computational power on the CPU in order
to feasibly schedule an additional software configuration.
Feasibilityconstraintsforsoftwaretasksaretypicallyex-
pressed for fixed task sets, while in our case the set of active
software configurations TSrunning on the CPU frequently
changes. However, it can be trivially proven that, if a soft-
ware configuration is considered to be active on the CPU
until the deadline of its last software job, then the classic
EDF utilization bound:
∀t ≥ 0,UTS(t)≤ 1
can still be applied.
Relocating tasks that have different areas is more diffi-
cult. We can clearly always perform relocation by first ac-
tivating on the CPU the software configurations of all tasks
T
= {TS→S ∪ TS→H,TH→H ∪ TH→S},we
T= {TS→S∪ TH→S,TH→H∪ TS→H}. Hence, TS→S
iis activated on the CPU while τh
jis loaded in the device. Finally, at the
jis terminated and τh
iis suspended. Then,
jis started.
(11)
5