PreprintPDF Available

Leveraging Test Logs for Building a Self-Adaptive Path Planner

Authors:
Preprints and early-stage research may not have been peer reviewed yet.

Abstract and Figures

Recent approaches in testing autonomous driving systems (ADS) are able to generate a scenario in which the autonomous car collides , and a different ADS configuration that avoids the collision. However, such test information is too low level to be used by engineers to improve the ADS. In this paper, we consider a path planner component provided by our industry partner, that can be configured through some weights. We propose a technique to automatically re-engineer the path planner in terms of a self-adaptive path planner (SAPP) following the MAPE loop reference architecture. The Knowledge Base (KB) of SAPP contains descriptions of collision scenarios discovered with testing, and the corresponding alternative weights that avoid the collisions. We forecast two main usages of SAPP. First of all, designers are provided with a prototype that should facilitate the re-implementation of the path planner. As second usage, SAPP can be useful for improving the diversity of testing, as performing test case generation on SAPP will guarantee to find dangerous situations different from those used to build SAPP. Preliminary experiments indicate that SAPP can effectively adapt on the base of the solutions stored in KB.
Content may be subject to copyright.
Leveraging Test Logs for Building a Self-Adaptive Path Planner
Kun Liu
Peking University
China
kunl@pku.edu.cn
Xiaoyi Zhang
National Institute of Informatics
Japan
xiaoyi@nii.ac.jp
Paolo Arcaini
National Institute of Informatics
Japan
arcaini@nii.ac.jp
Fuyuki Ishikawa
National Institute of Informatics
Japan
f-ishikawa@nii.ac.jp
Wenpin Jiao
Peking University
China
jwp@sei.pku.edu.cn
ABSTRACT
Recent approaches in testing autonomous driving systems (ADS)
are able to generate a scenario in which the autonomous car col-
lides, and a dierent ADS conguration that avoids the collision.
However, such test information is too low level to be used by en-
gineers to improve the ADS. In this paper, we consider a path
planner component provided by our industry partner, that can be
congured through some weights. We propose a technique to auto-
matically re-engineer the path planner in terms of a self-adaptive
path planner (SAPP) following the MAPE loop reference architec-
ture. The Knowledge Base (
KB
) of SAPP contains descriptions of
collision scenarios discovered with testing, and the corresponding
alternative weights that avoid the collisions. We forecast two main
usages of SAPP. First of all, designers are provided with a prototype
that should facilitate the re-implementation of the path planner. As
second usage, SAPP can be useful for improving the diversity of
testing, as performing test case generation on SAPP will guarantee
to nd dangerous situations dierent from those used to build SAPP.
Preliminary experiments indicate that SAPP can eectively adapt
on the base of the solutions stored in KB.
CCS CONCEPTS
Software and its engineering Software testing and de-
bugging.
KEYWORDS
path planner, testing, self-adaptation, MAPE loop
ACM Reference Format:
Kun Liu, Xiaoyi Zhang, Paolo Arcaini, Fuyuki Ishikawa, and Wenpin Jiao.
2020. Leveraging Test Logs for Building a Self-Adaptive Path Planner. In
IEEE/ACM 15th International Symposium on Software Engineering for Adap-
tive and Self-Managing Systems (SEAMS ’20), October 7–8, 2020, Seoul, Re-
public of Korea. ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/
3387939.3391606
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specic permission and/or a
fee. Request permissions from permissions@acm.org.
SEAMS ’20, October 7–8, 2020, Seoul, Republic of Korea
©2020 Association for Computing Machinery.
ACM ISBN 978-1-4503-7962-5/20/05. . . $15.00
https://doi.org/10.1145/3387939.3391606
1 INTRODUCTION
Automated and autonomous driving is an active research area aim-
ing at drastically changing our personal mobility and the way we
transport goods. Although one of the main promises of automated
and autonomous driving is to drastically reduce the number of ac-
cidents, reports about collisions of self-driving cars [
17
,
21
] demon-
strate that the technology is not mature enough, and big eorts
should be spent on testing and improving this kind of systems.
Testing of autonomous driving system (ADS) cannot be based
only on real test drives [
13
,
23
]. A valid alternative is to perform
virtual tests [
24
] in computer simulations. In this line of research,
works have been proposed for testing ADS [
2
,
4
,
10
]. In ADS testing,
the oracle problem [
3
] is particularly challenging as, when a collision
is found by testing, it is not always easy to assess whether the
collision is the responsibility of the ego car. Moreover, even if the
ego car is not responsible for the collision, one may be interested
in knowing whether the ego car should have been able to avoid it.
To address this issue, we have recently proposed [
6
] the notion of
avoidable collision, i.e., a collision that can be avoided by conguring
the ADS in a dierent way.
In [
6
] and here, we consider the path planner component of an
ADS, provided by our industry partner. The path planner can be
congured through parameters (called weights) that are involved
in the decision process of generating and selecting paths. Each
value assignment for the weights can be seen as a new path planner
instance. In [
6
], we have proposed a search-based test case gen-
eration method for the path planner that searches for avoidable
collisions. The method searches, at the same time, a scenario
S
in
which using the original path planner the car collides, and a set of
alternative weights
w
that avoid the collision (i.e., using the path
planner congured with weights
w
on scenario
S
, the car does
not collide). The advantage of that approach is that it does not
only exhibit a failure of the system (as in classical testing), but also
nds a solution to x the corresponding fault. In [
6
], we run the
test generation method over 7 search spaces: each search space is
dened by a baseline scenario that can be modied by changing
some of its elements (e.g., the position and speed of another car);
in this way, we found several dierent collision scenarios
S
with
their corresponding solutions w.
While some of these collision scenarios are solved by similar
alternative weights
w
(the collisions happen for the same reason),
other collisions are more dierent and need specic weights
w
to
be avoided (the collisions happen for dierent reasons).
SEAMS ’20, October 7–8, 2020, Seoul, Republic of Korea Liu, et al.
Using the information obtained with testing, the designers can re-
engineer the path planner so that it can avoid the types of collision
for which testing showed a solution. However, this can be both
time consuming and dicult, as the information given by testing
could be too low level to inuence any re-engineering decision.
We claim that, before asking the designers to x the path planner,
we should try to automatically re-engineer it, to incorporate the
information discovered by testing. We think that self-adaption [
7
,
8
,
14
] provides the right architectural framework for doing such
automatic re-engineering.
Contribution. The main contribution of this paper is the integra-
tion of path planner testing and its automatic re-engineering as a
self-adaptive path planner (SAPP). To the best of our knowledge,
the construction of an adaptive path planner only based on testing
information has never been proposed so far.
In particular, we propose to construct a Knowledge Base starting
from the test logs generated with testing. On the top of this, the SAPP
follows the typical reference architecture used in self-adaptation, in
which the system is divided into a managed system and a managing
system. The managed system is a slightly modied version of the
original path planner, which, at runtime, receives new weights from
the managing system and changes its behaviour accordingly.
The execution of SAPP is as follows. The managing system mon-
itors the runtime information given by the path planner (i.e., the
values of the cost function), checks whether these are similar to
those recorded for dangerous situations discovered during testing
(stored in the Knowledge Base), and, if this is the case, retrieves the
weights
w
that could solve this dangerous situation (because they
solved a similar one) and re-congures the path planner with them.
These are the two main potential applications of SAPP:
1)
designers obtain an early prototype that can be used as a start-
ing point for the implementation of a new version of the path
planner. We claim that, for engineers to understand the reason
why a given solution is able to avoid a collision, it is easier to
use the SAPP rather than just observing the test. During test,
the alternative weights are xed at the beginning and remain
unchanged, so making it dicult to spot dangerous situations;
with SAPP, weights change exactly when the execution becomes
dangerous, so making the explanation apparent;
2)
testing can benet from the SAPP. Indeed, from the results in [
6
],
we noticed that, in some cases, dierent executions of the test
generation approach (over dierent search spaces) generate dif-
ferent collisions solved by the same solution: this means that
the found collisions are of the “same type”. We think that, once
we nd one type of solution for a particular kind of collision,
we should embed it in the SAPP and then use this in subsequent
testing executions. This should guarantee to nd diverse types
of collision; indeed, since the SAPP already knows how to avoid
the types of collision found in previous testing phases, the test
generation search will be guided towards new types of collision.
Paper structure. Sec. 2 provides some basic denitions regarding
the considered path planner, and our test generation approach
for it. Then, Sec. 3 presents the approach we propose to build the
self-adaptive path planner starting from the generated tests, and
Sec. 4 describes a preliminary evaluation. Sec. 5 provides a critical
discussion about the advantages and limits of the approach. Finally,
Sec. 6 reviews some related work, and Sec. 7 concludes the paper.
2 DEFINITIONS
In the following, we provide the minimal denitions that are needed
for understanding our approach. Please refer to [
6
,
16
] for a more
extensive description of our simulation environment.
2.1 Path planner
We name ego car the autonomous car running the path planner.
Denition 2.1 (Path). Apath
p
of the ego car is a sequence of
car states
[st0, . . . , stn]
where the time dierence between two
consecutive states is xed, i.e.,
ti+1ti=tsim
(for
i=
0
, . . . , n
1),
being
tsim
a simulation parameter
1
. Each state
sti
identies the
position, direction, velocity, and acceleration of the car at time ti.
Denition 2.2 (Scenario). Ascenario
S
denes the dynamic en-
vironment in which the ego car is driving: (i) a map of the road
structure; (ii) the initial state of the ego car; (iii) a target destination
for the ego car; (iv) the initial states and dynamic behaviors of other
cars; (v) a timeout until which the simulation is run.
In an automated or autonomous [
20
] driving system, the path
planner is repeatedly executed in order to select the trajectory of the
ego car in a two-dimensional space. At each time step, the position
of the surrounding elements (within a given range) is analyzed,
and some short-term paths are calculated. The best one is selected,
and followed by the ego car for
tsim
seconds. Then, the process
restarts. So, the nal path
p
is given by the concatenation of the
rst section of the selected short-term paths.
The path planner provided by our industry partner uses a weighted
cost function (more details are given later in Def. 2.6) to rank short-
term paths and select the least costly one. The function uses some
weights that give dierent importance to elements as maximum
speed, lateral acceleration, curvature, etc.
Denition 2.3 (Path Planner). We identify with
PPw
the path
planner congured with given weight values
w=[w1, . . . , w8]
. It
can be seen as a function that, given a scenario
S
, produces a path
p
for the ego car up to the max simulation time. Formally,
PPw(S)=p
.
From here on, we identify with
PPwo
the path planner congured
with the original weight values
wo
provided by our industry partner.
Moreover, we identify with
PPw
the path planner congured with
alternative weight values w.
We have previously said that, at each time step, the path planner
scores the dierent possible short-term paths using a weighted
function of dierent aspects. The function rst computes 18 cost
components (each one related to a given aspect), and then aggregates
them. For our purposes, we are interested in the sequence of such
cost components along the path.
Denition 2.4 (Cost function log). Given a scenario
S
, a path
planner
PPw
, and the path
PPw(S)=[st0, . . . , stn]
computed by
PPw
over
S
, we dene the cost function log as the sequence
[cv0,
. . . , cvn]
, where each
cvi=[cv1
i, . . . , cv18
i]
contains the computed
cost components of the cost function at time
ti
. In particular,
cvj
i=
fj(w,xi)
, where
xi
are characteristics of the ego car and of the
environment at time
ti
(e.g., lateral acceleration, speed, etc.), and
1In our experiments, tsim =0.1s.
Leveraging Test Logs for Building a Self-Adaptive Path Planner SEAMS ’20, October 7–8, 2020, Seoul, Republic of Korea
fj
is a function that aggregates some of the values of
xi
with some
of the weights in w.
Each function
fj
considers dierent aspects like safety (e.g., get-
ting too close to other cars, etc.), trac regulation (e.g., violation
of the speed limit), or comfort (e.g., too much acceleration or cur-
vature, etc.). For example, the short-term paths requiring too much
acceleration are penalized w.r.t. the ones requiring less accelera-
tion.2
2.2 Testing for the path planner
Denition 2.5 (Avoidable collision). Let
PPwo
be the original path
planner and
S
a scenario for which the ego car collides. We say that
the collision is avoidable if there exist some dierent weights
w
such that
PPw
(i.e., the path planner congured with the dierent
weights w) does not lead to a collision. We identify the avoidable
collision as S,w.
In [
6
], we have proposed a testing approach that searches for
avoidable collisions: in particular, the approach searches for colli-
sions that can be avoided with the minimum change of the weights.
In the following, a test is an avoidable collision
S,w
. Starting
from a test t, we can obtain further information as follows.
Denition 2.6 (Test log). Given a test
S,w
, we identify as test
log the tuple
S,w,po,p,CVo,CV
where: (i)
po
is the path com-
puted by the original path planner over
S
, i.e.,
po=PPwo(S)
; (ii)
p
is the path computed by the alternative path planner over
S
, i.e.,
p=PPw(S)
; (iii)
CVo
is the cost function log of
po
; and (iv)
CV
is the cost function log of p.
Using the approach proposed in [
6
], we found dierent avoidable
collisions, constituting our test suite
T
. Starting from these, we
derived a set of test logs TL.
3 SELF-ADAPTIVE PATH PLANNER (SAPP)
In the following, we describe how we automatically re-engineer the
path planner, using the information (i.e., the test logs
TL
) we obtain
from testing.
Our aim is to obtain a self-adaptive path planner (SAPP) that
behaves as the original path planner when the execution is safe, but
adapts at runtime in case of dangerous situation. The adaptation of
SAPP is driven by the two following MAPE loops (following the
classical reference architecture [5, 14] for self-adaptive systems):
1)
the Reconguration Loop determines whether the current exe-
cution is potentially dangerous by comparing it with dangerous
executions obtained through testing. If this is the case, it deter-
mines how the original weights should be re-congured in order
to improve safety, using some of the weights discovered with
testing.
2)
the Reset Loop identies whether the path planner re-congured
with the new weights should be reset to the original weights.
This is necessary because the weights discovered with testing
are only known to avoid dangerous situations.
Note that, at runtime, only one MAPE loop is active at a time, and
the interaction of the two loops is as follows. At the beginning of
2
Note that the complete implementation of functions cannot be provided for IP pro-
tection. In any case, it is not needed for understanding the proposed approach.
Figure 1: Architecture of the SAPP
the execution, the SAPP is congured with the original weights and
only the Reconguration loop is activated and performs monitoring.
After the reconguration of the path planner (i.e., the whole loop is
executed), the Reconguration loop is de-activated, and the Reset
loop starts by monitoring the total cost values of the adapted path
planner. Once the path planner is recongured with the original
weights (i.e., the whole loop is executed), the Reset loop is de-
activated and the Reconguration loop starts again.
3.1 Reconguration Loop
Fig. 1 shows the structure of the Reconguration Loop. The man-
aged system is the original path planner that can receive some alter-
native weights and change them at runtime. Instead, the managing
system responsible for self-adaptation, is structured as follows:
1)
the Knowledge Base (
KB
) contains a processed version of the raw
information we obtain in testing (the test logs in Def. 2.6);
2)
the Monitor observes and collects data about the runtime state
of the ego car (in terms of values of the cost function);
3)
the Analyzer analyzes the runtime state and assesses whether it
is similar to some dangerous situation (stored in
KB
) discovered
with testing; if it is so, it triggers the Planner;
4)
the Planner
3
tries to nd a solution (i.e., alternative weights)
among those discovered by testing and stored in the KB; and
5)
the Executor changes the conguration of the path planner ac-
cordingly (i.e., it changes the weights).
In the following, we rst describe each element of the loop (the
KB
, and the four MAPE components) in details. Sec. 3.1.1 describes
the knowledge base.
4
Sects. 3.1.2, 3.1.3, 3.1.4, and 3.1.5 describe the
MAPE loop that is executed to adapt
PPwo
, i.e., the path planner
congured with the original weights.
3.1.1 Knowledge Base. The SAPP should adapt at runtime on the
base of the information we learned from testing. Such information
is stored in the knowledge base, denoted by
KB
. Formally, it is de-
ned as the set
KB ={⟨CV kb
1,w
1,Γ
1, . . . , CV kb
m,w
m,Γ
m⟩}
. Each
element CV kb
i,w
i,Γ
iis dened as follows:
CV kb
i=[cv1, . . . , cvK]
is a series of
K
cost value tuples com-
puted by the original path planner
PPwo
for some scenario
S
.
Namely, they are selected cost value tuples of some system exe-
cutions that have been shown to be unsafe (leading to collision)
during testing using the original weights
wo
. Intuitively,
CV kb
i
characterizes a dangerous situation. The value
K
is a parameter
of the approach. In Sec. 3.3, we will describe in details how we
3
Note that all the occurrences of planner without “path” refer to the MAPE component.
4
We describe the construction of
KB
later in Sec. 3.3, as the rationale of building the
KB is better understandable if the reader is aware of the functioning of the SAPP.
SEAMS ’20, October 7–8, 2020, Seoul, Republic of Korea Liu, et al.
Algorithm 1 Analyzer
Require: OCV =[cvr
1, . . . , cvr
H]
: cost values observed at runtime
Ensure: MS: Matched Solutions
1: MS ← ∅ Matched solutions
2: for CV kb,w,Γ⟩ ∈ KB do CV kb =[cv 1, . . . , cvK]
3: Dists ← ∅
4: for i=1, . . . , KH+1do
5: Dists Dists ∪ {dis([cvi, . . . , cvi+H1],OCV)}
6: MinDist min(Dists)
7: if MinDist Θthen
8: MS MS ∪ {⟨w,MinDist }
9: return MS
perform the selection starting from the test logs
TL
obtained with
testing.
w
are the alternative weights that in testing showed to be able
to solve the unsafe situation characterized by
CV kb
i
(i.e., the path
planner congured with wdoesn’t lead to collision over S).
Γ
i
is the total value of the cost function (computed using weights
w
), below which an execution with
PPw
can be considered safe.
3.1.2 Monitor. If the SAPP is operating using the original weights
wo
(i.e.,
PPwo
is running), at runtime the monitor component ob-
serves the execution of the ego car and records the cost values of
the selected short-term paths. In particular, at each time step
ti
, it
collects the last
H
Observed Cost Values
OCV =[cvr
1, . . . , cvr
H]
and
gives these to the analyzer component. Note that
H
is a parameter
of the approach that is required to be less or equal to the parameter
K(i.e., the length of the cost values sequences stored in KB).
3.1.3 Analyzer. The aim of the analyzer is to detect if the current
execution of the ego car is similar to some dangerous situation
stored in
KB
. Alg. 1 shows the operations performed by the analyzer.
The analyzer receives as inputs the last
H
cost values
OCV
collected
by the monitor. For each CV kb,w,Γin the KB (line 4):
it computes the Euclidean distance between
OCV
and each sub-
sequence of consecutive Helements in CV kb (line 5);
it collects the distances in
Dists
and selects the minimum one
MinDist
(line 6).
MinDist
determines the similarity degree be-
tween the runtime cost values OCV and the KB element;
if
MinDist
is at most
Θ
(line 7), the pair
w,MinDist
is selected
as matched solution and added to the set
MS
(line 8). Note that
Θ
is a parameter of the approach and determines the precision we
require during adaptation. If
Θ
is very low, we want that the SAPP
only adapts when it observes situations that are very similar to
those recorded in
KB
; therefore, the SAPP will adapt in few cases.
On the other hand, higher values of
Θ
make the condition less
demanding and so the SAPP will adapt more frequently.
When the analyzer terminates, it triggers the planner by providing
all the matched solutions in MS.
3.1.4 Planner. The planner component must decide, among all
the matched solutions, which one to apply to the path planner.
Dierent policies could be applied, e.g., taking the solution with the
minimum cost value distance, or the one with maximum weights
distance to the original weights
wo
. In the current approach, we
follow the latter strategy: the rationale is that more diverse weights
Algorithm 2 Planner
Require: MS: set of solutions matched by the analyzer
Ensure: Selected weights wor null
1: if MS ,then
2: w,MinDist⟩ ← arg max w
,MinDist⟩∈MS di (wo,w)
3: return w
4: else return null
should guarantee to have a more diverse behaviour (and so avoid
the potential collision). The implementation of the planner is shown
in Alg. 2. If
MS
is empty, it means that no possible solution has
been matched by the analyzer: in this case,
null
is returned (line 4),
meaning that no adaptation is needed. Otherwise, it selects the
solution whose weights
w
have the bigger dierence with the
original weights
wo
(line 2), and returns
w
as selected weights
(line 3). If alternative weights
w
have been selected, the planner
triggers the executor component.
3.1.5 Executor. The executor simply noties the path planner (the
managed system) about the new weights
w
, and the path planner
re-congures with
w
. Note that the whole MAPE loop execution
should not last more than
tsim
(the time dierence between two
consecutive computations of the short-term path. See Sec. 3.1.1),
i.e., if a dangerous situation is detected for the cost values up to
time
tk
, the path planner should be notied about the new weights
before computing the new short-term path at time tk+1.
Once the path planner changes its weights from
wo
to
w
i
, the
managing system stops executing the Reconguration Loop, and
starts the Reset Loop described in the following section.
3.2 Reset Loop
While the path planner is in adaptation mode with weights
w
(i.e.,
PPw
), the managing system performs the Reset Loop that tries to
determine when the path planner can be re-congured with the
original weights
wo
. The MAPE loop has monitoring and analysis
components that, at each timestep, check whether the total cost
value is lower than the threshold value
Γ
stored in
KB
(the same used
in the Reconguration loop. See Sec. 3.1.1) for the current weights
w
(i.e.,
CV kb,w,Γ
). If this is the case, the executor component
applies the original weights
wo
. Indeed, the SAPP should use the
alternative weights as less as possible, only when it is really needed
for avoiding a dangerous situation. Such requirement is particularly
important for testing, as we will discuss in Sec. 5.
3.3 Construction of the KB from Test Logs
The SAPP described in the previous section uses a knowledge base
KB
composed of elements
CV kb,w,Γ
where each
CV kb
charac-
terizes a dangerous situation (in terms of sequence of cost value
tuples),
w
are the weights that have demonstrated to avoid the
dangerous situation, and Γis the total cost value below which the
situation is no more dangerous.
Note that each element of
KB
is constructed from one of the test
logs in
TL
, generated with the testing approach presented in [
6
].
In this section, we describe how we perform such construction.
Firstly, we provide some notation for describing the modication
at runtime of the weights of the path planner.
Leveraging Test Logs for Building a Self-Adaptive Path Planner SEAMS ’20, October 7–8, 2020, Seoul, Republic of Korea
Denition 3.1 (Path planner reconguration). Given a path plan-
ner
PPw
, and some alternative weights
w
, we denote with
PP
w
tk
w
the path planner that uses the weights
w
up to time
tk
1, and, at
time tk, changes them to w.
3.3.1 Leading principles. In order to construct the
KB
, we follow
two leading principles:
P1
The SAPP should not adapt (i.e., change its weights) in situations
that are safe. To this aim, the
KB
should only contain sequences of
cost value tuples that are proven to be unsafe, so that the analyzer
of the SAPP (see Sec. 3.1.3) decides to adapt only if really needed;
P2
The SAPP should adapt before reaching situations that are too
unsafe, i.e., from which it is no more possible to avoid the collision.
To this aim, the
KB
should only contain sequences of cost values
tuples describing situations in which changing the weights can still
allow to avoid the collision.
3.3.2 Construction of
KB
.Following the previous two principles,
we lter the test logs
TL
by trying to select only the sections of
system executions that are dangerous (i.e., they deserve adaptation)
and from which it is still possible to recover to a safe situation. The
KB is then built as follows:
1)
given all the test logs
TL
, we select some representative ones.
To this aim, given all the solutions of all the Pareto fronts ob-
tained in dierent executions
5
, we take the solutions on the
reference Pareto front, i.e., the non-dominated solutions across
all the Pareto fronts. In this way, we guarantee not to add to
KB
redundant solutions. Let’s call f
TL these selected test logs.
2)
then, for each test log
S,w,po,p,CVo,CV f
TL
, we perform
the following steps:
a)
We identify the last time point
lp
from which the modied
path planner found in testing would still be able to avoid the
collision. Specically, we select the time point
tlp
, such that
PP
wo
tlp
w(S)
does not lead to a collision and
PP
wo
tlp+1
w(S)
does. The rationale is that the state reached by the ego car
after such a point is too unsafe and cannot be recovered to a
safe state anymore. This is in accordance with principle P2;
b)
following principle
P1
, we do not want to match logs that
represent too safe situations, i.e., we do not want to change
the weights if not needed. For this reason, we select only
the part of the log that immediately precedes
lp
. Namely, we
specify a number
K
of time steps that we want to keep from
the test logs (this is a parameter of the approach);
c)
given
lp
and the parameter
K
, we lter the cost values
CVo=
[cv0, . . . , cvn]
as follows:
CV kb =[cvlpK+1, . . . , cvlp ]
. Note
that, if
lp <K
1, we select
CV kb =[cv0, . . . , cvlp]
. If
lp <H
1, the test log is not selected for the knowledge base:
indeed, the ltered test log is not long enough to be matched
by Hvalues during the analysis phase (see Sec. 3.1.3).
d)
we also select the earliest safe time point
ep
, i.e., the rst
time point from which it is safe to return back to the orig-
inal weights. Namely, we select the rst
ep >lp
such that
PP
wtep1
wo(S)
leads to collision and
PP
wtep
wo(S)
doesn’t; we
then select from
CV
the cost value tuple
cv
ep
, and calculate
5
The test generation in [
6
] is based on a multi-objective search that returns a Pareto
front of solutions.
Table 1: Experiments
(a) Benchmarks
ID Scenarios used
B142 scenarios contained in f
TL
B2
523 scenarios leading to avoid-
able collisions
B3
582 scenarios that are know
to be safe
(b) Results
ID # exec. Safe exec. Adaptation
# % Total Safe
# % # %
B142 40 95.2% 40 95.2% 40 100.0%
B2523 347 66.3% 373 71.3% 347 93.0%
B3582 581 99.8% 255 43.8% 254 99.6%
Γ
=
Ícv cv
ep cv
.
Γ
is the total cost value that is used, when
the path planner is adaptation mode, to decide whether to
return to the original weights (see Sec. 3.2);
e)
we add the tuple
CV kb,w,Γ
to
KB
. Therefore, each tuple of
the
KB
corresponds to one particular test
t
: the information
regarding “how to avoid the collision”, that in the test is
implicit, has been made explicit in the tuple of
KB
, so that it
can be used for self-adaptation.
4 PRELIMINARY EVALUATION
SAPP construction. We have built the SAPP using the test logs of
the tests generated with the approach in [
6
]. There, we performed
7 dierent test case generation using 7 search spaces
6
; each type of
search was repeated 30 times: in total, we have obtained a set of
565 test logs
TL
. From these, we have selected 42 representative test
logs
f
TL7
and built the
KB
as described in Sec. 3.3.2. As parameters
for the SAPP, we have selected
K=
10,
H=
2, and
Θ=
10
E
6.
Note that K=10 means that we select one second before lp.
Experiments. As initial evaluation of the approach, we have per-
formed three experiments by running the SAPP over three dierent
sets of scenarios. Table 1a reports the three benchmarks. B
1
is the
same set used to build the SAPP: this will be used as sanity check.
B
2
is composed of scenarios (dierent from those used in building
the
KB
) that lead to a collision with the original path planner, and
we know that the collision is avoidable: this will be used to assess
to what extent the SAPP can recover from dangerous situations not
seen before. Finally, B
3
contains only scenarios that we know do not
lead to collisions with the original path planner: this will be used
to check that the SAPP does not introduce dangerous situations.
Table 1b reports experimental results. It reports the number of
scenarios in the benchmark, how many have been run by the SAPP
without collision, how many adaptations have been performed, and
how many of these did not lead to a collision (i.e., they were safe).
Regarding B
1
, we observe that almost all scenarios used to build
the SAPP are executed safely, i.e., the SAPP adapts at runtime
and avoids the collision. Only for two scenarios the SAPP collides:
we discovered that the collision could be avoided only using the
alternative weights since the beginning of the system execution:
however, this is not possible, as the SAPP needs to observe at least
H
cost value tuples before adapting (see Sec. 3.1.3). Note that these
two scenarios are indeed not present in
KB
, as their test logs have
been discarded during KB construction (see item 2c in Sec. 3.3.2).
6
A search space is given by a baseline scenario in which some elds (e.g., position and
acceleration of another car) can be changed.
7
Note that not all the runs in [
6
] generated some avoidable collisions. Moreover, some
Pareto fronts only contain one solution.
SEAMS ’20, October 7–8, 2020, Seoul, Republic of Korea Liu, et al.
Regarding B
2
(i.e., scenarios characterizing avoidable collisions
not present in
KB
), we observe that 66.3% of the SAPP executions
are able to avoid the collision. We believe that these are encouraging
results, as we are able to avoid new collisions for which the original
path planner collides. In 7% of the cases in which the SAPP adapts,
the collision still occurs: this is due to the fact that, although the
SAPP recognized a dangerous situation, it adapted too late. We
further analyzed the results and we observed that for 26 scenarios,
the only way to avoid to collision would be to use the alternative
weights since the beginning: as explained before, the SAPP cannot
safely adapt in these cases, as it needs to start from a safe situation
and then recognize the dangerous situation.
Regarding B
3
(i.e., scenarios that are known to be safe), the SAPP
never collides, except for one case. We analyzed this case and we
discovered that using the original path planner the ego car would
follow a path very similar to a dangerous path in the
KB
: the SAPP
recognized this very dangerous situation and tried to adapt.
5 DISCUSSION
In the following, we provide a critical review of our eorts and
outline some possible lines of improvement.
Overtting. Given the way the SAPP is constructed, we are aware
that it will tend to overt to the test logs, i.e., it will specialize in
solving the avoidable collisions described in
KB
(or very similar
ones), but it will not be too general. Actually, this is not an issue in
our context. Indeed, one of the main usages of the SAPP will be to
improve the diversity of tests discovered with testing; we plan to
use it in an iterative process in which (i) we rst do testing using
the approach in [
6
], obtaining test logs
TL1
; (ii) we build the SAPP
using
TL1
; (iii) we do testing on the SAPP itself and we use the
new test logs
TL2
to further improve SAPP; this last step can be
applied iteratively. In such a testing context, we aim at nding the
most diverse collisions. Hence, we only require that SAPP is able to
avoid the collisions found in previous testing executions, so that
the test generation will be guided towards new types of collisions.
As additional requirement for testing, we want that SAPP uses the
alternative weights only when really needed, and re-congures
with the original ones as soon as possible: in this way, the testing
activity will focus in nding collisions for the original path planner
(that is our target program).
Conguration of the SAPP. The presented approach depends on
the selection of some parameters (
K
,
H
, and
Θ
). For our prelimi-
nary evaluation, we tried few combinations (not reported in this
paper) and we reported the one having the best results. However,
more experiments are needed to assess the inuence of parameter
selection on the quality of the SAPP. Moreover, we believe that the
parameter
K
(i.e., the length of the test log fragments in
KB
) should
not be unique for all the test logs, but should be computed for each
test log: to this aim, we will perform a more detailed analysis of the
cost value tuples to properly characterize the dangerous zone.
Considering other elements during adaptation. In our approach,
the adaptation process is only based on the values of the cost func-
tion. However, the test logs (see Def. 2.6) returned with testing
contain other information that could be exploited to characterize
dangerous situations, as the position of the ego car and of the other
cars, the road condition, etc. This is the reason why in Fig. 1 we
also show the environment in which the ego car is operating, as
this could be used in future implementations of the SAPP. Note
that this way of adapting based on the driving scenario is currently
adopted in some path planner implementations [1].
6 RELATED WORK
Due to the lack of space, we here focus on works that are related to
the improvement of path planners. In this area, researchers usually
extend the planning algorithms by introducing parameters related
to the environment [
18
,
19
], or adding adaptive mechanisms to
the inner controllers based on control theory [
11
,
12
,
25
]. A work
related to ours was proposed by Kim et al. [
15
] who used inverse
reinforcement learning to train a model of human behaviors that
can guide the planner of a robot, in particular when interacting with
humans; the main dierences with our approach are that: they do
not derive information from testing, they do not use a self-adaptive
architecture, and their target application is not ADS.
Some approaches have been proposed for path planner testing.
For example, Geng et al. [
9
] propose a model-based testing approach
based on hidden Markov Models, but they did not consider the re-
engineering of the path planner based on testing results.
Some recent works related to path planning approaches focus on
the situations involving multiple automotive agents [
19
,
22
]. As fu-
ture work, we plan to improve our simulation environment (where
the SAPP is executed and tested) by providing more intelligence to
the other cars running in the simulation. This will allow to explore
more dierent interactions of the ego car with the other cars.
7 CONCLUSIONS
In this paper, we proposed a technique to automatically re-engineer
a path planner in terms of a self-adaptive path planner (SAPP),
so that it can avoid dangerous situations discovered with testing.
Preliminary experimental results show that the SAPP can safely
avoid more than half of (avoidable) collisions not seen before (i.e.,
not present in the
KB
), and it almost always does not collide in
executions that are known to be safe.
Our immediate future work is to integrate the SAPP in an iterative
testing framework that works as follows: at each iteration, it rst
performs testing and then improves the
KB
of the SAPP with the
test logs found in testing. Since the preliminary experiments show
that the SAPP can eectively adapt at runtime, we believe that test
generation will be guided towards dangerous situations dierent
from those already present in KB.
ACKNOWLEDGMENTS
X. Zhang, P. Arcaini and F. Ishikawa are supported by ERATO
HASUO Metamathematics for Systems Design Project (No. JPM-
JER1603), JST. Funding Reference number: 10.13039/501100009024
ERATO. K. Liu and W. Jiao are sponsored by the National Natural
Science Foundation of China (61620106007). We thank our industry
partner Mazda for providing the software used as the target in our
work and discussing principles in testing and improving complex
real-world automotive systems. The provided software is a proto-
type constructed for the purpose of evaluating new testing methods
and its quality has no relation with the quality of Mazda products.
Leveraging Test Logs for Building a Self-Adaptive Path Planner SEAMS ’20, October 7–8, 2020, Seoul, Republic of Korea
REFERENCES
[1]
2020. Apollo Auto – Planning component. https://github.com/ApolloAuto/
apollo/tree/master/modules/planning Accessed April 3, 2020.
[2]
Matthias Altho and Sebastian Lutz. 2018. Automatic generation of safety-critical
test scenarios for collision avoidance of road vehicles. In 2018 IEEE Intelligent
Vehicles Symposium (IV). IEEE, 1326–1333.
[3]
Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015.
The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software
Engineering 41, 5 (May 2015), 507–525. https://doi.org/10.1109/TSE.2014.2372785
[4]
Halil Beglerovic, Michael Stolz, and Martin Horn. 2017. Testing of autonomous
vehicles using surrogate models and stochastic optimization. In Intelligent Trans-
portation Systems (ITSC), 2017 IEEE 20th International Conference on. IEEE, 1–6.
[5]
Yuriy Brun, Giovanna Di Marzo Serugendo, Cristina Gacek, Holger Giese, Holger
Kienle, Marin Litoiu, Hausi Müller, Mauro Pezzè, and Mary Shaw. 2009. Engineer-
ing Self-Adaptive Systems through Feedback Loops. Springer Berlin Heidelberg,
Berlin, Heidelberg, 48–70. https://doi.org/10.1007/978-3- 642-02161-9_3
[6]
Alessandro Calò, Paolo Arcaini, Shaukat Ali, Florian Hauer, and Fuyuki Ishikawa.
2020. Generating Avoidable Collision Scenarios for Testing Autonomous Driving
Systems. In 2020 IEEE International Conference on Software Testing, Verication
and Validation (ICST). (to appear).
[7]
Betty H. Cheng, Rogério Lemos, Holger Giese, Paola Inverardi, Je Magee, Jesper
Andersson, Basil Becker, Nelly Bencomo, Yuriy Brun, Bojan Cukic, Giovanna
Marzo Serugendo, Schahram Dustdar, Anthony Finkelstein, Cristina Gacek, Kurt
Geihs, Vincenzo Grassi, Gabor Karsai, Holger M. Kienle, Je Kramer, Marin Litoiu,
Sam Malek, Raaela Mirandola, Hausi A. Müller, Sooyong Park, Mary Shaw,
Matthias Tichy, Massimo Tivoli, Danny Weyns, and Jon Whittle. 2009. Software
Engineering for Self-Adaptive Systems. Springer-Verlag, Berlin, Heidelberg,
Chapter Software Engineering for Self-Adaptive Systems: A Research Roadmap,
1–26. https://doi.org/10.1007/978-3- 642-02161-9_1
[8]
Rogério De Lemos, Holger Giese, Hausi A Müller, Mary Shaw, Jesper Andersson,
Marin Litoiu, Bradley Schmerl, Gabriel Tamura, Norha M Villegas, Thomas Vogel,
et al
.
2013. Software engineering for self-adaptive systems: A second research
roadmap. In Software Engineering for Self-Adaptive Systems II. Springer, 1–32.
https://doi.org/10.1007/978-3- 642-35813-5_1
[9]
Xinli Geng, Huawei Liang, Biao Yu, Pan Zhao, Liuwei He, and Rulin Huang. 2017.
A scenario-adaptive driving behavior prediction approach to urban autonomous
driving. Applied Sciences 7, 4 (2017), 426.
[10]
Florian Hauer, Alexander Pretschner, and Bernd Holzmüller. 2019. Fitness Func-
tions for Testing Automated and Autonomous Driving Systems. In International
Conference on Computer Safety, Reliability, and Security. Springer, 69–84.
[11]
José Miguel Horcas, Julien Monteil, Mélanie Bouroche, Mónica Pinto, Lidia
Fuentes, and Siobhán Clarke. 2018. Context-dependent reconguration of au-
tonomous vehicles in mixed trac. Journal of Software: Evolution and Process 30,
4 (2018), e1926.
[12]
Todd Jochem and Dean Pomerleau. 1996. Life in the fast lane: The evolution of
an adaptive vehicle control system. AI magazine 17, 2 (1996), 11–11.
[13] Nidhi Kalra and Susan M Paddock. 2016. Driving to safety: How many miles of
driving would it take to demonstrate autonomous vehicle reliability? Transporta-
tion Research Part A: Policy and Practice 94 (2016), 182–193.
[14]
Jerey O. Kephartand David M. Chess. 2003. The Vision of Autonomic Computing.
Computer 36, 1 (Jan. 2003), 41–50. https://doi.org/10.1109/MC.2003.1160055
[15]
Beomjoon Kim and Joelle Pineau. 2016. Socially adaptive path planning in
human environments using inverse reinforcement learning. International Journal
of Social Robotics 8, 1 (2016), 51–66.
[16]
Thomas Laurent, Paolo Arcaini, Fuyuki Ishikawa, and Anthony Ventresque.
2019. A Mutation-based Approach for Assessing Weight Coverage of a Path
Planner. In 2019 26th Asia-Pacic Software Engineering Conference (APSEC). 94–
101. https://doi.org/10.1109/APSEC48747.2019.00022
[17]
Sam Levin and Julia Wong Carrie. 2018. Self-driving Uber kills Arizona woman in
rst fatal crash involving pedestrian. https://www.theguardian.com/technology/
2018/mar/19/uber-self- driving-car-kills- woman-arizona-tempe Accessed April
3, 2020.
[18]
Guangsheng Li and Wusheng Chou. 2018. Path planning for mobile robot using
self-adaptive learning particle swarm optimization. Science China Information
Sciences 61, 5 (2018), 052204.
[19]
Xinyu Liu, Yun Li, Jing Zhang, Jian Zheng, and Chunxi Yang. 2019. Self-adaptive
dynamic obstacle avoidance and path planning for USV under complex maritime
environment. IEEE Access 7 (2019), 114945–114954.
[20]
SAE. 2014. Denitions for Terms Related to On-Road Motor Vehicle Automated
Driving Systems. J3016, SAE International Standard (2014).
[21]
Jack Stewart. 2018. Tesla’s Self-Driving Autopilot Involved
in Another Deadly Crash. https://www.wired.com/story/
tesla-autopilot- self-driving-crash-california/ Accessed April 3, 2020.
[22]
Jeroen van de Sande, Martijn van Riet, Arno Duijster, and Robbert van Vossen.
2019. Autonomous multi-vehicle contact reacquisition using feature-based navi-
gation and in-situ adaptive path planning for AUVs. In OCEANS 2019-Marseille.
IEEE, 1–7.
[23]
Walther Wachenfeld and Hermann Winner. 2016. The release of autonomous
vehicles. In Autonomous Driving. Springer, 425–449.
[24]
Ding Zhao and Huei Peng. 2017. From the Lab to the Street: Solving the Chal-
lenge of Accelerating Automated Vehicle Testing. CoRR abs/1707.04792 (2017).
arXiv:1707.04792
[25]
Pan Zhao, Jiajia Chen, Yan Song, Xiang Tao, Tiejuan Xu, and Tao Mei. 2012.
Design of a control system for an autonomous vehicle based on adaptive-pid.
International Journal of Advanced Robotic Systems 9, 2 (2012), 44.
Full-text available
Article
The unmanned surface vehicle (USV) is usually required to perform some tasks with the help of static and dynamic environmental information obtained from different detective systems such as shipborne radar, electronic chart, and AIS system. The essential requirement for USV is safe when suffered an emergency during the task. However, it has been proved to be difficult as maritime traffic is becoming more and more complex. Consequently, path planning and collision avoidance of USV has become a hot research topic in recent year. This paper focuses on dynamic obstacle avoidance and path planning problem of USV based on the Ant Colony Algorithm (ACA) and the Clustering Algorithm (CA) to construct an auto-obstacle avoidance method which is suitable for the complicated maritime environment. In the improved ant colony-clustering algorithm proposed here, a suitable searching range is chosen automatically by using the clustering algorithm matched to different environmental complexities, which can make full use of the limited computing resources of the USV and improve the path planning performances firstly. Second, the dynamic searching path is regulated and smoothed by the maneuvering rules of USV and the smoothing mechanism respectively, which can effectively reduce the path length and the cumulative turning angle. Finally, a simulation example is provided to show that our proposed algorithm can find suitable searching range according to different obstacle distributions, as well as accomplish path planning with good self-adaptability. Therefore, a safe dynamic global path with better optimize performances is achieved with the help of multi-source information.
Full-text available
Conference Paper
Advancement in testing and verification methodologies is one of the key requirements for the commercialization and standardization of autonomous driving. Even though great progress has been made, the main challenges encountered during testing of autonomous vehicles, e.g., high number of test scenarios, huge parameter space and long simulation runs, still remain. In order to reduce current testing efforts, we propose an innovative method based on surrogate models in combination with stochastic optimization. The approach presents an iterative zooming-in algorithm aiming to minimize a given cost function and to identify faulty behavior regions within the parameter space. The surrogate model is updated in each iteration and is further used for intensive evaluation tasks, such as exploration and optimization.
Full-text available
Article
Driving through dynamically changing traffic scenarios is a highly challenging task for autonomous vehicles, especially on urban roadways. Prediction of surrounding vehicles' driving behaviors plays a crucial role in autonomous vehicles. Most traditional driving behavior prediction models work only for a specific traffic scenario and cannot be adapted to different scenarios. In addition, priori driving knowledge was never considered sufficiently. This study proposes a novel scenario-adaptive approach to solve these problems. A novel ontology model was developed to model traffic scenarios. Continuous features of driving behavior were learned by Hidden Markov Models (HMMs). Then, a knowledge base was constructed to specify the model adaptation strategies and store priori probabilities based on the scenario's characteristics. Finally, the target vehicle's future behavior was predicted considering both a posteriori probabilities and a priori probabilities. The proposed approach was sufficiently evaluated with a real autonomous vehicle. The application scope of traditional models can be extended to a variety of scenarios, while the prediction performance can be improved by the consideration of priori knowledge. For lanechanging behaviors, the prediction time horizon can be extended by up to 56% (0.76 s) on average. Meanwhile, long-term prediction precision can be enhanced by over 26%.
Conference Paper
Automated and autonomous driving systems (ADS) are a transformational technology in the mobility sector. Current practice for testing ADS uses virtual tests in computer simulations; search-based approaches are used to find particularly dangerous situations, possibly collisions. However, when a collision is found, it is not always easy to automatically assess whether the ADS should have been able to avoid it, without relying on offline analyses by domain experts. In this paper, we propose a definition of avoidable collision that does not rely on any domain knowledge, but only on the fact that it is possible to reconFigure the ADS (in our case, the path planner component provided by our industry partner) in a way that the collision is avoided. Based on this definition, we propose two search-based approaches for finding avoidable collisions. The first one (named sequential approach), based on current industrial practice, first searches for a collision, and then searches for an alternative configuration of the ADS which avoids it. The second one (named combined approach), instead, searches at the same time for the collision and for the alternative configuration which avoids it. Experiments show that the combined approach finds more avoidable collisions, even when the sequential approach doesn't find any; indeed, the sequential approach, in the first search, may find too severe collisions for which there is no alternative configuration that can avoid them.
Article
Human drivers naturally adapt their behaviour depending on the traffic conditions, such as the current weather and road type. Autonomous vehicles need to do the same, in a way that is both safe and efficient in traffic composed of both conventional and autonomous vehicles. In this paper, we demonstrate the applicability of a reconfigurable vehicle controller agent for autonomous vehicles that adapts the parameters of a used car-following model at runtime, so as to maintain a high degree of traffic quality (efficiency and safety) under different weather conditions. We follow a dynamic software product line approach to model the variability of the car-following model parameters, context changes and traffic quality, and generate specific configurations for each particular context. Under realistic conditions, autonomous vehicles have only a very local knowledge of other vehicles' variables. We investigate a distributed model predictive controller agent for autonomous vehicles to estimate their behavioural parameters at runtime, based on their available knowledge of the system. We show that autonomous vehicles with the proposed reconfigurable controller agent lead to behaviour similar to that achieved by human drivers, depending on the context.
Article
As a challenging optimization problem, path planning for mobile robot refers to searching an optimal or near-optimal path under different types of constrains in complex environments. In this paper, a self-adaptive learning particle swarm optimization (SLPSO) with different learning strategies is proposed to address this problem. First, we transform the path planning problem into a minimisation multi-objective optimization problem and formulate the objective function by considering three objectives: path length, collision risk degree and smoothness. Then, a novel self-adaptive learning mechanism is developed to adaptively select the most suitable search strategies at different stages of the optimization process, which can improve the search ability of particle swarm optimization (PSO). Moreover, in order to enhance the feasibility of the generated paths, we further apply the new bound violation handling schemes to restrict the velocity and position of each particle. Finally, experiments respectively with a simulated robot and a real robot are conducted and the results demonstrate the feasibility and effectiveness of SLPSO in solving mobile robot path planning problem.
Article
As automated vehicles and their technology become more advanced and technically sophisticated, evaluation procedures that can measure the safety and reliability of these new driverless cars must develop far beyond existing safety tests. To get an accurate assessment in field tests, such cars would have to be driven millions or even billions of miles to arrive at an acceptable level of certainty - a time-consuming process that would cost tens of millions of dollars. Instead, researchers affiliated with the University of Michigan's Mcity connected and automated vehicle center have developed an accelerated evaluation process that eliminates the many miles of uneventful driving activity to filter out only the potentially dangerous driving situations where an automated vehicle needs to respond, creating a faster, less expensive testing program. This approach can reduce the amount of testing needed by a factor of 300 to 100,000 so that an automated vehicle driven for 1,000 test miles can yield the equivalent of 300,000 to 100 million miles of real-world driving. While more research and development needs to be done to perfect this technique, the accelerated evaluation procedure offers a ground-breaking solution for safe and efficient testing that is crucial to deploying automated vehicles.