Detecting repeated motion patterns via Dynamic Programming using motion density
ABSTRACT In this paper, we propose a method that detects repeated motion patterns in a long motion sequence efficiently. Repeated motion patterns are the structured information that can be obtained without knowledge of the context of motions. They can be used as a seed to find causal relationships between motions or to obtain contextual information of human activity, which is useful for intelligent systems that support human activity in everyday environment. The major contribution of the proposed method is twofold: (1) motion density is proposed as a repeatability measure and (2) the problem of finding consecutive time frames with large motion density is formulated as a combinatorial optimization problem which is solved via Dynamic Programming (DP) in polynomial time O(N log N) where N is the total amount of data. The proposed method was evaluated by detecting repeated interactions between objects in everyday manipulation tasks and outperformed the previous method in terms of both detectability and computational time.

Conference Paper: Detecting repeated patterns using Partly Locality Sensitive Hashing
[Show abstract] [Hide abstract]
ABSTRACT: Repeated patterns are useful clues to learn previously unknown events in an unsupervised way. This paper presents a novel method that detects relatively long variablelength unknown repeated patterns in a motion sequence efficiently. The major contribution of the paper is twofold: (1) Partly Locality Sensitive Hashing (PLSH) [1] is employed to find repeated patterns efficiently and (2) the problem of finding consecutive time frames that have a large number of repeated patterns is formulated as a combinatorial optimization problem which is solved via Dynamic Programming (DP) in polynomial time O(N<sup>1+1/α</sup>) thanks to PLSH where N is the total amount of data. The proposed method was evaluated by detecting repeated interactions between objects in everyday manipulation tasks and outperformed previous methods in terms of accuracy or computational time.Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on; 11/2010  SourceAvailable from: Keyi Wang
Article: Mixed Signature: An Invariant Descriptor for 3D Motion Trajectory Perception and Recognition
[Show abstract] [Hide abstract]
ABSTRACT: Motion trajectory contains plentiful motion information of moving objects, for example, human gestures and robot actions. Motion perception and recognition via trajectory are useful for characterizing them and a flexible descriptor of motion trajectory plays important role in motion analysis. However, in the existing tasks, trajectories were mostly used in raw data and effective descriptor is lacking. In this paper, we present a mixed invariant signature descriptor with global invariants for motion perception and recognition. The mixed signature is viewpoint invariant for local and global features. A reliable approximation of the mixed signature is proposed to reduce the noise in highorder derivatives. We use this descriptor for motion trajectory description and explore the motion perception with DTW algorithm for salient motion features. To achieve better accuracy, we modified the CDTW algorithm for trajectory matching in motion recognition. Furthermore, a controllable weight parameter is introduced to adjust the global features for tasks in different circumstances. The conducted experiments validated the proposed method.Mathematical Problems in Engineering 01/2012; 2012. · 1.08 Impact Factor 
Conference Paper: Learning Motion Patterns from Multiple Observations along the Action Phases of Manipulative Tasks
[Show abstract] [Hide abstract]
ABSTRACT: In this work we present a probabilistic approach to find motion patterns in manipulative tasks by looking for similarities among the relevant features along of the actions phases of a trajectories dataset. From multiples observations of human movements we can align all signals temporally to perform a learning process based on selection of relevant features by analyzing their probability distribution and finding correspondent features with high probability in each phase of the trajectories of a dataset. Using the spatiotemporal information of the learned features we can generate a generalized trajectory of the dataset using a polynomial regression to fit the features data by successive approximations. The smoothed trajectory can be used as a prototype/template for matching (1:1) or for classification (1:N) using Bayesian techniques to know if a new observation is similar to a specific task or to recognize a task. The intention here is to have an approach that is able to learn and generalize a specific movement by their patterns to be applied in the future for different contexts. We are not going through the imitation learning part, but we are focusing on the ability of learning to reach some intelligence to approximate a movement generalization, tasks that humans do in a natural and easy way.IEEE/RSJ IROS'2010: Workshop on Grasping Planning and Task Learning by Imitation; 10/2010
Page 1
Detecting Repeated Motion Patterns
via Dynamic Programming using Motion Density
Koichi Ogawara and Yasufumi Tanabe and Ryo Kurazume and Tsutomu Hasegawa
Abstract—In this paper, we propose a method that detects
repeated motion patterns in a long motion sequence efficiently.
Repeated motion patterns are the structured information that
can be obtained without knowledge of the context of motions.
They can be used as a seed to find causal relationships between
motions or to obtain contextual information of human activity,
which is useful for intelligent systems that support human
activity in everyday environment.
The major contribution of the proposed method is twofold:
(1) motion density is proposed as a repeatability measure and
(2) the problem of finding consecutive time frames with large
motion density is formulated as a combinatorial optimization
problem which is solved via Dynamic Programming (DP) in
polynomial time O(N logN) where N is the total amount
of data. The proposed method was evaluated by detecting
repeated interactions between objects in everyday manipulation
tasks and outperformed the previous method in terms of both
detectability and computational time.
I. INTRODUCTION
Supporting daily life is one of the upcoming key applica
tions of robotics technology and the capability to understand
human activity is essential for this purpose. One of the com
mon approaches to understand human activity is to design
a set of necessary and sufficient taskdependent recognizers
that detect significant actions as well as capture the necessary
parameters to describe the action [1], [2], [3]. However, the
variety of human activity in everyday environment is very
diverse as opposed to those in a welldesigned environment
such as a factory, it is not practical to prepare recognizers to
cover all of them.
For this reason, a desirable system should have a mecha
nism to learn new knowledge, new recognizers, from obser
vation incrementally. As a bootstrap process to realize this
mechanism, we are trying to detect repeated motion patterns
as a structured information in observation data that can be
extracted without knowledge of the context of a task. The
basic idea is if a particular motion pattern appears many
times in observation data, this pattern must be meaningful
to the user or to the task. When a system detects repeated
motion patterns while observing daily activity for a long
period of time, e.g. several days, these patterns can be
considered as meaningful actions and the causal relationships
between these actions can be used to predict the next action
of a user or to learn contextual information of human activity,
K. Ogawara is with Faculty of Engineering, Kyushu University, Fukuoka,
JAPAN ogawara@is.kyushuu.ac.jp
Y. Tanabe is with Department of Electrical Engineering and Computer
Science, Kyushu University, Fukuoka, JAPAN
R. Kurazume and T. Hasegawa are with Faculty of Information Science
and Electrical Engineering, Kyushu University, Fukuoka, JAPAN
which is useful for intelligent systems that support human
activity in everyday environment.
In this paper, we propose a method that detects previously
unknown repeated motion patterns in a long motion sequence
efficiently. The contribution of the proposed method is two
fold. The first contribution is to introduce a notion of
motion density as a repeatability measure which evaluates the
number of similar motions for each time frame. The second
contribution is that the problem of finding variablelength re
peated motion patterns, that is consecutive time frames with
large motion density, is formulated as a combinatorial opti
mization problem which is solved via Dynamic Programming
(DP) in polynomial time O(N logN) in average where N is
the total amount of data, while most of the previous methods
assume knownlength patterns [4] or take O(N2) time [5].
Among few exceptions is [6] which takes O(N1+1/α) time,
however it solves the problem in threesteps. On the other
hand, the proposed method finds repeated motion patterns
under a single energy minimization framework.
The remainder of this paper is organized as follows. In
Section II, we give an overview of the related research.
Then we introduce a notion of motion density in Section
III and extend the framework to cover interrelated multiple
time series data for our particular application in Section IV.
The details of the proposed method is explained in Section
V. The experimental results are presented in Section VI and
we conclude the paper in Section VII.
II. RELATED WORK
T
Trajectory
Fig. 1.Repeated Patterns in a Time Series Data
While there has been a large body of work regarding
efficiently locating previously known patterns in time series
data [7], [8], [9], we focus on locating previously unknown
repeated patterns in this study as shown in Fig. 1. Finding
unknown repeated patterns, or motifs, has been a wellknown
task in the bioinformatics community [10], in the data mining
community [4], [11] and in the motion analysis community
[12], [5], [6].
Finding repeated patterns has been studied extensively in
analysis of nucleic acid sequences decades ago where the
time series data is a sequence of discrete symbols from
2009 IEEE International Conference on Robotics and Automation
Kobe International Conference Center
Kobe, Japan, May 1217, 2009
9781424427895/09/$25.00 ©2009 IEEE1743
Page 2
a set of 4 bases. Staden proposed a method that detects
the most repeated knownlength pattern by a voting scheme
considering all the possible combination of patterns [10]. The
computational time is linearly proportional to N, where N
is the total amount of data.
The continuous counterpart has received much attention in
the data mining community as well as in the motion analysis
community. In the early stages, Liu et al. proposed a method
in that a continuous time series data is first discretized into
a sequence of symbols and the most repeated knownlength
pattern is found probabilistically by a voting scheme using
a hash function in O(N) time [4].
To deal with unknownlength repeated patterns, several
approaches based on Dynamic Programming (DP) have been
proposed. Ogawara et al. proposed a method that detects a set
of motion patterns appeared in all M observation data in the
same order using multidimensional DP matching in O(NM)
time [13]. Uchida et al. proposed a method that detects pair
wise similar motion patterns from a single observation data
using logical DP matching in O(N2) time [5].
To reduce computational time, Yankov et al. proposed a
method that extends [4] to deal with uniform scaling of
the knownlength repeated motion patterns in O(N1+1/a)
time [11]. To find totally unknownlength repeated motion
patterns, Meng et al. proposed a threesteps method in
O(N1+1/a) time in that it finds knearestneighbor data
points using Locality Sensitive Hashing (LSH), then it con
nects these detected data points along time axis to find
repeated motion candidates and finally it performs spectral
clustering to find the repeated motion patterns [6]. However,
the output from each step cannot be guaranteed to be the
global optimum.
In contrast to Meng’s method, the proposed method finds
repeated motion patterns from a continuous time series
data under a single energy minimization framework in
O(N logN) time, thus it finds the global optimum in terms
of the given energy function. Another advantage is that it can
detect consistent repeated motion patterns among interrelated
multiple continuous time series data which is not possible
with other methods.
Ogawara et al. previously proposed a method that finds
consistent repeated motion patterns among interrelated mul
tiple continuous time series data in O(N logN) time [14].
However, the problem is formulated as a combinatorial
optimization problem with two correlated parameters. These
two parameters are estimated alternately but independently,
thus the global optimum is not guaranteed to be obtained.
In the proposed method, the problem is reformulated as a
combinatorial optimization problem with a single parameter
and the global optimum is analytically obtained via Dynamic
Programming (DP).
Finding repeated patterns can be seen as a data com
pression problem where repeated patterns would be coded
with a shorter word. Zhao et al. proposed a method that
compresses a vector quantized motion data using Huffman
coding which minimizes both the amount of information to
code the entire data and the size of the dictionary under
Minimum Description Length (MDL) principle [12]. It works
well on ballet motion, however the result is severely affected
by nonrepeated patterns which dominates a time series data
in general.
III. MOTION DENSITY
Fig. 1 shows an example of the problem of interest. Given
a long time series data of arbitrary dimensions, we want
to find unknown repeated patterns efficiently where minor
variation of shape and length is allowed.
Here, we introduce a notion of motion density as a
repeatability measure which evaluates the density of similar
motions at around each time frame. With that, the problem
is formulated as to find consecutive time frames with large
motion density under an energy minimization framework.
Motion density md(x,t) represents the density of similar
motion segments in the neighborhood of data point ox,t at
time t in time series data x. In this study, motion density
is defined as the number of motion segments found within
radius R around ox,texcluding the one to which ox,tbelongs:
md(x,t) = #{msjox,t/ ∈ msj,msj⊂ ∪ox,k,ox,k−ox,t < R}
(1)
where motion segment msjis a continuous sequence of ox,k
without a gap. If motion density is high, that means there
are many repeated motion patterns similar to the one around
ox,t.
(1) Mo?on Density is high(2) Mo?on Density is low
(3) Mo?on Density is high
1 ,tx o
Mo?on Segments
in the neighborhood
Mo?on Segment
of Interest
Pose Space
(4) Mo?on Density is low
Shape Space
2
,tx o
3
,tx o
3
,tx o
R
Fig. 2. Motion Density
Fig. 2 shows examples of motion density. The curve in Fig.
2 (1) to (3) represent the 2D slice of a time series data of
arbitrary dimensions in Pose Space. Here, we assume that the
time series data represents a trajectory, e.g. 6 D.O.F. position
of a rigid object or joint angles of a human body, and Pose
Space is the space where the time series data is directly
projected without conversion.
1744
Page 3
1. Normaliza?on
T
2. Capturing Local Shape Informa?on
1
1
t
DCT_WIDTH
3. Applying DCT of Gaussian
Dimensionality Reduc?on
in ith parameter space
Dim. = DCT_BANK x 6
DCT_BANK
4. Applying PCA
Dimensionality Reduc?on
in the en?re parameter space
PCA_BANK
X
Cosine table
Dim. = 6
Dim. = DCT_WIDTH x 6
Dim. = PCA_BANK
T
1
1
Mo?on Density in Shape Space
Mo?on Density in Pose Space
Fig. 3. Conversion from Pose Space to Shape Space
In the case of Fig. 2 (1), there are several similar motion
segments in the neighborhood of ox,t1, 2 in this case, thus the
motion density mdp(x,t1) is relatively high, On the other
hand, in the case of Fig. 2 (2), there is no other motion
segment in the neighborhood of ox,t2, thus the motion density
mdp(x,t2) is low.
In the case of Fig. 2 (3), there are several motion segments
in the neighborhood of ox,t3, thus mdp(x,t3) is high.
However, as shown in the figure, the shape of these motion
segments are completely different, thus these should not be
considered as repeated motion patterns.
To take the similarity of shape into consideration, another
space named Shape Space is defined. In this space, each data
point at time t encodes the local shape of the original time
series data at around ox,tso that data points similar in shape
locally are projected to be in the neighborhood.
Motion density is also computed in Shape Space and both
of the motion densities are used as a repeatability measure
at time t. Fig. 2 (4) shows the converted trajectory of Fig. 2
(3) into Shape Space and motion density mds(x,t3) is low
in this space.
A. Definition of Shape Space
In Shape Space, the metric between two data points is
computed as the similarity in shape between two motion
segments around the data points. The conversion from Pose
Space to Shape Space is made of 4 steps as shown in Fig.
3.
Here, we assume that the original time series data in Pose
Space represents a trajectory of an object and each data point
is made of 6 parameters, 3 for translation and 3 for rotation
(quaternion). However, the following conversion generalizes
to any parameterization including joint angles.
1. Building KdTree
2. Finding data points in the neighborhood
3. Embedded segmenta?on algorithm in Kdtree search
Table
Mo?on Segment
Link List
Start
End
Start
End
Start
End
R
tx o,
…
…
t
t1
K1
K+1
K
K+1K K1
t1 t
Tbl[i]
.Visit_count
.index
.p_MS
Fig. 4. Kdtree with Embedded Segmentation Algorithm
1) Normalization
Because the range of values is totally different among
the parameters, translation v.s. rotation, the values are
normalized in each parameter space independently.
Motion density in Pose Space is computed from the
normalized data points.
2) Capturing Shape Information
To capture the local shape information, DCT WIDTH
normalized data points around time t are gathered to
form a shape vector in ith parameter space. The num
ber of dimension is increased from 6 to DCT WIDTH
× 6.
3) Applying DCT of Gaussian
Discrete Cosine Transform (DCT) of Gaussian is ap
plied to the shape vectors for dimensionality reduction
in ith parameter space. The number of dimension is
reduced to DCT BANK × 6.
4) Applying PCA
Principal Component Analysis (PCA) is applied to
the set of reduced shape vectors for dimensionality
reduction in the entire parameter space. The number
of dimension is further reduced to PCA BANK.
We can directly apply PCA to a shape vector without ap
plying DCT of Gaussian. The reason why DCT of Gaussian
is applied intermediately is that PCA in high dimensional
space takes time. PCA after DCT of Gaussian dramatically
reduces the computational time, while the quality of dimen
sionality reduction is comparable with PCA only approach.
From the viewpoint of computational complexity, the time
for applying PCA is considered to be a constant, but it makes
the algorithm considerably faster in practice.
B. Finding Motion Segments using Kdtree
To count the number of motion segments within the hyper
sphere of radius R around a data point, Kdtree with an
embedded segmentation algorithm is used. The algorithm
works as shown in Fig. 4.
1745
Page 4
1) Building Kdtree
Two Kdtrees are built separately for the set of data
points both in Pose Space and in Shape Space. The data
points are recursively subdivided by a hyperplane per
pendicular to the axis along which the variance of the
remaining data points becomes maximum. The com
putational time for building Kdtree is O(N logN).
2) Finding Data Points in the Neighborhood
For each data point at time t, all the data points within
the hypersphere of radius R are searched. The average
computational time is close to O(logN), however it
becomes O(N) in the worst case if all the data points
lie within the hypersphere.
3) Embedded Segmentation Algorithm in Kdtree
Search
After finding all the data points within the hyper
sphere, these points have to be grouped into segments.
However, if segmentation is performed separately after
Kdtree search finishes, additional time is required to
sort the data points whose computational time is ether
O(N) or O(K logK) where K is the number of the
data points found within the hypersphere.
Thus, a segmentation algorithm is embedded in the
search algorithm of Kdtree without increasing com
putational complexity. As shown in Fig. 4, it maintains
a table of length N each of which holds 3 parameters:
(1) visit count is used to check if it is already visited
of not, (2) index holds the tableindex of another data
point that belongs to the same motion segment and
(3) p MS is a pointer to the corresponding Motion
Segment structure. It also maintains the linkedlist of
Motion Segment structure each of which holds the start
and end time of the motion segment.
The flow of the algorithm is presented in TABLE I.
This algorithm is called with the time parameter t
whenever a new data point within radius R is found
during a single Kdtree search. If the mean number
of data points around a data point is ¯K, then this
is called ¯K × N times in total. For each call, the
set of Motion Segment structure is updated using one
of the following functions: mergeMotionSegment() is
to merge two consecutive segments together, enlarge
MotionSegment() extends the range of the segment to
include time t and addNewMotionSegment() creates a
new segment at time t. visit count is incremented once
in each Kdtree search, that is N times.
When all the data points within the hyper sphere are
visited, the data points are already grouped into motion
segments and motion density is calculated by just
counting the number of segments.
C. Consistency between Motion Densities
Since repeated motion patterns must appear in both motion
densities, a motion segment that is not shared by both of
the motion densities is removed. Using remaining motion
segments, motion density is calculated as eq.(1).
TABLE I
EMBEDDED SEGMENTATION ALGORITHM
1
2
3
4
5
6
7
8
9
Input: t;
idxl = idxr = t;
if tbl[t1].visit count == visit count
idxl = t1;
do (idxl = tbl[idxl].index)
while (idxl != tbl[idxl].index);
tbl[t1].index = idxl;
end;
if tbl[t+1].visit count == visit count
idxr = t+1;
do (idxr = tbl[idxr].index)
while (idxr != tbl[idxr].index);
tbl[t+1].index = idxr;
end;
if idxl != t && idxr != t
mergeMotionSegment(tbl[idxl].p MS, tbl[idxr].p MS);
tbl[t].idx = idxl;
else if idxl != t
enlargeMotionSegment(tbl[idxl].p MS, t);
tbl[t].idx = idxl;
else if idxr != t
enlargeMotionSegment(tbl[idxr].p MS, t);
tbl[t].idx = idxr;
else
addNewMotionSegment(t);
tbl[t].idx = t;
end;
tbl[t].visit count = visit count;
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
IV. EXTENSION TO INTERRELATED MULTIPLE
CONTINUOUS TIME SERIES DATA
T
T
T
Object 1
Object 2
Object 3
T
Trajectory
Conflict
NN A N A CA C N N N NN NNAA C C N
Data Label
Fig. 5. Finding Consistent Repeated Motion Patterns
So far, we implicitly assume a single time series data.
Here, we extend our framework to cover interrelated multiple
continuous time series data. Fig. 5 shows an example of the
problem. The input is multiple trajectories of objects, 3 in
this case, and the algorithm is required to output the motion
patterns that appear many times in the input trajectories.
If the multiple trajectories are independent of each other,
then the algorithm has only to be applied to each trajectory
separately. However, if the trajectories are interrelated with
each other, the algorithm has to find the consistent repeated
motion patterns among the trajectories.
This is the case when repeated interactions between ob
jects are of interest. In Fig. 5, there are 3 possible interactions
1746
Page 5
between: (A) Obj.1 and Obj.2, (B) Obj.1 and Obj.3, (C)
Obj.2 and Obj.3. If two of them are detected at the same
time, this means a conflicting situation.
In the proposed method, we assume that there occurs a
single significant interaction at most at a certain moment.
This assumption is not necessarily true when there are 4 or
more objects, or a motion is composed of more than two
objects, but it is satisfied in most cases.
Then, the problem is formulated as a combinatorial opti
mization problem in that a data label x ∈ {A,B,C,Non}
is assigned to each time frame as shown in the bottom of
Fig. 5.′′A′′,“B′′,“C′′mean there occurs a repeated motion
between the respective object pairs, while′′Non′′means
there is nonrepeated motion at that time.
Please note that this formulation is a natural extension
from the single time series data case in that we have only
two labels: repeated and notrepeated.
V. ALGORITHM DETAILS
A. Problem Formulation
The problem is formulated as a combinatorial opti
mization problem regarding data labels X = {xtxt ∈
{A,B,··· ,Non},1 ≤ t ≤ N} as shown in Fig. 5. There
are two types of data labels:′′A′′,“B′′,··· mean the current
data point is a repeated motion belonging to the time series
data of that label, while′′Non′′means the current data point
is nonrepeated motion.
In the case of finding consistent interactions between
objects, there are M = m(m − 1)/2 data labels where m
is the number of objects in the scene and data point oi,t
represents relative position between ith object pairs at time
t.
Given multiple time series data TS = {oi,t1 ≤ i ≤
M,1 ≤ t ≤ N}, we find X that minimizes an energy
function defined as
E(TS,X) = Ev(TS,X) + Em(TS,X) + Es(X)
(2)
Ev(TS,X) is a term regarding velocityi,t, that is the
velocity of the relative position between ith object pairs at
time t, and it penalizes stationary or noninteracted objects.
In our particular application, objects stay still in most of
the period in a time series data, thus we have to avoid the
case where huge number of motionless regions are detected
as repeated motion patterns. Otherwise, this term is not
necessary. It is decomposed into
Ev(TS,X) =
?
t
ev(TS,xt,t)
ev(TS,xt,t) = −log(1 − exp(−
velocityxt,t
< velocityxt,t>))
where < velocityxt,t> is the mean value of velocityxt,t.
Em(TS,X) is a term regarding motion density and it
penalizes data points with small number of similar motion
segments in the neighborhood. It is decomposed into
Em(TS,X) =
?
t
em(TS,xt,t)
em(TS,xt,t)=−log(1 − exp(−
mdp(xt,t)
< mdp(xt,t) >))
mds(xt,t)
< mds(xt,t) >))−log(1 − exp(−
where mdp(xt,t) and mds(xt,t) are the motion densities
defined in Pose Space and Shape Space. < mdp(xt,t) >
and < mds(xt,t) > are the mean values of mdp(xt,t) and
mds(xt,t) respectively.
Es(X) is a term regarding the prior distribution of X.
It penalizes the difference between consecutive data labels
which leads to reject short motion segments. It is decom
posed into
Es(X) =
?
t
es(xt,xt+1)
e(xt,xt+1) = T(xt?= xt+1) · K,
where
true, otherwise T(s) = 0.
K isaconstant and
T(s)=1 iff s=
B. Energy Minimization via Dynamic Programming
A
A
B
???
???
???
???
??????
),(
1
+
tts
xxe
),(),(
tmtv
xTSexTSe
+
NN N N A CN C N N N N
N NNAAA A B
Data Label
Object 1 Object 2
Object 1 Object 3
Non
Find the shortest PATH
N means “Non”
Fig. 6.Estimation of Data Labels via Dynamic Programming
All the terms in eq.(2) satisfies first order Markov property,
the energy minimization problem can be analytically solved
by Dynamic Programming as shown in Fig. 6.
The trouble is we know nothing about the energy terms Ev
and Emregarding data label′′Non′′. In general, the lower
these energy terms are, the fewer the number of detected
repeated motion patterns be. However, it is not clear how
these terms are determined, since the appropriate value is
context dependent. So we decided to adaptively change the
values of these terms so as to output the desired number of
motion patterns which is provided by a user.
First, the range of the value ev(TS,xt,t)+em(TS,xt,t)
is calculated by checking all the combination of t and xt.
Then, assuming eNon,t= ev(TS,Non,t)+em(TS,Non,t)
is a fixed value independent of t, the optimal eNon,twithin
the range is obtained in binarysearch fashion so as to best
outputs the desired number of repeated motion patterns via
Dynamic Programming.
1747
Page 6
C. Separation of Motion Patterns on a Same Time Series
Data
So far, different motion patterns on a same time series data
are labelled as the same. To separate these motion patterns,
the following algorithm is applied.
A A AAAA
AA A A A
A
A
A
A
A
1
ms
2
ms
3
ms
A A AAAA
AA A A A
A
A
A
A
A
1
ms
2
ms
3
ms
Nearest data point within radius R
(1) Overlap Ra?o from to (2) Overlap Ra?o from to
1
ms
2
ms
2
ms
1
ms
radius R
radius R
Fig. 7. Separation of motion patterns using mutual overlap ratio
Suppose one of the trajectory is labelled as in Fig. 7 and
3 motion segments are detected via DP. Then, for each pair
of motion segments msiand msj, the overlap ratio ORi,jis
calculated as the number of nearest data points within radius
R in msj divided by the length of msi. If mutual overlap
ratio, both ORi,jand ORj,i, is greater than 0.5, two motion
segments are grouped as the same motion pattern.
In the case of Fig. 7, OR1,2=
ms1and ms2are grouped as the same.
The nearest data point on each motion segment within
radius R is computed during Kdtree search, thus there is no
increase in computational complexity.
4
5and OR2,1=
4
6, thus
D. Computational Complexity
The required computational time is O(N logN) for build
ing Kdtree, O(N logN) in average for motion density esti
mation and O(M2N) for DP analysis. By assuming M <<
N, the total average computational time is O(N logN).
VI. EXPERIMENTAL RESULT
A. Experimental Setup
4 different manipulation tasks were performed by a subject
and were used to evaluate the proposed method. There
were 4 objects in the scene and an electromagnetic motion
tracking system (Polhemus FASTRAK) was used to observe
the trajectory of each object at 30Hz during demonstrations.
Fig. 8 shows the objects used in this experiment.
As shown in Fig. 9, 6 actions are defined. A subject
was instructed to perform a task following a scenario made
by combining 6 actions. To emulate the change in the
environment during longterm observation, the subject was
instructed to relocate the objects on the table several times
so that the relative relationship between stationary objects
was changed.
B. Evaluation
In all the experiments, we use 33 for DCT WIDTH, 4
for DCT BANK and 6 for PCA BANK so as to equalize
the dimensions of Pose Space and Shape Space. Both of the
radius of the hypersphere in Pose Space and in Shape Space
Obj.1
Obj.2
Obj.3
Obj.4
Fig. 8.Objects used in the Experiment
A1: Pour from Obj.1 to Obj.2 from the le?
A2? Pour from Obj.1 to Obj.2 from the right
B1: Pour from Obj.3 to Obj.2
C1: Mix inside Obj.2 with Obj.4
C2: Spoon up from Obj.1 with Obj.4
C3: Put into Obj.2 with Obj.4
A1A2B1 C1C2 C3
Fig. 9. 6 Repeated Actions in the Scenario
is 0.4 and K in Esterm is 15. These values were determined
empirically.
The proposed method was compared with the method
proposed in [14]. There are two noticeable differences be
tween them. The first difference is that the desired number of
motion patterns is provided by a user in the proposed method,
while it is automatically selected in the previous method,
though it is highly sensitive to the parameters. The second
difference is that motion patterns in a same time series data
can be discriminated in the proposed method, while it is not
possible in the previous method.
TableII, III, IV, V shows the results of detecting repeated
motion patterns using two different methods. All the com
putations were done on Xeon 3.0GHz PC.
As an error measure, Precision and Recall are calculated
from True Positive (TP), False Positive (FP) and False
Negative (FN) as
Precision =
TP
TP + FP, Recall =
TP
TP + FN.
The proposed method outperformed the previous method
in all the scores. However, there is a noticeable failure in
TABLE V where the number of detected action C3 is zero.
This is because, in this scenario, C3 always follows just after
C2 and these two consecutive actions were detected as a
single action which were labelled as C2.
VII. CONCLUSION
This paper presents a method that detects repeated motion
patterns from a continuous time series data under a single
energy minimization framework in O(N logN) time where
1748
Page 7
TABLE II
EVALUATION OF DATA SET 1 [896 FRAMES]
Action
Presented Number
A1
4TPFNFP Precision RecallTime[msec]
Proposed Method
Previous Method [14]
4
4
4
4
0
0
0
0
1.00
1.00
1.00
1.00
1164
15873
TABLE III
EVALUATION OF DATA SET 2 [1428 FRAMES]
Action
Presented Number
A1
3
A2
3 TP FNFP Precision RecallTime[msec]
Proposed Method
Previous Method [14]
336
5
0
1
0
2
1.00
0.71
1.00
0.83
2401
806245
TABLE IV
EVALUATION OF DATA SET 3 [3737 FRAMES]
Action
Presented Number
A1
6
B1
7
C1
6 TPFN FP PrecisionRecall Time[msec]
Proposed Method
Previous Method [14]
5
0
7
6
6
2
18
8
1
11
0
0
1.00
1.00
0.95
0.42
14800
1769685
TABLE V
EVALUATION OF DATA SET 4 [5177 FRAMES]
Action
Presented Number
B1
7
C1
5
C2
10
C3
10TP FN FPPrecisionRecall Time[msec]
Proposed Method
Previous Method [14]
7
5
36
5
0 16
10
16
22
5
4
0.76
0.71
0.50
0.31
26703
8388985
N is the total amount of data. The algorithm outputs top
L repeated motion patterns, where the desired number L is
provided by a user.
The problem is formulated as a combinatorial optimization
problem in that the data label with large motion density
is assigned to each time frame, which then be solved via
Dynamic Programming (DP) analytically.
The proposed method was evaluated by detecting repeated
interactions between objects in everyday manipulation tasks
and outperformed the previous method[14] in terms of both
detectability and computational time.
ACKNOWLEDGMENTS
This study was supported by Program for Improvement of
Research Environment for Young Researchers from Special
Coordination Funds for Promoting Science and Technology
(SCF) commissioned by the Ministry of Education, Culture,
Sports, Science and Technology (MEXT) of Japan.
REFERENCES
[1] K. Ikeuchi and T. Suehiro, “Toward an assembly plan from observation
part i: Task recognition with polyhedral objects,” IEEE Trans. Robotics
and Automation, vol. 10, no. 3, pp. 368–384, 1994.
[2] Y. Kuniyoshi, M. Inaba, and H. Inoue, “Learning by watching,” IEEE
Trans. Robotics and Automation, vol. 10, no. 6, pp. 799–822, 1994.
[3] K. Bernardin, K. Ogawara, K. Ikeuchi, and R. Dillmann, “A sensor
fusion approach for recognizing continuous human grasping sequences
using hidden markov models,” IEEE Transactions on Robotics, vol. 21,
no. 1, pp. 47–57, 2005.
[4] J. Lin, E. Keogh, S. Lonardi, and P. Patel, “Finding motifs in time
series,” in Proc. of the 2nd Workshop on Temporal Data Mining, 2002,
pp. 53–68.
[5] S. Uchida, A. Mori, R. Kurazume, R. Taniguchi, and T. Hasegawa,
“Logical dp matching for detecting similar subsequence,” in Proc. of
Asian Conference of Computer Vision, 2007.
[6] J. Meng, J. Yuan, M. Hans, and Y. Wu, “Mining motifs from human
motion,” in Proc. of EUROGRAPHICS’08, 2008.
[7] R. Agrawal, C. Faloutsos, and A. Swami, “Efficient similarity search
in sequence databases,” in Proc. of 4th International Conference on
Foundations of Data Organization and Algorithms, 1993, pp. 69–84.
[8] C.S. Perng, H. Wang, S. R. Zhang, and D. S. Parker, “Landmarks:
A new model for similaritybased pattern querying in time series
databases,” in 16th International Conference on Data Engineering
(ICDE’00), 2000, pp. 33–42.
[9] T. Mori, Y. Nejigane, M. Shimosaka, Y. Segawa, T. Harada, and
T. Sato, “Online recognition and segmentation for timeseries motion
with hmm and conceptual relation of actions,” in Int. conf. on
Intelligent Robots and Systems, 2005, pp. 2569–2574.
[10] R. Staden, “Methods for discovering novel motifs in nucleic acid
sequences,” Computer Applications in the Biosciences, vol. 5, no. 5,
pp. 293–298, 1989.
[11] D. Yankov, E. Keogh, J. Medina, B. Chiu, and V. Zordan, “Detecting
time series motifs under uniform scaling,” in Proc. of the 13th ACM
KDD Intl. Conf. on Knowledge Discovery and Data Mining, 2007, pp.
844–853.
[12] T. Zhao, T. Wang, and H. Shum, “Learning a highly structured motion
model for 3d human tracking,” in Proc. of Asian Conference of
Computer Vision, 2002.
[13] K. Ogawara, J. Takamatsu, H. Kimura, and K. Ikeuchi, “Extraction
of essential interactinos through multiple observations of human
demonstrations,” IEEE Transactions on Industrial Electronics, vol. 50,
no. 4, pp. 667–675, 2003.
[14] K. Ogawara, Y. Tanabe, R. Kurazume, and T. Hasegawa, “Learning
meaningful interactions from repetitious motion patterns,” in IEEE/RSJ
2008 Int. Conf. on Intelligent Robots and Systems, 2008, pp. 3350–
3355.
1749
View other sources
Hide other sources
 Available from Ryo Kurazume · Jun 1, 2014
 Available from ac.jp