Conference PaperPDF Available

Real-time kinematic modeling and prediction of human joint motion in a networked rehabilitation system

Authors:

Abstract

In this paper, a networked-based rehabilitation system is introduced for lower-extremity tele-rehabilitation. In order to enable high-level motion planning of the rehabilitation robot in real-time for enhanced safety and appropriate human-robot interactions, a time series model is proposed to capture the kinematics of knee joint rotations. A major challenge in such a system is that measurement data might be delayed or lost due to wireless communication. With a delay and loss compensation mechanism, a modified recursive least square (mRLS) algorithm is applied for real-time modeling and prediction of knee joint rotations in the sagittal plane, and convergence of the proposed algorithm is studied. Simulation and experimental results are presented to verify the performance of the proposed algorithm.
Real-time Kinematic Modeling and Prediction of Human Joint Motion
in a Networked Rehabilitation System
Wenlong Zhang, Xu Chen, Joonbum Bae, and Masayoshi Tomizuka
Abstract In this paper, a networked-based rehabilitation
system is introduced for lower-extremity tele-rehabilitation. In
order to enable high-level motion planning of the rehabilitation
robot in real-time for enhanced safety and appropriate human-
robot interactions, a time series model is proposed to capture
the kinematics of knee joint rotations. A major challenge in such
a system is that measurement data might be delayed or lost due
to wireless communication. With a delay and loss compensation
mechanism, a modified recursive least square (mRLS) algorithm
is applied for real-time modeling and prediction of knee joint
rotations in the sagittal plane, and convergence of the proposed
algorithm is studied. Simulation and experimental results are
presented to verify the performance of the proposed algorithm.
I. INTRODUCTION
In view of the rapidly increasing number of patients and
elderly people who need physical therapy, there is a large
demand for gait rehabilitation and assistive devices. Many
companies and researchers have developed various kinds
of assistive robots to facilitate the rehabilitation treatment
and patients’ daily life [1]–[3]. As rehabilitation systems
become more sophisticated, mobility becomes an important
issue, which motivates us to integrate network media into a
rehabilitation system, as is shown in this paper.
In order to achieve smart rehabilitation treatment, it is
important to understand the user’s joint movement. This has
motivated intensive studies of human motion capture and
analysis with optical sensors [4], inertial sensors [5], and
electromyography (EMG) sensors [6]. For lower-extremity
motion analysis, joint kinematic model is often employed to
detect gait phases [7] for identifying gait abnormality. It is
also frequently used to estimate human motion intention [8]
for trajectory planning of the assistive robot. Joint kinematics
is also particularly useful in fall prediction to enhance the
safety of a rehabilitation robot [9].
Motivated by the importance of joint kinematic modeling,
an autoregressive integrated (ARI) model was built based
on time series analysis to predict knee joint rotation in a
network-based rehabilitation system [10]. While the pro-
posed technique can provide a reliable human knee joint
model for motion prediction, it is difficult to implement
This work was supported by National Science Foundation under Grant
CMMI-1013657.
W. Zhang and M. Tomizuka are with the Department of Mechanical
Engineering, University of California, Berkeley, CA 94720 USA (e-mail:
wlzhang @berkeley.edu;tomizuka@me.berkeley.edu)
X. Chen is with the Department of Mechanical Engineering, University
of Connecticut, Storrs, CT 06269 USA (e-mail: xchen@engr.uconn.edu)
J. Bae is with the School of Mechanical and Nuclear Engineering, Ulsan
National Institute of Science and Technology, Ulsan 689-798, Korea (e-mail:
jbbae@unist.ac.kr)
the algorithm online due to its computational complexity.
It would be ideal if an online adaptive human motion model
can be built based on new measurements from the user. Such
a model is able to capture the change of walking dynamics
more accurately and provide more insights to the user.
In this paper, an online adaptive knee joint rotation model
is built based on a modified recursive least square (mRLS)
algorithm. Since measurement data are transmitted over the
wireless network, network-induced constraints, such as time
delay and packet loss, need to be handled in the modeling
process. A delay and loss compensator is thus proposed with
the proof of convergence in this paper. Simulation and exper-
imental results are demonstrated to verify the effectiveness of
the proposed algorithm. The influences of time delay, packet
loss, and forgetting factors in the algorithm are analyzed
based on the simulation and experimental results.
The remainder of this paper is organized as follows. In
Section II, the networked rehabilitation system is briefly in-
troduced and joint angle measurement for one healthy subject
is presented. Section III proposes the mRLS algorithm for
joint rotation modeling and illustrates how it can handle
the network-induced challenges. Convergence analysis is
presented in Section IV. Section V shows simulation results
and analyzes the performance of the proposed approach.
Experimental results with another healthy subject are shown
in Section VI. Conclusion and future work are given in
Section VII.
II. JOINT ANGLE MEASUREMENT IN A
REHABILITATION SYSTEM
In our previous work [11], a network-based rehabilitation
system was proposed for improved mobility and in-home
tele-rehabilitation. The system consists of a wireless body
sensor network and a rehabilitation robot controlled over
a high-speed wireless network. A computer at the patient’s
home wirelessly connects to all sensors and design the proper
control signals for the robot. The proposed algorithm will be
implemented in the local computer.
In the proposed system, human joint rotation in three
dimensions can be captured by several inertial sensors.
A wireless inertial measurement unit (IMU) is shown in
Fig. 1. An IMU node consists of a three degrees of freedom
accelerometer, magnetometer, and gyroscope. The measure-
ment data can be transmitted to the local computer wirelessly.
The IMU node is powered by a Li-Po battery and it can work
continuously for 90 minutes. The dimension of one wireless
IMU node is 2 inches ×1.4 inches ×0.6 inches and its
weight is around 0.15 lbs including the battery. The wireless
Fig. 1: An inertial measurement unit (IMU) and a human
subject walking on a treadmill with IMUs
IMU can achieve a sampling rate up to 100Hz and it is very
convenient to be attached to a subject’s lower extremities
using velcros, as is shown in Fig. 1.
To examine the performance of the proposed algorithm,
several sets of knee joint rotation measurements were col-
lected. With two IMUs attached to his left thigh and shank,
a 25-year old male user was asked to walk on a treadmill at
1 mph in the Mechanical Systems Control Laboratory at the
University of California, Berkeley. The user had no known
walking abnormalities. The measurement data were recorded
at 50 Hz and representative knee joint angles in the sagittal
plane is shown in Fig. 2. The measurement data will be used
in the modeling and simulation study.
III. HUMAN MOTION MODELING WITH A
MODIFIED RECURSIVE LEAST SQUARE METHOD
A. Problem Formulation and Algorithm Design
In this paper, the following time-varying linear kinematic
model is built
y(k) = ΦT(k)θ(k) + v(k),(1)
where y(k)is the measured human joint angle and Φ (k) =
y(k1) y(k2) · · · y(kn)TRnis the re-
gressor vector that stores the previous measurement of the
joint angles, θ(k) = θ1(k)θ2(k)· · · θn(k)TRn
is a time-varying parameter vector that needs to be estimated.
v(k)is white noise with zero mean and variance σ2
v, and it
is independent of the current and previous regressors, i.e.,
E[v(k)] = 0, E v2(k)=σ2
v<,(2)
E[v(k)v(j)] = 0,k=j, (3)
E (ki)v(k)] = 0,i0.(4)
The proposed model (1) is an autoregressive model, and its
capability of capturing human joint kinematics was validated
in [10]. In order to estimate the parameter vector, the
following least square cost function is considered:
J(k) = min
ˆ
θ(k)
1
2
k
j=1
λkjy(j)ΦT(j)ˆ
θ(k)2
.(5)
0 10 20 30 40 50 60 70 80
0
20
40
60
Time (sec)
Knee joint angle (deg)
Fig. 2: Knee joint angle measurement
This problem can be solved recursively as follows:
eo(k) = y(k)ΦT(k)ˆ
θ(k),(6)
F(k) = 1
λF(k1) F(k1)Φ(kT(k)F(k1)
λT(k)F(k1)Φ(k),(7)
ˆ
θ(k+ 1) = ˆ
θ(k) + F(k) Φ (k)eo(k),(8)
where eo(k)is the a-priori estimation error and F(k)is
the adaptation gain matrix. Note that (7) is equivalent to
F1(k) = λF 1(k1) + Φ (k) ΦT(k).λis a constant
forgetting factor that needs to be selected prior to the
modeling process. Furthermore, it is assumed that the true
parameters follow a random walk process as follows:
θ(k+ 1) = θ(k) + w(k),(9)
where w(k)is a white noise with zero mean and variance
σ2
w, and it is independent of the true parameters, current and
previous regressors, and process noise, i.e.,
E[w(k)] = 0, E wT(k)w(k)=σ2
w<,(10)
Ew(k)wT(j)=0,k=j, (11)
EΦ (ki)wT(k)=0,i0,(12)
E[w(k)v(i)] = 0,i. (13)
The random walk assumption of the true model parameters
follows the observation that the gait pattern of a subject is
consistent between steps with the same walking speed and
road condition. However, there must be some minor differ-
ences of walking behaviors between steps. The differences
are small for healthy subjects but large for patients with gait
abnormalities. Therefore, σ2
wis typically small for healthy
subjects and large for patients.
B. Model Structure Selection
In this paper, the model to be built has the form (1), and the
order of the model is firstly determined. It is shown in [10]
that one can decide the order of the model using Box-Jenkins
approach [12]. Applying this approach to the data shown in
Fig. 2 suggests a ninth-order autoregressive model with a
first-order integration action. Thus, the order of the proposed
model is picked as 10, i.e., Φ (k)R10 and θ(k)R10.
C. Time Delay and Packet Loss
In a network-based rehabilitation system, sensing packet is
transmitted to the estimator over the wireless network. Time
delay may happen during data transmission, making the most
recent sensing packet unavailable for parameter update. A
Sensor
Estimator
Time
s
kT
(
)
1
s
+
(
)
2
s
k T
+
(
)
0
s
k n T
+
(
)
0
1
s
k n T
+ +
k
τ
1
k
τ
+
2
k
τ
+
0
k n
τ
+
Fig. 3: Timing diagram of the measurement packet
representative timing diagram of the sensing packet delivery
is shown in Fig. 3, where τkis the delay of the kth
measurement packet and Tsis the sampling interval of the
system. Similarly, packet loss of sensing packets can also
happen randomly, which leads to irregular parameter update
and makes the convergence analysis challenging.
Time delay and packet loss have two major negative effects
to the parameter estimation. On one hand, the measurement
y(k)might be unavailable for calculating eo(k)using (6).
On the other hand, delayed or lost packet might need to be
used in the regressor Φ (k)to update model parameters and
the adaption gain.
In order to deal with varying time delay during net-
work transmission, the following compensation mechanism
is used. If an output measurement y(k)is not available, the
predicted output ˆy(k)from the identified model is used, and
the parameter will not be updated for this step. The identified
model will also be used to predict missing elements in the
regressor Φ (k). The mRLS algorithm can be expressed as
eo(k) = z(k)ΦT
e(k)ˆ
θ(k),(14)
F(k) = 1
λF(k1) F(k1)Φe(kT
e(k)F(k1)
λT
e(k)F(k1)Φe(k),(15)
ˆ
θ(k+ 1) = ˆ
θ(k) + F(k) Φe(k)eo(k),(16)
where z(k) = γ(k)y(k) + [1 γ(k)] ˆy(k),ˆy(k) =
ΦT
e(k)ˆ
θ(k), and γ(k)Bernoulli (γ)is an indicator of
successful transmission of the measurement packet y(k).
Φe(k) = s(k1) s(k2) · · · s(kn)TRn,
where
s(ki) = βki(k)y(ki) + [1 βki(k)] ˆy(ki).
βki(k)is an availability indicator of measurement packet
y(ki)at the kth time step. Note that a packet will be
available in the regressor several steps later if it is delayed,
but it will not be available if it is lost without retransmission.
IV. CONVERGENCE ANALYSIS
In this section, performance of the proposed mRLS algo-
rithm is examined analytically by deriving the upper bound
of the parameter estimation errors. Based on system model
(1), assumptions for v(k)and w(k)in (2)-(4) and (10)-
(13), and mRLS algorithm (14)-(16), the following further
assumptions are made for derivation of the error bounds [13]:
(A1) For some constants 0< α β < and an integer
Nn, the following strong persistent excitation
condition holds:
αIn1
N+ 1
N
i=0
Φe(k+i) ΦT
e(k+i)βIn,(17)
where Inis an n-dimensional identity matrix.
(A2) The estimator gain matrix is initialized as F(1) =
p0In, where 1λ
(N+1)βp01λ
α.
(A3) The parameter estimate is initialized as ˆ
θ(0) = c01n,
where c0is a small positive number and 1nis an n-
dimensional vector with all elements equal to 1. In
addition, ˆ
θ(0) is independent of v(k).
(A4) Parmeter estimation error is defined as
˜
θ(k) = ˆ
θ(k)θ(k),(18)
where ˜
θ(0) satisfies E˜
θT(0) ˜
θ(0)=δ0<.
The assumptions above lead to the following lemma:
Lemma 1: For the system described in (1) and mRLS
algorithm expressed in (14)-(16), if assumption (A1) holds,
then for kNand 0<λ<1the adaption gain matrix
F(k)satisfies
λNα
1λIn+λk+1 F1(1) α
1λInF1(k)(19)
(N+ 1) β
1λIn+λk+1 F1(1) (N+ 1) β
1λIn.
Proof: The proof is similar to that of Lemma 1 in [13]
and is therefore omitted.
Note from Lemma 1 that if both (A1) and (A2) are
satisfied, the following inequality is satisfied
1λ
(N+ 1) βInF(k)1λ
λNαIn(20)
for kN. Now the theorem that addresses the upper bounds
of parameter estimate errors can be provided.
Theorem 1: Consider the system described in (1), mRLS
algorithm expressed in (14)-(16), and v(k)and w(k)satis-
fying (2)-(4) and (10)-(13), respectively. If the assumptions
(A1)-(A4) are satisfied, the expected norm of estimation error
˜
θ(k)satisfies the following upper bound for all Nk < .
E
˜
θ(0),
γ(1),··· (k),
β0(1),··· k(k)
[
˜
θ(k)
2]α2p2
0λ2(kN+1) (1 λ)2δ0(21)
+1λ
λNαnB (k) + n(1λ)
αλN1σ2
v+(N+1)2β2
α2λ2(N1)(1λ)2σ2
w.
where αand βare defined in (A1), and B(k)depends on
the performance of network transmission.
Proof: Based on definition in (18) and mRLS algorithm
(14)-(16), the following equations are obtained:
˜
θ(k+ 1) = ˆ
θ(k+ 1) θ(k+ 1) (22)
=ˆ
θ(k+ 1) [θ(k) + w(k)]
=˜
θ(k) + F(k) Φe(k)v(k)F(k) Φe(k) ΦT
e(k)˜
θ(k)
+F(k) Φe(k) (k)Φe(k)]Tθ(k)w(k).
By defining (k) = Φe(k)Φ (k), the equation above can
be written as
˜
θ(k+ 1) = ˜
θ(k)w(k) + F(k) Φe(k)v(k)(23)
F(k) Φe(k) ΦT
e(k)˜
θ(k)F(k) Φe(k) T(k)θ(k)
=InF(k) Φe(k) ΦT
e(k)˜
θ(k) + F(k) Φe(k)v(k)
F(k) Φe(k) T(k)θ(k)w(k).
Note (15) yields Φe(k) ΦT
e(k) = F1(k)λF 1(k1).
Thus, the equation above can be rewritten as
˜
θ(k+ 1) = λF (k)F1(k1) ˜
θ(k) + F(k) Φe(k)v(k)(24)
F(k) Φe(k) T(k)θ(k)w(k).
Induction of (15) and (24) yields
F1(k1) = λF 1(k2) + Φe(k1) ΦT
e(k1) ,(25)
˜
θ(k) = λF (k1) F1(k2) ˜
θ(k1) (26)
+F(k1) Φe(k1) v(k1) w(k1)
F(k1) Φe(k1) T(k1) θ(k1) .
Plugging in F1(k1) and ˜
θ(k)into (24) results in the
following equation
˜
θ(k+ 1) = λ2F(k)F1(k2) ˜
θ(k1) (27)
F(k) Φe(k) T(k)θ(k)
λF (k) Φe(k1) T(k1) θ(k1)
+λF (k) Φe(k1) v(k1) + F(k) Φe(k)v(k)
λF (k)F1(k1) w(k1) w(k).
Continuing the induction yields
˜
θ(k+ 1) = λk+1F(k)F1(1) ˜
θ(0) (28)
k
i=0
λkiF(k) Φe(i) T(i)θ(i)
k
i=0
λkiF(k)F1(i)w(i)
+
k
i=0
λkiF(k) Φe(k)v(i)
=α1(k) + α2(k) + α3(k) + α4(k),
where the upper bounds of expected 2-norm of α1(k),
α3(k), and α4(k)can be similarily calculated as shown in
[13]. For the bounds of α2(k), the following transformation
is performed:
α2(k) =
k
i=0
λkiF(k) Φe(i) T(i)θ(i)(29)
=F(k)GT(k)D(k),
where
G(k) =
ΦT
e(k)
µΦT
e(k1)
.
.
.
µkΦT
e(0)
R(k+1)×n, µ := λ,
D(k) =
T(k)θ(k)
µT(k1) θ(k1)
.
.
.
µkT(0) θ(0)
Rk+1.
The following assumption is made
E
βk1(k),··· kn(k)D(k)DT(k)B(k)
since both (k)and θ(k)are finite as long as k < .
Note that F1(k) = GT(k)G(k) + λk+1F1(1). This
equation yields
tr F(k)GT(k)G(k)=tr Inλk+1F(k)F1(1)(30)
tr 1(1 λ)λk+1
λNαp0In
=n1α1p1
0λkN+1 (1 λ)
based on (20). Similarly, one can get
tr F(k)GT(k)G(k)=tr Inλk+1F(k)F1(1)(31)
=tr Inλk+1F(k)
p0n.
In this case, the following upper bound of E
α2(k)
2
can be derived (the subscript of expectation is omitted)
E
α2(k)
2=Etr F(k)GT(k)D(k)DT(k)G(k)F(k)
(32)
tr EF(k)GT(k)G(k) 1λ
λNαB(k)
1λ
λNαnB (k).
Therefore the estimation error is bounded. Proof is complete.
It is clear that the performance of parameter inference
depends on the bound B(k), which is a function of the time
delay and packet loss. In the next section, simulations will
be conducted to examine the influence of networked-induced
constraints to parameter adaption and motion prediction.
V. SIMULATION STUDY
A. Simulation Results without Network-induced Constraints
In this subsection, the proposed mRLS algorithm is imple-
mented without any network-induced constraints. The knee
joint rotation data shown in Fig. 2 were used to build a
10th order linear model and the parameter adaption result
with a forgetting factor λ= 1 is shown in Fig. 4, from
which one can observe that the coefficient for the most recent
measurement (θ1)dominates. Moreover, the parameters did
not converge even at the end of the simulation. In order
to achieve a faster convergence of the model parameters,
forgetting factors are introduced into parameter adaption.
Parameter adaption result with a forgetting factor λ= 0.997
is shown in Fig. 5, which confirms the convergence of the
model parameters. Comparing with Fig. 4, one can conclude
that introducing forgetting factors into the mRLS algorithm
leads to faster parameter convergence with fluctuations.
Using the identified model shown in Fig. 5, a 5-step
prediction is performed to examine the accuracy of the model
and prediction errors are shown in Fig. 6. It is clear that
the prediction error is less than 1 degree, which verifies the
performance of the identified model. As a baseline algorithm,
the simplest way of 5-step prediction is to use the current
measurement as a rough estimate, which is equivalent to
Max. Delay γλ
0.993 0.995 0.997 0.999 1
0 100% 0.1875 0.1891 0.1918 0.1985 0.2078
2Ts95% 0.6422 0.6368 0.6369 0.6475 0.6732
2Ts90% 1.0002 0.9088 0.8653 0.8165 0.8017
4Ts90% 2.1186 2.1713 2.0135 1.8206 1.7627
TABLE I: Root-mean-square (RMS) errors of 5-step predic-
tion with different forgetting factors (deg)
0 10 20 30 40 50 60 70 80
-2
-1
0
1
Time (sec)
Parameter values
θ
1
θ
2
θ
3
θ
4
θ
5
(a) Adaption of the first five model parameters
0 10 20 30 40 50 60 70 80
-0.2
-0.1
0
0.1
0.2
Time (sec)
Parameter values
θ
6
θ
7
θ
8
θ
9
θ
10
(b) Adaption of the last five model parameters
Fig. 4: Adaption of model parameters without forgetting
factors
a zero-order hold (ZOH) and it yields a root-mean-square
(RMS) error of 5.0146 degrees. In order to further examine
the effectiveness of the forgetting factors, the 5-step predic-
tion errors of the identified model with different forgetting
factors are shown in the first row of Table I, which verifies the
performance improvement brought by the forgetting factors.
It is also verified that the prediction errors become smaller
as forgetting factors decrease.
It is noticed that the forgetting factors are chosen to be
very close to 1 in the simulations, which is due to the nature
of the human walking pattern. Since walking patterns in each
gait cycle cannot be the same, the true kinematic model
is also varying. Moreover, there are inevitably unexpected
walking patterns and measurement noises in the parameter
adaption process. Thus, although choosing small forgetting
factors might make the parameters converge faster, it will
make the algorithm aggressive and result in large oscillations
in the parameter adaption. Applying a low-pass filter to
the measurement data might reduce such oscillations, but
it might eliminate useful information in the measurements.
B. Simulation Results with Time Delay and Packet Loss
This section shows the simulation results of model identi-
fication with time delay and packet loss. We chose the suc-
cessful transmission rate γto be 90% and 95%, respectively,
and the time delay of the kth packet τkunif (0,4Ts)and
τkunif (0,2Ts)respectively. Parameter adaption result of
0 10 20 30 40 50 60 70 80
-2
0
2
4
Time (sec)
Parameter values
θ
1
θ
2
θ
3
θ
4
θ
5
(a) Adaption of the first five model parameters
0 10 20 30 40 50 60 70 80
-0.4
-0.2
0
0.2
0.4
Time (sec)
Parameter values
θ
6
θ
7
θ
8
θ
9
θ
10
(b) Adaption of the last five model parameters
Fig. 5: Adaption of model parameters with a forgetting factor
λ= 0.997
0 10 20 30 40 50 60 70 80
-1
-0.5
0
0.5
1
Time (sec)
Prediction error (deg)
Fig. 6: Performance of 5-step prediction using identified
linear model (λ= 0.997)
the proposed algorithm with γ= 95%,τkunif (0,2Ts),
and λ= 0.997 is shown in Fig. 7. It is verified that the
parameters converge even with packet loss and time delay,
but the identified model parameters are quite different.
The effectiveness of forgetting factors is a bit more com-
plicated in the cases with network-induced constraints, as is
shown in Table I. When the time delay and packet loss are
not very significant (the case with a maximum of two-step
delay and γ= 95%), reducing forgetting factors may lead
to an improvement of the prediction accuracy. However in
this case, when the forgetting factor is reduced to 0.993, the
prediction error is even larger than the case with a forgetting
factor of 0.995. Moreover, when the packet loss and time
delay become more severe, reducing forgetting factors will
lead to large fluctuations of the adapted parameters and
prediction errors may be even larger than the case without
forgetting factors. To summarize, larger forgetting factors
need to be chosen with network-induced constraints.
VI. EXPERIMENTAL RESULTS
In this section, the proposed system is implemented in
LabVIEW to build the knee joint kinematic model in real-
time. The wireless IMU sensors were used to measure knee
joint rotation angles in the sagittal plane. The sampling rate
0 10 20 30 40 50 60 70 80
-2
-1
0
1
2
3
Time (sec)
Parameter values
θ
1
θ
2
θ
3
θ
4
θ
5
(a) Adaption of the first five model parameters
0 10 20 30 40 50 60 70 80
-1
-0.5
0
0.5
1
Time (sec)
Parameter values
θ
6
θ
7
θ
8
θ
9
θ
10
(b) Adaption of the last five model parameters
Fig. 7: Adaption of model parameters with a maximum two-
step delay, γ=95%, and λ= 0.997
0 10 20 30 40 50 60 70 80
-2
0
2
4
6
Time (sec)
Parameter values
θ
1
θ
2
θ
3
θ
4
θ
5
θ
6
(a) Adaption of the first six model parameters
0 10 20 30 40 50 60 70 80
-1
0
1
2
Time (sec)
Parameter values
θ
7
θ
8
θ
9
θ
10
θ
11
θ
12
(b) Adaption of the last six model parameters
Fig. 8: Adaption of model parameters in the experiment with
a forgetting factor λ= 0.995
was chosen to be 50Hz and there might be time delay and
packet loss due to the wireless transmission of the sensor
signals. In order to verify the performance of the proposed
algorithm, a different subject was selected (a 21-year old
male without known walking abnormalities) and he was
asked to walk on a treadmill at 2 mph. A pre-trial was
conducted for 10 seconds and the recorded data were used to
determine the order of the model. Based on the techniques
presented in Section III-B, the order was determined to be
12. In the formal experiment the subject was asked to walk
on the treadmill with the same speed.
The identified model parameters for the first 80 seconds of
formal experiment are shown in Fig. 8. The forgetting factor
was chosen as 0.995. It is evident that all parameters finally
converge. The identified model was used to achieve 5-step
prediction, and the RMS prediction error of the proposed
algorithm is 2.4027 degrees. The 5-step RMS prediction
error from the baseline algorithm is 11.2674 degrees due
to the increased speed, from which the effectiveness of the
proposed algorithm is confirmed.
VII. CONCLUSION
In this paper, a networked rehabilitation system was
introduced for lower-extremity rehabilitation. In order to
enable high-level motion planning for enhanced safety and
appropriate human-robot interactions, a time series model
was proposed to describe the knee joint rotation. In view
of time delay and packet loss of the measurement packets,
a modified recursive least square (mRLS) algorithm was
proposed for real-time modeling of the knee joint rotation
and convergence of the proposed algorithm was studied.
Effectiveness of the time delay, packet loss, and forgetting
factors was verified in simulations and experiments.
The proposed algorithm has many potentially important
applications, one of which is to achieve fall prediction based
on a fast change of identified model parameters. Another
application is to achieve high-level trajectory planning of the
rehabilitation robot based on the predicted human motion.
One of the ongoing work is to get the data from stroke
and Parkinson’s disease patients with walking abnormalities.
Model orders and parameters of the patients will be com-
pared and analyzed.
REF ER E NC ES
[1] “Lokomat,” http://www.hocoma.com/.
[2] R. J. Farris, H. A. Quintero, and M. Goldfarb, “Preliminary evaluation
of a powered lower limb orthosis to aid walking in paraplegic
individuals, IEEE Trans. Neural Syst. Rehabil. Eng., vol. 19, no.
6, pp. 652–659, 2011.
[3] S. K. Banala, S. Kim, S. K. Agrawal, and J. P. Scholz, “Robot assisted
gait training with active leg exoskeleton (ALEX), IEEE Trans. Neural
Syst. Rehabil. Eng., vol. 17, no. 1, pp. 2–8, 2009.
[4] “Vicon Bonita, http://www.vicon.com/.
[5] “Xsens MVN,” http://www.xsens.com/.
[6] X. Zhang, X. Chen, Y. Li, V. Lantz, K. Wang, and J. Yang, “A
framework for hand gesture recognition based on accelerometer and
emg sensors,” IEEE Trans. Syst., Man, Cybern. A, vol. 41, no. 6, pp.
1064–1076, 2011.
[7] I. Pappas, T. Keller, S. Mangold, M. Popovic, V. Dietz, and M. Morari,
“A reliable gyroscope-based gait-phase detection sensor embedded in
a shoe insole,” IEEE Sensors J., vol. 4, no. 2, pp. 268–274, 2004.
[8] H. Varol, F. Sup, and M. Goldfarb, “Multiclass real-time intent
recognition of a powered lower limb prosthesis, IEEE Trans. Biomed.
Eng., vol. 57, no. 3, pp. 542–551, 2010.
[9] G. Wu and S. Xue, “Portable preimpact fall detector with inertial
sensors,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 16, no. 2, pp.
178–183, 2008.
[10] W. Zhang, M. Tomizuka, and J. Bae, “Time series prediction of knee
joint movement and its application to a network-based rehabilitation
system,” in Proc. 2014 American Control Conf., 2014, pp. 4810–4815.
[11] W. Zhang, X. Zhu, S. Han, N. Byl, A. K. Mok, and M. Tomizuka,
“Design of a network-based mobile gait rehabilitation system,” in
Proc. Int’l Conf. on Robotics and Biomimetics, 2012, pp. 1773–1778.
[12] G. Box, G. M. Jenkins, and G. C. Reinsel, Time Series Analysis:
Forecasting and Control, Wiley. com, 2013.
[13] F. Ding and T. Chen, “Performance bounds of forgetting factor least-
squares algorithms for time-varying systems with finite measurement
data,” IEEE Trans. Circuits Syst. I, vol. 52, no. 3, pp. 555–566, 2005.
... Human motion prediction is a sequence-to-sequence task where future motion sequences are predicted based on observed historical motion sequences. Accurate forecasting of future human poses is crucial for the success of various applications, such as human-robot interaction and collaboration (HRI/C) [1], [2], human tracking [3], autonomous driving [4], and rehabilitation [5]. ...
Preprint
Precise future human motion prediction over subsecond horizons from past observations is crucial for various safety-critical applications. To date, only one study has examined the vulnerability of human motion prediction to evasion attacks. In this paper, we propose BadHMP, the first backdoor attack that targets specifically human motion prediction. Our approach involves generating poisoned training samples by embedding a localized backdoor trigger in one arm of the skeleton, causing selected joints to remain relatively still or follow predefined motion in historical time steps. Subsequently, the future sequences are globally modified to the target sequences, and the entire training dataset is traversed to select the most suitable samples for poisoning. Our carefully designed backdoor triggers and targets guarantee the smoothness and naturalness of the poisoned samples, making them stealthy enough to evade detection by the model trainer while keeping the poisoned model unobtrusive in terms of prediction fidelity to untainted sequences. The target sequences can be successfully activated by the designed input sequences even with a low poisoned sample injection ratio. Experimental results on two datasets (Human3.6M and CMU-Mocap) and two network architectures (LTD and HRI) demonstrate the high-fidelity, effectiveness, and stealthiness of BadHMP. Robustness of our attack against fine-tuning defense is also verified.
... These strong needs motivate the teleoperation of lower limb exoskeletal robots. [1][2][3][4][5] Mechanical motion capture systems are often used in early days 6,7 and brain machine interface is emerging. 8 The motion capture systems composed of inertial 1 measurement units (IMUs) have many attractive advantages. ...
Article
Full-text available
Delay is a major challenge in teleoperation. To teleoperate a robot in real-time with specific human motion such as walking, prediction and compensation algorithms are needed to minimize the tracking delay. Human walking possesses the inherent features of synergy and quasi-periodicity with nonlinearity. By analyzing these features, three prediction methods were designed. A delay-coordinate phase space reconstruction method is based on the delay embedding theorem and uses the historical data. The echo prediction method uses data from the coupling joints of the other body side and takes the Euclidean distances in the delay-coordinate phase space reconstruction method for data search and match. The fused prediction method combines the delay-coordinate phase space reconstruction and echo prediction methods, uses the best matching historical data of the same joint and the corresponding joint of the other limb. These methods were implemented for teleoperation of an exoskeletal robot with inertial measurement unit measurements of human walking. The predicted trajectories were used to compensate the teleoperation delay and improve the robot tracking performance. Experimental results demonstrated that the fused method brought apparent improvement over other methods and can serve as a promising solution for time delay estimation, prediction, and compensation of human walking tracking and teleoperation.
... However, it requires historical system data to achieve a better performance which brings in inevitable dead time. There are also some other prediction methods, such as the autoregressive integrated model method, autoregressive integrated moving average method, autoregressive moving average model method [20], and modified recursive least square methods [21], which are also time series analysis methods but with undesired computational expense, dead time, or complex parameter tuning processes. Besides, the consistency and smoothness of human joint motion should not be ignored as human joints possess apparent biological damping [22][23][24] which damps out the high frequency vibration. ...
Article
Full-text available
Human motion detection is of fundamental importance for control of human–robot coupled systems such as exoskeletal robots. Inertial measurement units have been widely used for this purpose, although delay is a major challenge for inertial measurement unit-based motion capture systems. In this paper, we use previous and current inertial measurement unit readings to predict human locomotion based on their kinematic properties. Human locomotion is a synergetic process of the musculoskeletal system characterized by smoothness, high nonlinearity, and quasi-periodicity. Takens’ reconstruction method can well characterize quasi-periodicity and nonlinear systems. With Takens’ reconstruction framework, we developed improving methods, including Gaussian coefficient weighting and offset correction (which is based on the smoothness of human locomotion), Kalman fusion with complementary joint data prediction and united source of historical embedding generation (which is synergy-inspired), and Kalman fusion with the Newton-based method with a velocity and acceleration high-gain observer (also based on smoothness). After thorough analysis of the parameters and the effect of these improving techniques, we propose a novel prediction method that possesses the combined advantages of parameter robustness, high accuracy, trajectory smoothness, zero dead time, and adaptability to irregularities. The proposed methods were tested and validated by experiments, and the real-time applicability in a human locomotion capture system was also validated.
... The prediction of future human motion x H needs to be done in two steps: inference of the human's goal G H and prediction of the trajectory to the goal. Once the goal is identified (using the method in [23]), a linearized reaction model can be assumed for trajectory prediction [24], e.g. ...
Conference Paper
Full-text available
In factories of the future, humans and robots are expected to be co-workers and co-inhabitants in the flexible production lines. It is important to ensure that humans and robots do not harm each other. This paper is concerned with functional issues to ensure safe and efficient interactions among human workers and the next generation intelligent industrial co-robots. The robot motion planning and control problem in a human involved environment is posed as a constrained optimal control problem. A modularized parallel controller structure is proposed to solve the problem online, which includes a baseline controller that ensures efficiency, and a safety controller that addresses real time safety by making a safe set invariant. Capsules are used to represent the complicated geometry of humans and robots. The design considerations of each module are discussed. Simulation studies which reproduce realistic scenarios are performed on a planar robot arm and a 6 DoF robot arm. The simulation results confirm the effectiveness of the method.
Article
Wireless inertial motion capture holds promise for real-time human-machine interfaces and home-based rehabilitation applications. However, wireless data drop can cause significant estimation errors deteriorating performance or even making the system unusable. It is currently unclear how to estimate non-periodic kinematics with wearable inertial measurement units (IMUs) in the presence of wireless data drop (packet loss). We thus propose a novel inference encoder-decoder network model for real-time kinematics during dynamic movement. Twenty-four healthy subjects performed yoga, golf, swimming, dance, and badminton movement activities while wearing IMUs and 10-90% of each IMU's data were randomly removed to determine the effects of data drop on estimation accuracy with and without the proposed model. Results demonstrated a reduction in RMSE of 45.2% to 51.5% in the upper limb kinematic estimation of the proposed model compared to the No Prediction strategy, and a reduction of 19.1% to 31.3% of the proposed model compared with an baseline LSTM model. In addition, the proposed model has significantly less error (p<0.05) than the No Prediction strategy and the baseline LSTM model for 10%, 20%, 30%, 40%, 50%, 60%, 70%, and 80% data drop. These results could enable wearable, wireless IMU dynamic motion analysis and assessment with reduced kinematic estimation error in the presence of varying amounts of wireless data drop and thus could further facilitate human-machine interaction and home-based medical assessment and treatment.
Article
Full-text available
This paper describes a powered lower-limb orthosis that is intended to provide gait assistance to spinal cord injured (SCI) individuals by providing assistive torques at both hip and knee joints. The orthosis has a mass of 12 kg and is capable of providing maximum joint torques of 40 Nm with hip and knee joint ranges of motion from 105° flexion to 30° extension and 105° flexion to 10° hyperextension, respectively. A custom distributed embedded system controls the orthosis with power being provided by a lithium polymer battery which provides power for one hour of continuous walking. In order to demonstrate the ability of the orthosis to assist walking, the orthosis was experimentally implemented on a paraplegic subject with a T10 complete injury. Data collected during walking indicates a high degree of step-to-step repeatability of hip and knee trajectories (as enforced by the orthosis) and an average walking speed of 0.8 km/hr. The electrical power required at each hip and knee joint during gait was approximately 25 and 27 W, respectively, contributing to the 117 W overall electrical power required by the device during walking. A video of walking corresponding to the aforementioned data is included in the supplemental material.
Article
Full-text available
This paper describes a control architecture and intent recognition approach for the real-time supervisory control of a powered lower limb prosthesis. The approach infers user intent to stand, sit, or walk, by recognizing patterns in prosthesis sensor data in real time, without the need for instrumentation of the sound-side leg. Specifically, the intent recognizer utilizes time-based features extracted from frames of prosthesis signals, which are subsequently reduced to a lower dimensionality (for computational efficiency). These data are initially used to train intent models, which classify the patterns as standing, sitting, or walking. The trained models are subsequently used to infer the user's intent in real time. In addition to describing the generalized control approach, this paper describes the implementation of this approach on a single unilateral transfemoral amputee subject and demonstrates via experiments the effectiveness of the approach. In the real-time supervisory control experiments, the intent recognizer identified all 90 activity-mode transitions, switching the underlying middle-level controllers without any perceivable delay by the user. The intent recognizer also identified six activity-mode transitions, which were not intended by the user. Due to the intentional overlapping functionality of the middle-level controllers, the incorrect classifications neither caused problems in functionality, nor were perceived by the user.
Conference Paper
In this paper, a network-based rehabilitation system is introduced for improved mobility and tele-rehabilitation. Time series of knee joint rotation measurement is obtained using the rehabilitation device in the system, and an autoregressive integrated (ARI) model is built to achieve knee joint angle prediction during the rehabilitation process. It is shown that the predicted knee joint angles are reliable over 10 future time steps. The ARI model and the predicted knee joint angles can provide insight to patients and therapists for deep understanding of patients' walking behaviors. Moreover, it is shown in this paper that the predicted knee joint angles can also be used to compensate for time delay and packet loss in the networked rehabilitation system to achieve accurate torque tracking. Simulation and experimental results are provided to demonstrate the performance of the proposed algorithm.
Conference Paper
Gait training of stroke survivors can help in retraining their muscles and improving their gait pattern. Robot assisted gait training (RAGT) was developed for stroke survivors using ALEX and force-field controller, which use assist-as-needed paradigm for rehabilitation. In this paradigm undesirable gait motion is resisted and assistance is provided towards the desirable motion. The force-field controller achieves this paradigm by applying forces at the foot of the subject. Two stroke survivors participated in a 15-day gait training study each with ALEX. The results show that by the end of the training the gait pattern of the patients was improved towards healthy subjects gait pattern. Improvement is seen as increase in the size of the patientspsila gait pattern, increase in knee and ankle joint excursions and increase in their walking speed on the treadmill.
Conference Paper
In this paper, a network-based mobile gait rehabilitation system is proposed for improved mobility and tele-rehabilitation. In this system, an Internet-based body sensor network is employed for health monitoring and a local highspeed wireless network is developed to achieve precise control of the gait rehabilitation device, which is a compact rotary elastic actuator (cRSEA). A new wireless protocol called MBStarPlus is proposed to achieve a high sampling rate and real-time communication. To deal with packet loss in the wireless network, a modified linear quadratic Gaussian (LQG) controller is combined with a disturbance observer (DOB) to control the cRSEA over the wireless network. Successful system integration and promising experimental results validate both the system design and proposed control algorithm. The plan for clinical testing with the proposed system is discussed.
Article
This paper presents a framework for hand gesture recognition based on the information fusion of a three-axis accelerometer (ACC) and multichannel electromyography (EMG) sensors. In our framework, the start and end points of meaningful gesture segments are detected automatically by the intensity of the EMG signals. A decision tree and multistream hidden Markov models are utilized as decision-level fusion to get the final results. For sign language recognition (SLR), experimental results on the classification of 72 Chinese Sign Language (CSL) words demonstrate the complementary functionality of the ACC and EMG sensors and the effectiveness of our framework. Additionally, the recognition of 40 CSL sentences is implemented to evaluate our framework for continuous SLR. For gesture-based control, a real-time interactive system is built as a virtual Rubik's cube game using 18 kinds of hand gestures as control commands. While ten subjects play the game, the performance is also examined in user-specific and user-independent classification. Our proposed framework facilitates intelligent and natural control in gesture-based interaction.
Article
Gait training of stroke survivors is crucial to facilitate neuromuscular plasticity needed for improvements in functional walking ability. Robot assisted gait training (RAGT) was developed for stroke survivors using active leg exoskeleton (ALEX) and a force-field controller, which uses assist-as-needed paradigm for rehabilitation. In this paradigm undesirable gait motion is resisted and assistance is provided towards desired motion. The force-field controller achieves this paradigm by effectively applying forces at the ankle of the subject through actuators on the hip and knee joints. Two stroke survivors participated in a 15-session gait training study each with ALEX. The results show that by the end of the training the gait pattern of the patients improved and became closer to a healthy subject's gait pattern. Improvement is seen as an increase in the size of the patients' gait pattern, increased knee and ankle joint excursions and increase in their walking speeds on the treadmill.