Kinematically Optimal Catching a Flying Ball with a HandArmSystem
ABSTRACT A robotic ballcatching system built from a multi purpose 7DOF lightweight arm (DLRLWRIII) and a 12 DOF fourfingered hand (DLRHandII) is presented. Other than in previous work a mechatronically complex dexterous hand is used for grasping the ball and the decision of where, when and how to catch the ball, while obeying joint, speed and work cell limits, is formulated as an unified nonlinear optimization problem with nonlinear constraints. Three different objective functions are implemented, leading to significantly different robot movements. The high computational demands of an online realtime optimization are met by parallel computation on distributed computing resources (a cluster with 32 CPU cores). The system achieves a catch rate of > 80% and is regularly shown as a live demo at our institute.

Conference Paper: Achievement of complex contact motion with environments by musculoskeletal humanoid using humanlike shock absorption strategy
[Show abstract] [Hide abstract]
ABSTRACT: We have been developing and studying musculoskeletal humanoids. Our goal is to realize more humanlike humanoids which can do natural and dynamic motions as well as humans. Especially motions with complex contact with environments, which is jumping, running, catching a ball, and landing on one's hands, etc, are difficult to be achieved by humanoids. To achieve that motions, robots have to absorb shock force so as not to break ones' body structures, such as gears, motors, body links and so on. Musculoskeletal humanoids are suited for these situations, because they can easily have mechanical flexibilit for shock absorption by adding elastic units, which is nonlinear spring units, to its own tendons. In this paper, we propose shock absorption methods by musculoskeletal humanoids, which uses its own mechanical flexibilit and simple refle of each muscles based on tension sensor. This strategy is inspired by human's motion control, and it can achieve shock absorption tasks without any fast sensor feedback controls and any prediction ocntrols used by conventional robots with rigid bodies. We chose a catching a ball task as an example of complex contact motions, implemented the proposed strategy to musculoskeletal humanoid Kenzoh and confirme the feasibility of proposed method by actually catching a ball demonstration.Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on; 01/2012 
Conference Paper: Cooperative quadrocopter ball throwing and catching
[Show abstract] [Hide abstract]
ABSTRACT: This paper presents a method for enabling a fleet of circularly arranged quadrocopters to throw and catch balls with a net. Based on a firstprinciples model of the net forces, nominal inputs for all involved vehicles are derived for arbitrary target trajectories of the net. Two algorithms that generate openloop trajectories for throwing and catching a ball are also introduced. A set of throws and catches is demonstrated in the ETH Zurich Flying Machine Arena testbed.Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on; 01/2012  SourceAvailable from: Jan Peters
Conference Paper: Learning throwing and catching skills
[Show abstract] [Hide abstract]
ABSTRACT: In this video, we present approaches for learning throwing and catching skills. We first show how a hitting skill (i.e., table tennis) can be learned using a combination of imitation and reinforcement learning. This hitting skill is subsequently generalized to a catching skill. Secondly, we show how a robot can adapt a throwing skill to new targets. Finally, we demonstrate that a BioRob and a Barrett WAM can play catch together using the previously acquired skills.Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on; 01/2012
Page 1
Kinematically Optimal Catching a Flying Ball with a HandArmSystem
Berthold B¨ auml, Thomas Wimb¨ ock and Gerd Hirzinger
Abstract—A robotic ballcatching system built from a multi
purpose 7DOF lightweight arm (DLRLWRIII) and a 12 DOF
fourfingered hand (DLRHandII) is presented. Other than in
previous work a mechatronically complex dexterous hand is
used for grasping the ball and the decision of where, when
and how to catch the ball, while obeying joint, speed and work
cell limits, is formulated as an unified nonlinear optimization
problem with nonlinear constraints. Three different objective
functions are implemented, leading to significantly different
robot movements. The high computational demands of an
online realtime optimization are met by parallel computation on
distributed computing resources (a cluster with 32 CPU cores).
The system achieves a catch rate of > 80% and is regularly
shown as a live demo at our institute.
I. INTRODUCTION
Catching a thrown ball with a hand is not easy – neither
for humans nor for robots . It demands for a tight interplay of
skills in mechanics, control, planning and visual sensing to
reach the necessary precision in space and time. Because of
this, ball catching has been used for almost 20 years now as
a challenging benchmark system to develop and test robotics
key technologies [1], [2], [3], [4], [5], [6]. In all the works the
general setup is in principle the same: a stereo vision system
tracks the ball and predicts the balls trajectory, then the point
and time, where and in which orientation the robot should
intercept the ball on its trajectory, is determined. Next, the
robot configuration to reach the catch point is computed and
finally a path is generated, which brings the robot from its
start configuration to the desired catch configuration.
A. Related Work
In the famous and pioneering work [1], [2] the 4 DOF
“WAM” arm with a gripper to grasp the ball and an active
vision system is used. The heuristic catch point selection
chooses the closest point of the ball trajectory to the robot
base and orients the gripper perpendicular to the trajectory.
A cartesian path is generated as a third order polynomial
and executed by an inverse kinematics running in the control
loop.
A system with a 5 DOF arm on a humanoid robot (only the
arm is moving) with a “cooking basket” at the end effector
for catching the ball and an active vision system is presented
in [3]. The inverse kinematics is solved by a neural network,
which should lead to a human like movement behavior.
In our previous work [4] we used the 7 DOF DLRLWRII
arm with a small basket at the endeffector and a stationary
stereo camera and image processing system built from cheap
“offtheshelf” components (PAL cameras with 50Hz and
DLR Institute of Robotics and Mechatronics, M¨ unchnerstr. 20, 82234
Wessling, Germany berthold.baeuml@dlr.de
Fig. 1.
DLRLWRIII arm with N = 7 DOF [7] and the 12 DOF DLRHandII [8].
All joints are equipped with torque sensors and are impedance controlled.
Weight: LWRIII 14 kg; HandII 2 kg.
The handarm system catching a ball. The system is built from the
standard PC). The catch point is determined with a simple
heuristic that balances two goals: a) choose a point near the
robot, so that the robot has not to move very far and can,
although it is not very fast, reach the catch point in time
and b) choose a catch point far away from the robot, to
prohibit folding up of the robot and, hence, running into the
joint limits. Next, an inverse kinematics computes the catch
configuration, taking into account, that the orientation of the
catching basket opening should be perpendicular to the ball
trajectory. Finally, an interpolator generates the joint paths
with a trapezoidal velocity profile (in joint space).
In [5] the focus is on investigating motion primitives for
human like path generation. The catching is done with an arm
of a humanoid robot (only the arm moves) equipped with a
base ball glove to compensate for only determining the catch
point and not the endeffector orientation. The catch point is
chosen to be the intersection of the ball trajectory with a
horizontal plane at a given height and an inverse kinematics
computes the catch configuration.
A 6DOF robot arm, especially built for ball catching, with
a basket is described in [6]. The arm is built from commercial
offtheshelf parts with exceptionally high dynamics with
joint velocities up to 470 deg/s and power consumption of
up to 5kW, however. The main focus of the work is on tele
operated ball catching, but an autonomous catching using a
stereo vision system was also implemented as a benchmark.
A heuristic chooses the catch point as the intersection of the
ball trajectory with a plane in front of the robot and an an
alytical inverse kinematics computes the catch configuration
of the robot.
Page 2
2.
x 1m). The ball is thrown by a human from a distance
of about 5 m onto the robot with a speed of typically
7 m/s, resulting in a flight time of about 1s. The robot
(depicted in standard starting configuration) is placed inside
a working cell (mockup of a spacelab module) with walls,
that lie inside the kinematic work space of the robot. The
stereo camera system (two PAL cameras with 50Hz rate
for video fields) is placed above the throwing person at 2
m height and has a vertical base length of 1 m. The ball
has a diameter of 8.5 cm and a weight of 70 g resulting in
significant air drag effects (for a 5 m throw >20 cm shorter
flying distance as compared to a purely ballistic trajectory;
see [4] for details)
Overall geometrical setup (each tile of the floor is 1m
B. Contributions
In this paper we present a ball catching setup, which
significantly differs in two aspects from previous work. First,
we use a dexterous multipurpose fourfingered hand (DLR
HandII, see Fig. 1 and [8]) to grasp the ball. This is
challenging for the hardware and controllers as the hand has
to be able to close fast enough and in addition it has to
be robust enough, so that the impact of the ball does not
damage the highly complex mechatronic device. Moreover,
this is also challenging for the planning algorithms for the
arm movements, because not only the position but also the
orientation of the hand relative to the ball trajectory has to be
very precisely determined, so that the ball can be grasped and
does not hit (and damage) the fingers in a disadvantageous
configuration.
The second new aspect is, that, instead of using sep
arate steps for catch point selection, catch configuration
computation and path generation, we present an unified ap
proach, which subsumes all three steps in a single, nonlinear
optimization problem with nonlinear constraints. This not
only gives rise to a better overall performance (e.g. bigger
reachable workspace and faster thrown balls can be caught)
as one can get closer to the dynamical limits (maximal
velocity and acceleration) of the robot, but also allows to
optimize for different criteria, which leads to significantly
different catch behaviors.
II. OVERALL SETUP AND SYSTEM ARCHITECTURE
Fig. 2 shows the geometrical setup, which is almost the
same as in our previous work [4]. Also the same visual
tracking and Extended Kalman Filter based prediction of the
ball trajectory are used providing a new prediction every
20ms. The visual tracking is quite robust with successfully
tracking >80% of the thrown balls, but the prediction varies
(see Fig. 3) over time and typically reaches only about 100ms
before the catch the necessary precision of <2cm and <5ms
(for details on the grasp precision demands see Sec. III).
This makes it necessary, that the planning algorithm re
plans the robot movement every 20ms with as little delay as
possible to keep up with the changing trajectory prediction.
Because our planning algorithm is based on a nonlinear
optimization with nonlinear constraints (see Sec. IV), it is
�
�
�
�� �
�
� �
�
�
�
� �
�
����
�
�
���
��
�
�
0.00.1 0.20.3
t �s�
0.40.50.6
0.00
0.01
0.02
0.03
0.04
0.05
� x �m�
Fig. 3.
between the final catch point xcand the catch point prediction xb,t(tc) made
from the measurements until time t is plotted. xb,t is the ball trajectory
prediction as it is known at time t and tcis the final catch time.
Error of the ball trajectory prediction during a throw. The distance
computationally demanding, even though we use the highly
efficient SQP (sequential quadratic programming) method
[10] in an implementation taken from [11]. Running on up
todate hardware – a single core of a 2.2Ghz QuadCore
Xeon – the worst case runtime is 60ms with 10ms on average.
Hence, at least three CPU cores have to be used in parallel
in a round robin scheme, where each new ball trajectory
prediction is assigned to a currently idle core, to not omit
any prediction.
Finding the first optimal movement path given the first
ball trajectory prediction is computationally even more de
manding, as no previous solution is present, which could be
used for initializing the optimizer and which would have to
be only slightly adapted to account for the slight changes
in the prediction. Because of the many local minima, which
arise mainly due to the nonlinear constraints (see Sec. IV
for details), the optimization has to be executed with many
different starting solutions, to find the optimal (or at least a
good) solution with high probability. Therefore, a compute
cluster with up to 32 CPU cores is used to execute a parallel
multi start search for an optimal first solution. The search
is stopped after 60ms and results in about 100 optimized
solutions on average, including with high probability a nearly
globally optimal movement path as has been approved by
batch runs with more than 10000 start solutions.
Page 3
4.
and their mapping to the computation hardware as well as
the cycle times. The flying ball is observed by a stereo
camera system (standard PAL cameras, 50Hz video field
rate) which sends every 20ms the two video fields to a
Linux PC (Pentium IV, 2GHz), where the 2DTracking
modules and the estimation (with Extended Kalman Filter,
EKF) and prediction modules for the 3D ball trajectory are
running (overall delay from camera shutter until prediction
finished is 30ms). The resulting prediction is sent to the
online realtime planning module running on a cluster of re
altime computers (32 CPU cores built from 4x DualQuad
CoreXeon servers running the QNX [9] realtime OS). The
planner consists of two parts: the “compute slaves” (one for
each CPU core), which are actually executing the sequential
quadratic programming (SQP) optimization algorithm in
parallel, and the “move generator” which coordinates the
slaves and communicates with the vision and the robot
control system. After delay of 60ms the planner outputs
a path with 1ms time resolution and 1.8s duration (the
maximal allowed duration of a catch movement), which
contains for each time step the desired arms joint angles
and a flag for the hands grasp state (i.e. open or close).
The robot arm and hand controllers, running at 1kHz on
a QNX PC (Core2Duo, 3GHz), then execute the path and
close the hand at the desired catch time. The overall delay
from camera shutter to robot movement is 90ms.
System architecture showing the processing modules
The system architecture (see Fig. 4) has to provide dis
tributed computational resources while meeting hard realtime
constraints for the parallel planning as well as the robot
controllers. For the communication between the software
components running on the diverse computers we use the
aRD (agile robot development) software concept
which we especially developed for applications with complex
mechatronic components with high computational demands
under hard realtime constraints. By using Gigabit Ethernet
connections a worst case delay of < 1ms even for transmit
ting the largest data packets in the system (the arm movement
path for the 7 joint angles) with a size of 64KByte could be
achieved.
[12],
III. HAND
The robot system is composed of the DLRLWRIII and
the DLRHand II. In this section the properties and control of
the robotic subsystems are highlighted. The light weight arm
has a remarkable load to weight ratio of 1 : 1 with a mass
of 14 kg. The torque sensing is used on the one side for
interaction with unstructured environments and the human,
and on the other side it is used to damp vibrations of the light
weight structure [13]. The hand has four fingers with 3 DOF
each, resulting in overall 12 DOF (c.f. Fig. 1). In addition
to the position sensors the linkside torques are measured as
well. The control law runs on a Core2Duo, 3GHz with a
controller sample time of 1 ms.
The catching of a ball poses basically two main require
ments to the control system. First, the accurate reaching
of the desired catch point including tracking capabilities
for dynamic motions and vibration suppression. Second, the
grasp control has to provide a fast closing motion that is
compliant at the same time to reduce grasping and impact
forces.
Fig. 5.
ball and the catch frame that acts as tool center point (TCP) of the robot. In
the catch frame the zaxis (blue) represents the flight direction of the ball.
It is chosen such that the ball will approach the hand from its open side,
i.e. that the ball passes the opening between the widely open ring finger
and the thumb. Finally, hitting the proximal phalanxes of the index and the
middle finger. Furthermore, note that the thumb and the index finger are
already configured such that the ball can barely escape between them. In
the closed configuration the hand realizes a spherical grasp. In order to cage
the ball the thumb is moved into the direction of the ring finger. In this way
the flight channel to enter the hand is closed. Note that the CAD drawing
is missing a palm element mounted on the base of the palm that further
restricts the motion of the ball (compare the real hand in Fig. 8 with real
grasping).
The hand configurations before (left) and after (right) catching the
For the arm a joint space impedance control is applied
whose output is fed to a lowlevel torque controller that
realizes the vibration damping [14]. For the grasping also a
joint impedance tracking controller is applied. For the outer
loop of the control law a PD plus tracking controller of the
form
τd= Mqh¨ qh,d−Kd˙ e−Kpe+g(qh,q)
is applied [15]. The vector τd∈R12contains the desired joint
torques. The inertia matrix of all finger joints is summarized
in stacked notation in the blockdiagonal matrix Mqh. The
position error is defined as e = qh− qh,d, with qhthe
joint positions of the hand and qh,dthe desired equilibrium
position. The PD matrices Kp,Kdrepresent the stiffness and
(1)
Page 4
the damping behavior respectively. The term gravity term
g(qh,q) is compensated while the Coriolis and centrifugal
terms are neglected, with q ∈ R7the joint angles of the arm.
The desired control torque τdis used as set point for a low
level torque controller which is based on the joint torque
measurement, and offline estimated static and viscous motor
friction parameters. The joint velocity is estimated by means
of an observer that is described in [16].
Note, that even though the impedance parameters provide
compliance there is the time delay of 1ms for the controller
to react to the impact. The impact forces with a bandwidth
larger than the one of the controlled system are seen directly
by the structure and by the gears of the hand. In this case
the light weight structure of the hand is another advantage
because the bases of the fingers realize a mechanical base
compliance that acts as mechanical low pass filter1.
Besides the hand control it is a difficult problem how
to select the grasping strategy. That includes the choice of
impact point in the hand, which is closely related to the
selection of the so called catch frame that acts as tool center
point (TCP) of the robot (compare Fig. 5). Depending on this
choice of catch frame one needs to find a preshape of the
hand that is furthermore a function of the estimation accuracy
of the ball trajectory w.r.t. the arm coordinate system (here:
2 cm). The preshape is selected such that the ball including
its position accuracy can enter the hand. For the closing we
apply a caging strategy since the transitions within the hand
has multiple contacts with the fingers and the palm. It is
critical that the hand starts closing with a timing resolution
of less than 5ms2in order to avoid that the ball hits the
fingers or that the ball bounces back out of the hand. At
this point the preshape and the final grasp configuration are
optimized manually by using simulations in virtual reality
and by the experiences during the experiments. Currently,
we are working on methods to derive the methods to obtain
such strategies automatically.
IV. KINEMATICALLY OPTIMAL REALTIME PLANNER
When a ball is thrown at t0= 0, the planner has to
solve the following problem: given the start configuration
of the robotq(0) = q0and start joint velocities ω(0) = ω0,
at which time tc and in which configuration q(tc) = qc
with ω(tc) = 0 should the robot intercept the ball on its
trajectory, while obeying constraints like e.g. joint limits,
joint velocity limits or geometrical workcell limits ? Note,
that the robot does not have to stand still at be beginning,
which is especially important to allow for recommanding,
when the ball trajectory prediction changes over time.
The state of the art in general path planning are sam
ple based methods, like probabilistic roadmaps (PRM) and
rapidly exploring random trees (RRT), as presented e.g. in
[17], [18] and [19]. As is noted in [17], these methods
1Certainly, there exist configurations in which large forces can be exerted
on the finger. This is the case when the ball hits the finger in the direction
from the fingertip to the finger base.
2Assuming that the ball flies within the hand about 0.03 m with a velocity
of 6 m/s the time duration of 5 ms is obtained.
qmin[◦]
170
120
170
120
170
qmax[◦]
170
120
170
120
170
ωmax[◦/s]
τmax[Nm]
1
2
3
4
5
6
7
100
100
100
100
150
100
100
180
180
80
80
30
30
30
45
45
80
135
TABLE I
LIMITS OF THE LWRIII ARM AS USED IN THE PLANNER. THE
MAXIMAL ACCELERATION IS SET TO amax = 860◦/s2FOR ALL JOINTS.
are especially useful to find a first feasible solution in
problems, where the working cell is cluttered with geo
metrical obstacles. If a path is needed, which in addition
is optimal with regard to e.g. time, energy or path length,
the previously found solution is used in a second step as
a starting point in a general nonlinear optimization like in
[20]. For our ball catching setup, however, the first step of
finding a path by means of sample based methods is on
the one hand not necessary, because of the quite simple
geometrical constraints mainly at the bounds of the working
cell, and on the other hand not feasible, as those algorithms
are computationally too demanding to be solved under the
tight timing constraints we have to meet. Therefore, we
formulate the problem directly as a nonlinear optimization
with nonlinear constraints and avoid the problem of local
minima by using a parallel multi start technique.
A. Assumptions
Because solving even “only” the general nonlinear op
timization problem accurately is still computationally too
demanding to be executed in realtime, some assumptions
and approximations utilizing details of our setup have to be
made to simplify the problem significantly while still getting
a nearly optimal solution for the original problem.
1) Kinematically Planning: Although to catch a ball the
robot has to move fast and in general its dynamics plays
an important role for generating an optimal movement path,
we chose to do a purely kinematically optimization, i.e., the
objective function and constraints depend only on q, ˙ q and
¨ q and not on the joint torques τ.
This approximation is not too bad for the LWRIII, be
cause for fast motions the joint velocity limits dominate the
robot behavior: e.g. if the robot has to move in 1s (the balls
typical flight time in our setup) as far as possible, only 0.1s
are needed for accelerating and decelerating respectively,
but 80% of the time the robot moves at maximum joint
velocity. That means, that the details, of how the robot
accelerates/decelerates are not very important for the overall
performance, as long as the maximally allowed accelerations
are conservatively chosen, to never exceed the torque limits
of the robot (see Tab. I for the chosen parameters and limits).
2) Trapezoidal Joint Velocity Ramps: We restrict the
robot movements to trapezoidal velocity ramps, because, as
argued before, the precise characteristics of the acceleration
and deceleration phases are not essential for the overall
Page 5
Fig. 6.
small coordinate system near the hand; see also Fig. 5) is tested against this
model by calculating the smallest distance between the TCP position and
the collision objects. The model consists of five planes (floor, ceiling and
three walls) and three cylinders with spherical caps (robot platform and the
two stabilizing rods in the back). The platform cylinder is “blown up” to
additionally avoid self collisions of the robot. When calculating the distance
to the planes an additional safety distance of 0.3m is subtracted to account
for stretched out fingers.
Collision model. The TCP of the robot xR (the origin of the
performance and trapezoidal ramps can be easily analytically
handled. Moreover, for the objective functions, we present
here (see Sec. IVB.1), the trapezoidal ramps are even
optimal.
Using the step function
�
with
ˆθ(t,t1,t2)= θ(t1−t)−θ(t−t2), with t1≤t2, the trapezoidal
ramp for a single joint can be written as:
θ(t) =
1
0
if t ≥ 0
otherwise
�t
0dt�θ(t�) =tθ(t) and the definition
¨ q(t) = aˆθ(t,t1,t2),
˙ q(t) = ω0+atˆθ(t,t1,t2),
q(t) = q0+ω0t +a
2t2ˆθ(t,t1,t2).
(2)
The acceleration phase with aacc= a ends at t1, then a
phase with ˙ q = ωmax lasts until t2, when the deceleration
phase with adec= −a starts. There are two cases: first,
t1=t2, i.e. a triangle ramp, and second, t1<t2, i.e. a “full”
trapezoidal ramp, where, because of ˙ q(t1) = ωmax, it holds
t1=ωmax−ω0
a
. This means that the ramp in both cases is
completely determined by only the two parameters a and
t2.
3) Collision Avoidance Heuristics: Only the position of
the TCP of the robot (see Sec. 5) is tested against a collision
model of the environment and even that is done only for the
final catch configuration qc. That this guarantees a collision
free path of the whole robot during the whole movement,
is established by making the objects in the collision model
bigger than the physical objects (see Fig. 6), and by the
limited task that has to be fulfilled, i.e. to catch a ball coming
in from the front.
B. Optimization Problem
With this assumptions the nonlinear optimization problem
with nonlinear constraints can now be written as:
(qc,tc) = min
(q,t)∈SH(q,t),
(3)
S = {(q,t) ∈ RN×R :hi(q,t) = 0,i = 1...Nh,
gj(q,t) ≥ 0, j = 1...Ng},
with an objective function H, equality constraints hi and
inequality constraints gj. Note, that the optimization space
is only N +1 dimensional, because due to the assumptions
(especially the trapezoidal ramps assumption), the robot
movement is completely determined by specifying the catch
configuration qcand the catch time tc.
In the following the three objective functions, that lead
to significantly different catch behaviors, and the constraints
are presented.
1) Objective Function: The first catch behavior is called
“soft” mode, as it tries to make soft movements with accel
erations as small as possible for each joint:
�
N
Note, ai= ai(qi,t) is defined by substituting qi(tc) = qiand
˙ qi(tc) = 0 in eq. 2 and (analytically) solving the resulting
quadratic equations.
The second catch behavior is called “latest” mode. It tries
to intercept the ball as late as possible on its trajectory, hence
Hsoft(q,t) =
1
N
∑
i
�
ai
amax,i
�2
.
Hlatest(q,t) = −t.
Finally, we implemented the “cool” mode catch behavior,
where the robot first moves as fast as possible to a config
uration where it can intercept the ball on its trajectory and
then “cooly” waits for the ball reaching the hand and finally
suddenly closes the fingers to grasp the ball. Unlike for the
other catch modes, here one has to distinguish between the
time tc, the robot reaches the catch configuration qcand the
time tg, the ball is finally grasped.
�
with tmin,i=tmin,i(qi) being the minimal time the joint i needs
to reach the given angle qiwhen moving as fast as possible,
i.e. with ai = amax,i. The minimal time tmin,iis defined by
substituting ˙ qi(tmin,i) = 0 and qi(tmin,i) = qi in eq. 2 and
solving the resulting quadratic equation.
2) Constraints: The task – where, when and in which
orientation to catch the ball – is only completely de
fined by means of the equality constraints. Let TR(q) =
�ˆ ex(q), ˆ ey(q), ˆ ez(q),xR(q)�
time t. Let further ϕB(q,t) and ϑB(q,t) be the angles one
has to rotate the vector vB(t) first around ˆ ex(q) and then
around ˆ ey(q), respectively, so that the resulting vector v�
Hcool(q,t) =
4
1
N
N
∑
i
t4
min,i,
be the robot TCP frame (see
Fig. 5) and xB(t) and vB(t) the ball position and velocity at
B
Page 6
"soft""latest""cool"
0.00.20.40.6 0.81.0
�1
0
1
2
t �s�
q��rad�s�
0.00.20.40.60.81.0
�2
�1
0
1
t �s�
0.00.20.4 0.60.8 1.0
�1
0
1
2
3
t �s�
�7�
�6�
�5�
�4�
�3�
�2�
�1�
Fig. 7.
row the joints velocity ramps qi(t). For “latest” mode the robot catches the ball later (tc,latest= 1.0s) on the ball trajectory and has therefore to move much
faster and further, than in “soft” (tc,soft= 0.94s) and “cool” (tc,cool= 0.33s and tg= 0.94s) mode. The catch configurations for “soft” and “cool” mode are
almost the same, but the catch times differ significantly. The smoothest velocity ramps are achieved in “soft” mode, whereas the other two modes reach
the acceleration and velocity limits in some joints.
Planning results for a simulated ball trajectory. For each catch behavior the first row shows the resulting catch configuration qcand the second
becomes anti parallel to ˆ ez(q). The equality constraints can
then be written as
xR(q)
ϕB(q,t)
ϑB(q,t)
=
=
=
xB(t),
0,
0.
The inequality constraints take care, that the solution
(qc,tc) stays inside the feasible part of the solution space
and are defined as follows.
• joint angle limits:
qmin,i≤ qi≤ qmax,i,
• catch time limits, with tmax,c= 1.8s for our setup:
0 <t <tmax,c
i = 1...N
• minimal time limits, which avoid infeasible ramps, i.e.
ramps which need a a > amax or joint velocities ω >
ωmax:
t ≥tmin,i(qi),
with tmin,ias defined in Sec. IVB.1.
• work cell limits, where d(x,W) is the distance between
a point x and one of the NW= 8 workcell objects W as
defined in Fig. 6:
i = 1...N,
d(xR(q),Wk) > 0,
k = 1...NW.
In sum, there are Nh= 5 equality and Ng= 2N+2+N+
8 = 31 inequality constraints.
C. Simulation Results
To visualize the differences in the three catch behaviors,
for each mode the planner was run for the very same simu
lated ball trajectory and the very same start configuration q0
as shown in Fig. 2. The catch behavior differs significantly
and is discussed in Fig. 7.
0.600.65 0.70 0.75 0.80
�10
�5
0
5
q��rad�s�
�3�
�2�
�1�
0.600.650.70
t �s�
0.750.80
�0.4
�0.2
0.0
0.2
0.4
0.6
Τ �Nm�
�3�
�2�
�1�
Fig. 8.
(∆t = 40ms). According to the catch frame the ball approaches the index
and the middle finger. At the third image of the sequence the ball is already
caged but still moving. Note the fast motion of the thumb and the ring
finger to block the flight channel. In the last image the stable grasp of the
ball is observed. The second and third row show plots of the velocities and
joint torques of the thumb (for the very same throw as the “soft” mode
throw in Fig. 9). The impact at time tc= 0.63 can be clearly seen as it
is accompanied by a peak in velocity and impact torques that appear at
the joints. The large initial peak is due to the impact while the following
vibrations are also due to the short vibration phase of the arm that induce
inertial forces in the fingers. In the stable grasp phase the thumb torques
become very small indicating that the index and the ring finger fix the ball
in the hand.
First row shows the hand grasping the ball for a typical real catch
Page 7
"soft""latest"
0.00.20.4 0.6 0.8
�0.5
0.0
0.5
1.0
1.5
2.0
2.5
qd��rad�s�
�7�
�6�
�5�
�4�
�3�
�2�
�1�
0.00.20.40.6 0.8
�0.5
0.0
0.5
1.0
1.5
2.0
2.5
0.00.2 0.40.6 0.8
�0.5
0.0
0.5
1.0
1.5
2.0
2.5
q��rad�s�
0.0 0.2 0.40.6 0.8
�0.5
0.0
0.5
1.0
1.5
2.0
2.5
0.00.20.4
t �s�
0.6 0.8
�40
�20
0
20
40
Τ �Nm�
0.00.2 0.4
t �s�
0.6 0.8
�60
�40
�20
0
20
9.
“latest” mode catch behavior. For two al
most similar ball trajectories (thrown by a
human) each column depicts first the final
catch configuration and then the plots of
the commanded ˙ qd,i(t) and measured ˙ qi(t)
joint velocity ramps and the measured joint
torques. The catch times are tc,soft= 0.63s
and tc,latest= 0.70s.
A noticeable difference to the simulation
results are the kinks in the commanded
velocity ramps. These are due to the errors
in the ball trajectory prediction: the kinks
occur at the times, when the prediction (and
hence the prediction error) jumps (compare
Fig. 3 for the “soft” mode throw; note, that
the time delay of 90ms between camera
shutter time and the robot reacts to this new
measurement has been compensated for; so
a prediction at time t in Fig. 3 leads to robot
reaction at the same time t in Fig. 9).
The executed velocity ramps follow very
precisely the commanded ramps, except for
a little contouring error. Even the torques
roughly resemble the ideally expectable ac
celeration step functions, however super
posed with noise and dynamical and gravi
tational forces the planner does not account
for. This justifies our assumptions “kinemat
ically planning” and “trapezoidal velocity
ramps”. Also the timing is very precise:
the impact of the ball is clearly visible
in the torque signals and happens at the
commanded catch time.
Experimental results for “soft” and
Fig. 10.
and t ≈tc,latest+20ms = 0.72s for the last image.
Image sequence (from upper left to lower right) for “latest“ mode catching with ∆t = 120ms (same throw as in Fig. 9) and t ≈ 0s for the first
Fig. 11.
(“soft” mode).
Consecutive catch configurations for a continuous catch sequence without moving back to a start configuration before the next ball is thrown
Page 8
V. EXPERIMENTS
In Fig. 9 we show experimental results for the “soft”
and “latest” catch modes recorded for two similar (as fas
as possible for the human thrower) throws. The “cool”
mode is not shown, because, as discussed in Sec. IVC, the
catch configuration would be almost the same as for the
“soft” mode and the distinguishing property, that the catch
configuration is reached early and then the robot stays still
while waiting for the ball to arrive, is corrupted due to the
ever changing predictions of the ball trajectory because of
tracking errors in the vision system3.
For the “latest” mode Fig. 10 (same throw as in Fig. 9)
shows an image sequence of the catch movement and Fig. 8
a closeup of the hand while grasping the ball.
To get an impression of the wide range of feasible catch
configurations, in Fig 11 a sequence of consecutive catch
configurations for the “soft” mode is shown, where, other
than usual, the robot did not move back to its start config
uration before a new throw is made, but uses the last catch
configuration as its new start configuration, i.e. q(n+1)
This justifies our third assumption, the collision avoidance
heuristics.
The catch success rate (number balls caught versus the
number of balls thrown into the feasible catch spaces) is
> 80% for “soft“ and “latest“ mode, where the main source
of failures is the vision system with its prediction errors and
lost ball tracks. Additional results are contained in the the
video attachment to this paper.
0
=q(n)
c .
VI. CONCLUSION
Besides being an attraction in demos, in the first place ball
catching is, due to its dynamic nature, an excellent testbed
for robotic key skills and their tight interaction under hard
timing constraints, where even the robotic nonspecialist can
easily judge the performance of the system.
In our system currently the visual ball tracking is the
limiting factor for the catch rate, whereas the mechanics and
control of the arm and especially of the hand work highly
reliable in grasping the ball. Also, the unified view onto
the planning problem by using online kinematically realtime
optimization proves – besides some assumptions – to gen
erate movements which operate near the performance limits
(speed and acceleration) without damaging the mechanics
and can take fully advantage of the redundancy in the prob
lem4. Moreover, the chosen approach allows to intuitively
formulate objective functions to determine the desired catch
behavior. Finally, the system architecture permits a tight
interplay of the modules running in parallel on distributed
computing resources under hard realtime constraints.
Currently we are substantially extending our methods to
bring ball catching to our mobile humanoid robot “Rollin’
Justin” [21]. This poses, mainly due to the much larger
number of degrees of freedom (22 DOF including the mobile
3Hence, the robot can not stay still at all until the catch time !
4The robot has 7 DOF, but only 4 DOF are determined, because the
orientation of the catch frame around the zaxis and the catch time, when
to catch the ball on its trajectory, are arbitrary
base) with much more complex dynamical couplings, a
number of new, interesting challenges for all components
of the system.
REFERENCES
[1] B. Hove and J. Slotine, “Experiments in robotic catching,” in Proceed
ings of the 1991 American Control Conference, 1991, pp. 380–385.
[2] W. Hong and J. Slotine, “Experiments in handeye coordination using
active vision,” in Proceedings of the Fourth International Symposium
on Experimental Robotics, ISER95, 1995.
[3] K. NISHIWAKI, A. KONNO, K. NAGASHIMA, M. INABA, and
H. INOUE, “The humanoid saika that catches a thrown ball,” in
Proceedings of the lEEE Intermtional Workshop on Robot and Human
Communication, 1997, pp. 94–99.
[4] U. Frese, B. B¨ auml, S. Haidacher, G. Schreiber, I. Schaefer,
M. H¨ ahnle, and G. Hirzinger, “Offtheshelf vision for a robotic ball
catcher,” in Proc. IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS), 2001.
[5] M. Riley and C. G. Atkeson, “Robot catching: Towards engaging
humanhumanoid interaction,” Autonomous Robots, vol. 12, pp. 119—
128, 2002.
[6] C. Smith and H. I. Christensen, “Using cots to construct a high per
formance robot arm,” in IEEE International Conference on Robotics
and Automation (ICRA), 2007.
[7] G. Hirzinger, N. Sporer, A. AlbuSch¨ affer, M. H¨ ahnle, R. Krenn,
A. Pascucci, and M. Schedl, “DLR’s torquecontrolled light weight
robot III  are we reaching the technological limits now?” in Pro
ceedings of the IEEE/RSJ International Conference Robotics and
Automation (ICRA), 2002, pp. 1710–1716.
[8] J. Butterfaß, M. Grebenstein, H. Liu, and G. Hirzinger, “DLRHand
II: Next generation of a dextrous robot hand,” in Proceedings of the
IEEE/RSJ International Conference Robotics and Automation (ICRA),
2001, pp. 109–114.
[9] QNX Software Systems. [Online]. Available: http://www.qnx.com/
[10] P. Spellucci, “A sqp method for general nonlinear programs using only
equality constrained subproblems,” MATHEMATICAL PROGRAM
MING, vol. 82, pp. 413–448, 1998.
[11] ——,“Donlp2shortusers
darmstadt.de/pub/department/software/opti/DONLP2, 1999.
[12] B. B¨ auml and G. Hirzinger, “When hard realtime matters: Software
for complex mechatronic systems,” Robotics and Autonomous Systems,
vol. 56, no. 1, pp. 5–13, 2008.
[13] A. AlbuSch¨ affer, O. Eiberger, M. Fuchs, M. Grebenstein, S. Had
dadin, Ch. Ott, A. Stemmer, T. Wimb¨ ock, S. Wolf, and G. Hirzinger,
“Anthropomorphic soft robotics  from torque control to variable in
trinsic compliance,” in International Symposium on Robotics Research,
2009.
[14] Ch. Ott, A. AlbuSch¨ affer, A. Kugi, and G. Hirzinger, “On the passivity
based impedance control of flexible joint robots,” IEEE Transactions
on Robotics, vol. 24, no. 2, pp. 416–429, April 2008.
[15] R. M. Murray, Z. Li, and S. S. Sastry, A Mathematical Introduction
to Robotic Manipulation.CRC Press, 1994.
[16] M. G¨ orner, T. Wimb¨ ock, and G. Hirzinger, “The DLR crawler:
evaluation of gaits and control of an actively compliant sixlegged
walking robot,” Industrial Robot: An International Journal, vol. 36,
no. 4, pp. 344–351, 2009.
[17] S. M. LaValle, Planning Algorithms.
2006.
[18] H. GONZ´ALEZBA˜NOS, D. HSU, and J. LATOMBE:, “Motion
planning: Recent developments,” in Automous Mobile Robots: Sensing,
Control, DecisionMaking and Applications. CRC, S. u. F. L. GE, Ed.,
2006.
[19] D. BERENSON, S. SRINIVASA, D. FERGUSON, and J. KUFFNER,
“Manipulation planning on constraint manifolds,” in IEEE Interna
tional Conference on Robotics and Automation (ICRA09), 2009.
[20] J. BETTS, “Survey of numerical methods for trajectory optimization,”
Journal of Guidance, Control, and Dynamics, vol. 21, no. 2, pp. 193–
207, 1998.
[21] C. Borst, T. Wimb¨ ock, F. Schmidt, M. Fuchs, B. Brunner, F. Zacharias,
P. R. Giordano, R. Konietschke, W. Sepp, S. Fuchs, C. Rink, A. Albu
Sch¨ affer, and G. Hirzinger, “Rollin’ justin: Mobile platform with vari
able base,” in Proceedings of the 2009 IEEE International Conference
on Robotics and Automation (ICRA 2009), 2009, pp. 1597–1598.
Acknowledgements: We thank the arm and hand development teams.
guide,”ftp://ftp.mathematik.tu
Cambridge University Press,
View other sources
Hide other sources
 Available from Thomas Wimböck · May 21, 2014
 Available from DLR
 Available from dlr.de
 Available from dlr.de