Experimental analysis of the conditions of applicability of a robot sensorimotor coordination scheme based on expected perception
ABSTRACT This paper describes an experimental work conducted in order to estimate the conditions of applicability of expected perception (EP) based on a scheme for robot sensorimotor coordination. The starting hypothesis is that predictions of incoming sensory data can improve sensorymotor coordination respect to pure feedback loops. This implies that the environment presents a level of predictability, as in realistic environments. An implementation of the EP-based scheme has been realized on a platform composed by the Dexter 8-d.o.f. robotic arm and a color camera, for executing a pushing task in a real-world environment. Its performance, where defined as a combination of the error in the trajectory following and the computational effort, has been compared with that of a feedback-based system executing the same task in the same environmental conditions. The results have been put in relation with the degree of environmental predictability, which was controlled in the experimental trials. The experimental results give support and useful insights for analyzing the applicability of the EP-based scheme.
- SourceAvailable from: dspace.org[show abstract] [hide abstract]
ABSTRACT: A new architecture for controlling mobile robots is described. Layers of control system are built to let the robot operate at increasing levels of competence. Layers are made up of asynchronous modules that communicate over low-bandwidth channels. Each module is an instance of a fairly simple computational machine. Higher-level layers can subsume the roles of lower levels by suppressing their outputs. However, lower levels continue to function as higher levels are added. The result is a robust and flexible robot control system. The system has been used to control a mobile robot wandering around unconstrained laboratory areas and computer machine rooms. Eventually it is intended to control a robot that wanders the office areas of our laboratory, building maps of its surroundings using an onboard arm to perform simple tasks.IEEE Journal on Robotics and Automation 04/1986;
- [show abstract] [hide abstract]
ABSTRACT: The visual information obtained from a camera and an image processing unit are incorporated in an adaptive control algorithm to make a robotic manipulator grasp a moving object. Because of the inherent time delay caused by the image processing, the motion of the moving target is predicted in real time and is used in the online planning of the trajectory for the manipulator motion. Since the dynamics of the target are assumed to be unknown, the prediction is accomplished by means of an autoregressive discrete-time model. On the basis of the predicted motion of the object, the planner determines online at each control sampling instant the desired trajectory point (subgoal) for the controller. The subgoal point is tracked by controlling the end effector with self-tuner until grasping occurs. A simulation study and laboratory experiments are presented to demonstrate the performance of this visual feedback systemIEEE Transactions on Systems Man and Cybernetics 02/1991;
Conference Proceeding: Model based techniques for robotic servoing and grasping[show abstract] [hide abstract]
ABSTRACT: A robotic manipulation of objects typically involves object detection/recognition, servoing to the object, alignment and grasping. To perform fine alignment and final grasping, it is usually necessary to estimate the position and orientation (pose) of the object. In this paper we present a model based tracking system used to estimate and continuously update the pose of the object to be manipulated. Here, a wire-frame model is used to find and track features in the consequent images. One of the important parts of the system is the ability to automatically initiate the tracking process. The strength of the system is the ability to operate in an domestic environment (living room) with changing lighting and background conditions.Intelligent Robots and Systems, 2002. IEEE/RSJ International Conference on; 02/2002
Experimental Analysis of the Conditions of
Applicability of a Robot Sensorimotor
Coordination Scheme based on Expected
Edoardo Datteri∗§, Gioel Asuni∗, Giancarlo Teti∗, Cecilia Laschi∗, Paolo Dario∗, Eugenio Guglielmelli∗
∗Scuola Superiore Sant’Anna
ARTS Lab (Advanced Robotics Technology and Systems Laboratory)
Polo Sant’Anna Valdera
Viale Rinaldo Piaggio, 34, 56025 Pontedera (Pisa), Italy
§University of Pisa
Department of Philosophy
Piazza Torricelli 3/a, 56126 Pisa, Italy
Abstract—This paper describes an experimental work
conducted in order to estimate the conditions of applicability
of Expected Perception (EP) based a scheme for robot
sensorimotor coordination. The starting hypothesis is that
predictions of incoming sensory data can improve sensory-
motor coordination respect to pure feedback loops. This
implies that the environment presents a level of predictability,
as in realistic environments.
An implementation of the EP-based scheme has been
realized on a platform composed by the Dexter 8-d.o.f. robotic
arm and a color camera, for executing a pushing task in
a real-world environment. Its performance, where defined
as a combination of the error in the trajectory following
and the computational effort, has been compared with that
of a feedback-based system executing the same task in the
same environmental conditions. The results have been put in
relation with the degree of environmental predictability, which
was controlled in the experimental trials. The experimental
results give support and useful insights for analyzing the
applicability of the EP-based scheme.
An important challenge for robotics is developing robots
able to operate efficiently and autonomously in realistic,
non rigidly structured environments. This is a crucial
challenge for every domain of application that requires
interaction between humans and robots co-inhabiting the
environments in which humans spend their everyday live -
the so-called “real world”.
Real world is usually dynamic and rich of elements that
change in time and can seriously affect the performance of
the robot. In order to cope with such environments, reactive
architectures were proposed , , based on pure sensory
feedback: the agent must act only in response to local, just-
perceived stimuli. Memory, representations and symbolic
reasoning are often rejected in this conceptual framework,
as redundant and time-consuming. In this approach, real
world is often regarded as completely unstructured and
The broad assumption underlying the present work is
that real world is indeed dynamic and rich of potentially
disturbing elements, yet without being completely unstruc-
tured, chaotic and unpredictable. We claim that the real
world is mostly partially structured, or predictable. The
point is, broadly speaking, that robots can exploit the
predictability of the real world in order to avoid unnec-
essary full perceptual processing by means of anticipation
This approach opens interesting possibilities for over-
coming one of the major limitations deriving from the
unavoidable delays in perception-action loops, mainly aris-
ing when complex sensory data are involved. These delays
can make the robot motor response to external stimuli too
slow with respect to the strict time requirements imposed
by realistic environments. This problem is discussed in
robotics, for what concerns various domains of application
, , , . In  this issue was discussed in connec-
tion with explanatory models of sensorimotor coordination
in humans , .
In ,  the EP-based scheme was proposed, as a
general scheme for developing sensory-motor coordination
systems. It is based on a mechanism of sensory anticipa-
tion. During the execution of a task (a planned sequence
of actions), at each step the issued motor command is
the basis for the generation of an expected perception
(EP), that is a simulated sensory pattern (corresponding
to what the system should perceive in that moment with
its sensors). Other crucial elements are involved in the
generation of EPs; namely, an internal model representing
some aspects of the scenario, and an internal model of
the robot itself. After its generation, the EP is compared
with the perception acquired by the system sensors. If
the EP matches the actual perception, then everything is
going as expected, because the EP represents, in a sense,
the consequence of the correct execution of the planned
actions. Otherwise, if there is a mismatch between the
EP and the actual perception, it means that something
unexpected has occurred. In this case, sensory processing
is performed and next actions are replanned accordingly.
This mechanism is intended as a way for overcoming
full perceptual processing at each perception-action step.
Actual perception is first just compared with EPs; it is
fully processed only in case of unexpected events. The EP-
based scheme has been implemented on a anthropomorphic
robotic platform for manipulation and first data about its
feasibility have been acquired , .
II. OBJECTIVES AND METHODOLOGY
The main hypothesis motivating the present work is that
the EP-based scheme can be fruitfully applied for realizing
visuo-motor coordination systems, especially for manipu-
lation, (1) when the environment is predictable to some
extent, and (2) the perceptual apparatus that needs to be
employed involves time-consuming computations. When
both the conditions are true (we postpone to section IV
a discussion on environmental predictability), the adoption
of the EP-based scheme should determine higher perfor-
mances with respect to other architectures of sensorimotor
An exhaustive test of this broad hypothesis is beyond the
ambitions of this paper. The objective of the present work
is, rather, to provide some initial experimental support for
corroborating or rejecting it. In particular, experimental tri-
als performed in this work enable (a) to identify for which
degree of environmental predictability the EP scheme is
suitable, and (b) to evaluate to what extent the adoption
of the EP scheme determines an increase in performance.
These aims will be referred to, from now on, as the aim of
evaluating the conditions of applicability of the EP scheme.
This work is intended to give more experimental support
to purely theoretical considerations about the increase in
performance deriving from the adoption of the EP scheme,
proposed in . A measure of predictability has been in-
troduced. A paradigmatic implementation of the EP scheme
has been realized (EPbs from now on), whose performance
has been measured in different environments, each one
characterized by a different degree of predictability. Then,
it has been compared with the performance of another
implemented system, a feedback-based system (Fbs), that
continuously processes sensory data during task execution.
Both systems execute the same task (pushing an object
towards a desired point in space).
Details on the implementation of the two systems and
on the experimental task are provided in Section III. More
details about the measure of performance and the quan-
tification of the degree of environmental predictability are
reported in Section IV. Experimental data are summarized
in Section V, while Section VI discusses the impact of
experimental data on objectives (a) and (b) above.
A. Experimental setup
The anthropomorphic Dexter 8 d.o.f. robotic arm has
been used for the implementation. A 320*240 color cam-
era, located on top of the experimental setting, has been
used as sensory input. The experimental setting is showed
in Fig. 1.
Fig. 1.The experimental setting.
As mentioned, a pushing task has been selected for
the experimental analysis. Both the EPbs and the Fbs
are capable of pushing an object from an initial position
reference system. Experimental trials help show to what
extent the performance of the two systems changes in
dependence of the degree of predictability. Thus, they can
be used for achieving the objectives expressed above in
points (a) and (b), i.e. conditions of applicability the and
as experimental support for evaluating the likelihood of the
0to a desired position Posi
dexpressed in the image
B. Implementation of the EP based system
The functional structure of the EPbs is an instantiation
of the general scheme in Fig. 2.
Fig. 2.The EP-based scheme.
In Fig. 3 a flowchart illustrates the sequence of opera-
tions performed by the system, as explain below. The object
to be pushed is identified by color.
In the algorithm, a threshold value is first calculated
for selecting the object out of the background; the bitmap
corresponding to the object to be pushed is stored; its
centroid position Posi
tions. A motor command towards the desired position
from the image Cartesian reference system to the robot
reference system, is then sent to the robot; during the
arm motion (pushing), each ms milliseconds (ms = 500
in the performed trials), until the distance between object
0is calculated in its initial posi-
d∗ M, where M is a transformation matrix
Fig. 3.Flowchart of the EP based system.
centroid and target point is below a threshold, and the x
position of the object centroid is higher than the x position
of the target point (see Fig. 4), the system executes the
following actions: it acquires an image; it generates an EP
by the (“EXPECTED PERCEPTION” module in Fig. 2);
and it calculates the difference (error) between the acquired
image and the EP over a small attention window (see
described in Section III.
Illustration of some variables related to the pushing task and
between the EP and the actual image over the attention window.
Left: an EP. Middle: an acquired image. Right: the difference
If error is above a threshold (set to 6% of the whole
attention window area), the last image acquired by the
camera is processed by the (“SENSOR PROCESSING”
module) in order to find the object centroid position Posi
at current time t; the angle ang, laying between the segment
d, and the segment connecting
positioned as in Fig. 4, with orientation directly related
with the value of ang and position related with Posi
new motor command towards Posi
The EP generation mechanism includes a forward model
(i.e. a model that calculates sensory consequences of motor
actions). In the present implementation, it is articulated
in a forward dynamic model and a forward output model
(taking inspiration from ). The forward dynamic model
calculates the function
dis calculated; the robot end effector is
posed to be at time t, and Vr
The forward output model computes Posi
i.e. the (u,v) position in the image reference system corre-
sponding to Posr
to the object being pushed onto a previously acquired back-
ground, and obtains the EP. Note that the present forward
model makes use of background knowledge (“INTERNAL
MODEL” module of Fig. 2), including information about
arm velocity, visual background, and bitmap corresponding
to the object of interest.
The position of the attention window over the whole
EP is calculated on the basis of Posi
pixel difference is then performed between the acquired
image and the EP; pixels that correspond to a significant
difference are marked as red or green (depending on the
sign of the difference).
The SENSOR PROCESSING module is capable of
calculating the centroid of the object of interest. A blob-
coloring technique has been implemented in order to
identify the object of interest even if the image includes
many regions of the same color. The set of all the possible
regions of interest is identified by image segmentation. The
threshold for image segmentation is continuously updated
during thresholding: whenever a pixel whose color is near
to a reference color is found, the system searches for
similarly colored pixels in its vicinity. If other pixels are
found, the reference color is updated based on their color.
When a region has been completely identified, its centroid,
its area, its perimeter and its vertices are calculated. Area,
perimeter and vertices (for determining geometric shape)
are used in order to find the object of interest. Fig. 6 shows,
in a pictorial form, the result of the application of a raw
image to the SENSOR PROCESSING module. The whole
computation takes approximately 200 ms.
Note that the system can deal efficiently with detailed
or changing backgrounds even if the ‘EP-based loop’
is active. This occurs provided that at time t no object
moves within the region corresponding to the attention
window: the value of error does not depend on differences
between the two images and laying outside the attention
window. Moreover, the presence of static regions in the
image, possibly of the same color as the object of interest,
does not affect the calculation of error: as they are also
included in the background image, and consequently in
tis the position in which end-effector is sup-
tis the end-effector velocity.
t. Then, it pastes the bitmap corresponding
t. A simple pixel-
outputs of the Sensory Processing module. The pixels of the region of
interest (red cube) are shown, together with the centroid and the vertices.
Left: an acquired image. Right: an image showhing some of the
the EP, no difference can be detected. The capability of the
SENSOR PROCESSING module of identifying the object
of interest even in the presence of other similarly colored
regions, makes the system capable to deal with complex,
unstructured background even when the ‘Perception-Action
loop’ is active.
C. Implementation of the reactive feedback-based system
The Fbs is functionally described as a typical feedback-
based scheme (see Fig. 7).
Fig. 7.A Typical feedback-based scheme.
It involves the SENSOR PROCESSING module already
described in Section III-B. As in the EPbs, a threshold
value for performing good image thresholding is calcu-
lated; the initial centroid position Posi
motor command towards the desired position Posr
is acquired and the centroid position Posi
interest is calculated. The angle ang, defined as in the EPbs,
is then calculated; the robot end effector is positioned as
in Fig. 4, with orientation directly related with the value
of ang and position related with Posi
command towards Posi
0is calculated; a
d∗M is sent to the robot. Then, at each step, an image
tof the object of
t. Then, a new motor
IV. EXPERIMENTAL TRIALS
The two implemented systems have been tested in tasks
requiring interaction with objects whose behavior was
predictable to different extents. In the present approach,
objects are highly ”‘predictable”’ if their motion is caused
by (and can be mathematically obtained from) the robot
motion; objects that move autonomously with respect to
the robot motion, or that are only loosely dependent on
it, are predictable only at a lower degree. The degree of
predictability is indeed not only an environmental property,
rather depending on the perceptual and computational
capabilities of the system itself. The notion of “proactivity”
presented in  may help to refine this working definition.
To evaluate the benefits deriving from the adoption of
the EP scheme for sensorimotor coordination, a measure
of predictability has been proposed, that is highly specific
for the chosen pushing task (the possibility of applying the
proposed measure to other sensorimotor coordination tasks
has not been explored). Nevertheless, even if with respect
to the pushing task only, it helps appreciate interesting
features of the proposed scheme.
Let the desired trajectory be expressed as
k) | k = 1,...,m}
and the trajectory executed during the jthtrial be expressed
k) | k = 1,...,m}
Then, the Error of Position on Trajectory (EPT) at trial
jthis defined as
The EPT over n trials is defined as
EPT is regarded as a quantitative measure of the ”‘pre-
dictability”’ of a given object to follow the pushing tra-
jectory. For experimental trials, a some parallelepiped was
been provided with different degrees of predictability by
adding a small weight inside, close to one of its extremities
(see Fig. 8, lower part, for a top view of the parallelepiped).
Specifically, weights of 50g, 20g and no weight have been
Fig. 8. Values of EPT for different object/weight configurations.
The different degrees of predictability have been mea-
sured by pushing the parallelepiped along a given straight
trajectory for 10 times for each of the 3 cases: no weight,
20g weight, 50g weight. Trajectories have been sampled
at a frequency of 2 Hz, and the value of EPT has been
calculated for each of the 3 group of trials (see the plot
in Fig. 8). The no-weight case exhibits the minimum EPT
The performance of each implemented system has been
measured in the same task (pushing an object from Posr
figurations, corresponding to a different value of EPT.
Fig. 1 shows the experimental setting. The parameters for
measuring performances are:
• Error of Position on Path (EPPr) with respect to the
d), but with the three different parallelepiped con-
• Computational Reduction (for the EP based system
only; CR from now on),
• Error on Target (ET from now on)
EPPiis defined as follows. Let s be the line including
centroid has been detected while the object is pushed
0 < k ≤ m. Let d(Pk,s) be the distance between Pk
and s. Then, the EPPiat trial jth(EPPi
d; let m be the number of times the object
d; let Pk = (u,v) be the kthcentroid, with
j) and EPPiare
This quantity is expressed in pixels. EPPris the same
quantity, expressed in cm.
CR is defined as follows. Let the number of Total
Control Steps (TCS) of the EPbs represent the number of
times a comparison between EP and actual image has been
performed, during the pushing task. Let the number of Pro-
cessing Steps (PS) represent the number of times acquired
images have been processed in order to re-plan a path (in
case of a high value of error). CR is the percentage of
(TCS - PS) over TCS. It is closely related to the time of
execution as, the higher it is, the higher is the capability of
the system to avoid highly time-consuming sensory (visual,
in this case) processing operations. Moreover, as the Fbs
(and feedback-based systems in general) always executes
image processing, its CR value is 0% by definition. CR
is then to be intended as a measure of the computational
reduction associated to sensory processing, with respect to
the feedback-based system.
ET is defined as
?(xd− x)2+ (yd− y)2, that is the
distance between Posi
In the experimental trials, the desired position coordi-
nates were set to (x = 153;y = 102), with respect to the
reference system illustrated in Fig. 9. The initial position
of the object, with respect to the image reference system,
was approximately (260,102) in all the trials. Fig. 9 shows
the trajectory represented in the image space.
d= (xd,yd) and the final point
Initial and desired positions expressed in the image reference
V. EXPERIMENTAL RESULTS
The following plots summarize the experimental results.
First, the value of ET for the two systems, expressed in
cm, is reported (Fig. 10);
value of object ”‘predictability”’.
Values of ET for the two systems, expressed in cm, for the 3
The values of EPPrfor the two systems with respect
to the three degrees of environmental predictability are
indicated in Fig. 11.
the 3 levels of environment predictability. The darker line corresponds to
Fbs, while the lighter line corresponds to EPbs.
Values of EPPrfor the two systems, expressed in cm, for
The plot in Fig. 12 reports the mean value of CR of
the EPbs over 20 trials, for each degree of environmental
Fig. 12.Values of CR for the EPbs.
Note that the value of EPProf the Fbs is generally
lower than that of the EPbs. Nevertheless, the higher the
environmental predictability, the higher the accuracy of the
EPbs. The value of ET for the EPbs is always higher than
that of the Fbs; every system is capable of positioning the
cube within a small area centered on the desired point.
As regards the value of CR, expressing the computational
reduction with respect to the Fbs, it is always high, even if
it decreases as the degree of partial predictability increases.
Under this respect, the obtained experimental results show
that the adoption of the EPbs for the described pushing
task determined a large increase in performance (in all
the three degrees of partial predictability), as it allowed
for a remarkable reduction of the computational burden
associated to sensory processing (with respect to the Fbs).
In order to combine all these results into a coherent view
about the conditions of applicability of the EP scheme, a
measure of performance P for the two system has been
Performance = (1 − EPP?) ∗ KEPP+ (1 − ET?) ∗
KET+CR∗KCRwhere EPP?and ET?are EPP and ET
normalized into a [0,1] range and KEPP,KET, and KCR
are coefficients. If the contribution of CR is ignored for
evaluating the performance, (thus taking into account only
ET and EPP), the Fbs exhibits the better performance in
all the degrees of predictability (see Fig. 13). In Fig. 14
the value of Performance is plotted, taking into account
(even if only in low percentage) the contribution of CR. It
is easy to see that the performance of the EPbs increases
much more than that of the Fbs as the degree of predictabil-
ity increases, until it definitely overcomes the latter.
Performances of the two systems with KCR=0, KET=0.5 and
Performances of the two systems with KCR=0.2, KET=0.4
VI. CONCLUSIONS ON CONDITIONS OF APPLICABILITY
OF THE EP-BASED SCHEME
The obtained experimental results are consistent with the
following general conclusions concerning the conditions of
applicability of the EP scheme and the hypothesis under-
lying this work (see Section I). In environments whose
degree of predictability lies between the no predictability
and the maximum predictability (completely structured
environments), the performance of the EP scheme guar-
antees high computational reduction and a reasonably high
accuracy. The higher is the importance given to the purpose
of decreasing the computational reduction of sensorimotor
coordination, the higher are the benefits deriving from the
adoption of the EP scheme. Thus, the obtained results cor-
roborate the starting hypothesis that the EP-based scheme
can be fruitfully applied (determining higher performances
with respect to other architectures of sensorimotor coor-
dination) when the environment is predictable (even if
only in some degree), and the perceptual apparatus that
needs to be employed involves temporally time-consuming
computations, and the main interest is to reduce this kind
of computational burden. Different implementations of the
EP scheme may be useful for refining the estimation of
the conditions of applicability of the EP based scheme,
under many respects. As an example, the internal forward
model used for generating sensory expectations may be
made algorithmically more complex, allowing the system
to predict environmental changes more efficiently. It would
be useful to understand when the increase in complexity
of the internal model would make the generation of EPs
too much time-consuming, thus reducing the performance
of EP systems with respect to feedback-based systems.
Additionally, the notion of ’environmental predictability’
might be given different quantitative counterparts, taking
into account other environmental factors potentially inter-
fering with the system.
 R. C. Arkin. Behavior-based Robotics. MA: MIT Press, 1998.
 R. Brooks. A robust layered system for a mobile robot. IEEE
Journal of Robotics and Automation, 2:14–23, 1986.
 R. Moller.Perception through anticipation - an approach to
behaviour-based perception.In Proc. New Trends in Cognitive
Science, pages 184–190, 1997.
 A. J. Koivo. Real-time vision feedback for servoing robotic manip-
ulator with self-tuning controller. IEEE Transactions on Systems,
Man, and Cybernetics, 21(1):134–142, 1991.
 D. Kragic and H. I. Christensen.
robotic servoing and grasping. In Proceedings of the 2002 IEEE/RSJ
International Conference on Intelligent Robots and Systems, EPFL,
Lausanne, Switzerland, pages 299–304, 2002.
 A. Namiki and M. Ishikawa. Robotic catching using a direct map-
ping from visual information to motor command. In Proceedings
of the IEEE International Conference on Robotics and Automation,
pages 2400–2405, 2003.
 E. Datteri, G. Teti, C. Laschi, G. Tamburrini, P. Dario, and
E. Guglielmelli. Expected perception in robots: a biologically driven
perception-action scheme.In Proceedings of ICAR 2003, 11th
International Conference on Advanced Robotics, volume 3, pages
 R. S. Johansson. Sensory input and control of grip. In Sensory
Guidance of Movement, Novartis Foundation Symposium, volume
218, pages 45–59, 1998.
 D. M. Wolpert, R. C. Miall, and M. Kawato. The role of internal
models in the cerebellum. Trends in Cognitive Sciences, 2:338–347,
 E. Datteri, G. Teti, C. Laschi, G. Tamburrini, P. Dario, and
E. Guglielmelli.Expected perception: an anticipation-based
perception-action scheme in robots. In IROS 2003, 2003 IEEE/RSJ
International Conference on Intelligent Robots and Systems, Las
Vegas, Nevada, pages 934–939, 2003.
 D. M. Gabbay, R. Nossum, and M. Thielscher. Agents in proactive
environments. J. Log. Comput., 9(1):25–45, 1999.
Model based techniques for