Content uploaded by Yezhou 'YZ' Yang
Author content
All content in this area was uploaded by Yezhou 'YZ' Yang on Mar 21, 2015
Content may be subject to copyright.
Robot Learning Manipulation Action Plans by “Watching” Unconstrained Videos
from the World Wide Web
Yezhou Yang
University of Maryland
yzyang@cs.umd.edu
Yi Li
NICTA, Australia
yi.li@nicta.com.au
Cornelia Ferm¨
uller
University of Maryland
fer@umiacs.umd.edu
Yiannis Aloimonos
University of Maryland
yiannis@cs.umd.edu
Abstract
In order to advance action generation and creation in robots
beyond simple learned schemas we need computational tools
that allow us to automatically interpret and represent human
actions. This paper presents a system that learns manipula-
tion action plans by processing unconstrained videos from
the World Wide Web. Its goal is to robustly generate the se-
quence of atomic actions of seen longer actions in video in
order to acquire knowledge for robots. The lower level of the
system consists of two convolutional neural network (CNN)
based recognition modules, one for classifying the hand grasp
type and the other for object recognition. The higher level
is a probabilistic manipulation action grammar based pars-
ing module that aims at generating visual sentences for robot
manipulation. Experiments conducted on a publicly avail-
able unconstrained video dataset show that the system is able
to learn manipulation actions by “watching” unconstrained
videos with high accuracy.
Introduction
The ability to learn actions from human demonstrations is
one of the major challenges for the development of intel-
ligent systems. Particularly, manipulation actions are very
challenging, as there is large variation in the way they can
be performed and there are many occlusions.
Our ultimate goal is to build a self-learning robot that is
able to enrich its knowledge about fine grained manipulation
actions by “watching” demo videos. In this work we explic-
itly model actions that involve different kinds of grasping,
and aim at generating a sequence of atomic commands by
processing unconstrained videos from the World Wide Web
(WWW).
The robotics community has been studying perception
and control problems of grasping for decades (Shimoga
1996). Recently, several learning based systems were re-
ported that infer contact points or how to grasp an ob-
ject from its appearance (Saxena, Driemeyer, and Ng 2008;
Lenz, Lee, and Saxena 2014). However, the desired grasp-
ing type could be different for the same target object, when
used for different action goals. Traditionally, data about the
grasp has been acquired using motion capture gloves or hand
trackers, such as the model-based tracker of (Oikonomidis,
Copyright c
2015, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.
Kyriazis, and Argyros 2011). The acquisition of grasp in-
formation from video (without 3D information) is still con-
sidered very difficult because of the large variation in ap-
pearance and the occlusions of the hand from objects during
manipulation.
Our premise is that actions of manipulation are repre-
sented at multiple levels of abstraction. At lower levels the
symbolic quantities are grounded in perception, and at the
high level a grammatical structure represents symbolic in-
formation (objects, grasping types, actions). With the recent
development of deep neural network approaches, our system
integrates a CNN based object recognition and a CNN based
grasping type recognition module. The latter recognizes the
subject’s grasping type directly from image patches.
The grasp type is an essential component in the charac-
terization of manipulation actions. Just from the viewpoint
of processing videos, the grasp contains information about
the action itself, and it can be used for prediction or as a fea-
ture for recognition. It also contains information about the
beginning and end of action segments, thus it can be used to
segment videos in time. If we are to perform the action with
a robot, knowledge about how to grasp the object is neces-
sary so the robot can arrange its effectors. For example, con-
sider a humanoid with one parallel gripper and one vacuum
gripper. When a power grasp is desired, the robot should
select the vacuum gripper for a stable grasp, but when a pre-
cision grasp is desired, the parallel gripper is a better choice.
Thus, knowing the grasping type provides information for
the robot to plan the configuration of its effectors, or even
the type of effector to use.
In order to perform a manipulation action, the robot also
needs to learn what tool to grasp and on what object to per-
form the action. Our system applies CNN based recogni-
tion modules to recognize the objects and tools in the video.
Then, given the beliefs of the tool and object (from the out-
put of the recognition), our system predicts the most likely
action using language, by mining a large corpus using a
technique similar to (Yang et al. 2011). Putting everything
together, the output from the lower level visual perception
system is in the form of (LeftHand GraspType1 Object1 Ac-
tion RightHand GraspType2 Object2). We will refer to this
septet of quantities as visual sentence.
At the higher level of representation, we generate a sym-
bolic command sequence. (Yang et al. 2014) proposed a
context-free grammar and related operations to parse ma-
nipulation actions. However, their system only processed
RGBD data from a controlled lab environment. Further-
more, they did not consider the grasping type in the gram-
mar. This work extends (Yang et al. 2014) by modeling ma-
nipulation actions using a probabilistic variant of the context
free grammar, and explicitly modeling the grasping type.
Using as input the belief distributions from the CNN
based visual perception system, a Viterbi probabilistic parser
is used to represent actions in form of a hierarchical and
recursive tree structure. This structure innately encodes the
order of atomic actions in a sequence, and forms the basic
unit of our knowledge representation. By reverse parsing it,
our system is able to generate a sequence of atomic com-
mands in predicate form, i.e. as Action(Subj ect, P atient)
plus the temporal information necessary to guide the robot.
This information can then be used to control the robot effec-
tors (Argall et al. 2009).
Our contributions are twofold. (1) A convolutional neural
network (CNN) based method has been adopted to achieve
state-of-the-art performance in grasping type classification
and object recognition on unconstrained video data; (2) a
system for learning information about human manipulation
action has been developed that links lower level visual per-
ception and higher level semantic structures through a prob-
abilistic manipulation action grammar.
Related Works
Most work on learning from demonstrations in robotics has
been conducted in fully controlled lab environments (Aksoy
et al. 2011). Many of the approaches rely on RGBD sensors
(Summers-Stay et al. 2013), motion sensors (Guerra-Filho,
Ferm¨
uller, and Aloimonos 2005; Li et al. 2010) or specific
color markers (Lee et al. 2013). The proposed systems are
fragile in real world situations. Also, the amount of data used
for learning is usually quite small. It is extremely difficult to
learn automatically from data available on the internet, for
example from unconstrained cooking videos from Youtube.
The main reason is that the large variation in the scenery will
not allow traditional feature extraction and learning mecha-
nism to work robustly.
At the high level, a number of studies on robotic ma-
nipulation actions have proposed ways on how instruc-
tions are stored and analyzed, often as sequences. Work
by (Tenorth, Ziegltrum, and Beetz 2013), among others,
investigates how to compare sequences in order to reason
about manipulation actions using sequence alignment meth-
ods, which borrow techniques from informatics. Our paper
proposes a more detailed representation of manipulation ac-
tions, the grammar trees, extending earlier work. Chomsky
in (Chomsky 1993) suggested that a minimalist generative
grammar, similar to the one of human language, also ex-
ists for action understanding and execution. The works clos-
est related to this paper are (Pastra and Aloimonos 2012;
Summers-Stay et al. 2013; Guha et al. 2013; Yang et al.
2014). (Pastra and Aloimonos 2012) first discussed a Chom-
skyan grammar for understanding complex actions as a theo-
retical concept, (Summers-Stay et al. 2013) provided an im-
plementation of such a grammar using as perceptual input
only objects. (Yang et al. 2014) proposed a set of context-
free grammar rules for manipulation action understanding.
However, their system used data collected in a lab environ-
ment. Here we process unconstrained data from the internet.
In order to deal with the noisy visual data, we extend the ma-
nipulation action grammar and adapt the parsing algorithm.
The recent development of deep neural networks based
approaches revolutionized visual recognition research. Dif-
ferent from the traditional hand-crafted features (Lowe
2004; Dalal and Triggs 2005), a multi-layer neural network
architecture efficiently captures sophisticated hierarchies de-
scribing the raw data (Bengio, Courville, and Vincent 2013),
which has shown superior performance on standard object
recognition benchmarks (Krizhevsky, Sutskever, and Hinton
2013; Ciresan, Meier, and Schmidhuber 2012) while utiliz-
ing minimal domain knowledge. The work presented in this
paper shows that with the recent developments of deep neu-
ral networks in computer vision, it is possible to learn ma-
nipulation actions from unconstrained demonstrations using
CNN based visual perception.
Our Approach
We developed a system to learn manipulation actions from
unconstrained videos. The system takes advantage of: (1)
the robustness from CNN based visual processing; (2) the
generality of an action grammar based parser. Figure1 shows
our integrated approach.
CNN based visual recognition
The system consists of two visual recognition modules, one
for classification of grasping types and the other for recogni-
tion of objects. In both modules we used convolutional neu-
ral networks as classifiers. First, we briefly summarize the
basic concepts of Convolutional Neural Networks, and then
we present our implementations.
Convolutional Neural Network (CNN) is a multilayer
learning framework, which may consist of an input layer,
a few convolutional layers and an output layer. The goal
of CNN is to learn a hierarchy of feature representations.
Response maps in each layer are convolved with a number
of filters and further down-sampled by pooling operations.
These pooling operations aggregate values in a smaller re-
gion by downsampling functions including max, min, and
average sampling. The learning in CNN is based on Stochas-
tic Gradient Descent (SGD), which includes two main oper-
ations: Forward and BackPropagation. Please refer to (Le-
Cun and Bengio 1998) for details.
We used a seven layer CNN (including the input layer and
two perception layers for regression output). The first con-
volution layer has 32 filters of size 5×5, the second convo-
lution layer has 32 filters of size 5×5, and the third convo-
lution layer has 64 filters of size 5×5, respectively. The first
perception layer has 64 regression outputs and the final per-
ception layer has 6regression outputs. Our system considers
6grasping type classes.
Grasping Type Recognition A number of grasping tax-
onomies have been proposed in several areas of research, in-
Figure 1: The integrated system reported in this work.
cluding robotics, developmental medicine, and biomechan-
ics, each focusing on different aspects of action. In a recent
survey (Feix et al. 2013) reported 45 grasp types in the litera-
ture, of which only 33 were found valid. In this work, we use
a categorization into six grasping types. First we distinguish,
according to the most commonly used classification (based
on functionality) into power and precision grasps (Jeannerod
1984). Power grasping is used when the object needs to be
held firmly in order to apply force, such as “grasping a knife
to cut”; precision grasping is used in order to do fine grain
actions that require accuracy, such as “pinch a needle”. We
then further distinguish among the power grasps, whether
they are spherical, or otherwise (usually cylindrical), and
we distinguish the latter according to the grasping diame-
ter, into large diameter and small diameter ones. Similarly,
we distinguish the precision grasps into large and small di-
ameter ones. Additionally, we also consider a Rest position
(no grasping performed). Table 1 illustrates our grasp cat-
egories. We denote the list of these six grasps as Gin the
remainder of the paper.
Grasping
Types
Small Di-
ameter
Large Di-
ameter
Spherical
& Rest
Power
Precision
Table 1: The list of the grasping types.
The input to the grasping type recognition module is a
gray-scale image patch around the target hand performing
the grasping. We resize each patch to 32 ×32 pixels, and
subtract the global mean obtained from the training data.
For each testing video with Mframes, we pass the tar-
get hand patches (left hand and right hand, if present) frame
by frame, and we obtain an output of size 6×M. We sum
it up along the temporal dimension and then normalize the
output. We use the classification for both hands to obtain
(GraspType1) for the left hand, and (GraspType2) for the
right hand. For the video of Mframes the grasping type
recognition system outputs two belief distributions of size
6×1:PGraspT ype1and PGraspT y pe2.
Object Recognition and Corpus Guided Action Predic-
tion The input to the object recognition module is an RGB
image patch around the target object. We resize each patch to
32×32×3pixels, and we subtract the global mean obtained
from the training data.
Similar to the grasping type recognition module, we also
used a seven layer CNN. The network structure is the same
as before, except that the final perception layer has 48 re-
gression outputs. Our system considers 48 object classes,
and we denote this candidate object list as Oin the rest of
the paper. Table 2 lists the object classes.
apple, blender, bowl, bread, brocolli, brush, butter, carrot,
chicken, chocolate, corn, creamcheese, croutons, cucumber,
cup, doughnut, egg, fish, flour, fork, hen, jelly, knife, lemon,
lettuce, meat, milk, mustard, oil, onion, pan, peanutbutter,
pepper, pitcher, plate, pot, salmon, salt, spatula, spoon,
spreader, steak, sugar, tomato, tongs, turkey, whisk, yogurt.
Table 2: The list of the objects considered in our system.
For each testing video with Mframes, we pass the tar-
get object patches frame by frame, and get an output of size
48×M. We sum it up along the temporal dimension and then
normalize the output. We classify two objects in the image:
(Object1) and (Object2). At the end of classification, the ob-
ject recognition system outputs two belief distributions of
size 48 ×1:PObject1and PO bject2.
We also need the ‘Action’ that was performed. Due to the
large variations in the video, the visual recognition of actions
is difficult. Our system bypasses this problem by using a
trained language model. The model predicts the most likely
verb (Action) associated with the objects (Object1, Object2).
In order to do prediction, we need a set of candidate actions
V. Here, we consider the top 10 most common actions in
cooking scenarios. They are (Cut, Pour, Transfer, Spread,
Grip, Stir, Sprinkle, Chop, Peel, Mix). The same technique,
used here, was used before on a larger set of candidate ac-
tions (Yang et al. 2011).
We compute from the Gigaword corpus (Graff 2003) the
probability of a verb occurring, given the detected nouns,
P(Action|Object1, Object2). We do this by computing the
log-likelihood ratio (Dunning 1993) of trigrams (Object1,
Action, Object2), computed from the sentence in the English
Gigaword corpus (Graff 2003). This is done by extracting
only the words in the corpus that are defined in Oand V(in-
cluding their synonyms). This way we obtain a reduced cor-
pus sequence from which we obtain our target trigrams. The
log-likelihood ratios computed for all possible trigrams are
then normalized to obtain P(Action|Object1, Object2).
For each testing video, we can compute a belief distribution
over the candidate action set Vof size 10 ×1as :
PAction =X
Object1∈O
X
Object2∈O
P(Action|Object1, Object2)
×PObject1×PObj ect2.(1)
From Recognitions to Action Trees
The output of our visual system are belief distributions of
the object categories, grasping types, and actions. However,
they are not sufficient for executing actions. The robot also
needs to understand the hierarchical and recursive structure
of the action. We argue that grammar trees, similar to those
used in linguistics analysis, are a good representation cap-
turing the structure of actions. Therefore we integrate our
visual system with a manipulation action grammar based
parsing module (Yang et al. 2014). Since the output of our
visual system is probabilistic, we extend the grammar to a
probabilistic one and apply the Viterbi probabilistic parser
to select the parse tree with the highest likelihood among
the possible candidates.
Manipulation Action Grammar We made two exten-
sions from the original manipulation grammar (Yang et al.
2014): (i) Since grasping is conceptually different from other
actions, and our system employs a CNN based recognition
module to extract the model grasping type, we assign an ad-
ditional nonterminal symbol Gto represent the grasp. (ii) To
accommodate the probabilistic output from the processing
of unconstrained videos, we extend the manipulation action
grammar into a probabilistic one.
The design of this grammar is motivated by three obser-
vations: (i) Hands are the main driving force in manipula-
tion actions, so a specialized nonterminal symbol His used
for their representation; (ii) an action (A) or a grasping (G)
can be applied to an object (O) directly or to a hand phrase
(H P ), which in turn contains an object (O), as encoded in
Rule (1), which builds up an action phrase (AP ); (iii) an ac-
tion phrase (AP ) can be combined either with the hand (H)
or a hand phrase, as encoded in rule (2), which recursively
builds up the hand phrase. The rules discussed in Table 3
form the syntactic rules of the grammar.
To make the grammar probabilistic, we first treat
each sub-rule in rules (1) and (2) equally, and assign
equal probability to each sub-rule. With regard to the
hand Hin rule (3), we only consider a robot with
two effectors (arms), and assign equal probability to
‘LeftHand’ and ‘RightHand’. For the terminal rules
(4-8), we assign the normalized belief distributions
(PObject1, PO bject2, PGraspT y pe1, PGraspT ype2,PAction)
obtained from the visual processes to each candidate object,
grasping type and action.
AP →G1O1|G2O2|A O2|A HP 0.25 (1)
HP →H AP |HP AP 0.5 (2)
H→‘LeftHand0|‘RightH and00.5 (3)
G1→‘GraspT ype10PGraspT y pe1(4)
G2→‘GraspT ype20PGraspT y pe2(5)
O1→‘Object10PO bject1(6)
O2→‘Object20PO bject2(7)
A→‘Action0PAction (8)
Table 3: A Probabilistic Extension of Manipulation Action
Context-Free Grammar.
Parsing and tree generation We use a bottom-up vari-
ation of the probabilistic context-free grammar parser that
uses dynamic programming (best-known as Viterbi parser
(Church 1988)) to find the most likely parse for an input vi-
sual sentence. The Viterbi parser parses the visual sentence
by filling in the most likely constituent table, and the parser
uses the grammar introduced in Table 3. For each testing
video, our system outputs the most likely parse tree of the
specific manipulation action. By reversely parsing the tree
structure, the robot could derive an action plan for execu-
tion. Figure 3 shows sample output trees, and Table 4 shows
the final control commands generated by reverse parsing.
Experiments
The theoretical framework we have presented suggests two
hypotheses that deserve empirical tests: (a) the CNN based
object recognition module and the grasping type recognition
module can robustly recognize input frame patches from
unconstrained videos into correct class labels; (b) the inte-
grated system using the Viterbi parser with the probabilistic
extension of the manipulation action grammar can generate
a sequence of execution commands robustly.
To test the two hypotheses empirically, we need to de-
fine a set of performance variables and how they relate to
our predicted results. The first hypothesis relates to visual
recognition, and we can empirically test it by measuring the
precision and recall metrics by comparing the detected ob-
ject and grasping type labels with the ground truth ones. The
second hypothesis relates to execution command generation,
and we can also empirically test it by comparing the gen-
erated command predicates with the ground truth ones on
testing videos. To validate our system, we conducted experi-
ments on an extended version of a publicly available uncon-
strained cooking video dataset (YouCook) (Das et al. 2013).
Dataset and experimental settings
Cooking is an activity, requiring a variety of manipulation
actions, that future service robots most likely need to learn.
We conducted our experiments on a publicly available cook-
ing video dataset collected from the WWW and fully la-
beled, called the Youtube cooking dataset (YouCook) (Das
et al. 2013). The data was prepared from 88 open-source
Youtube cooking videos with unconstrained third-person
view. Frame-by-frame object annotations are provided for
49 out of the 88 videos. These features make it a good em-
pirical testing bed for our hypotheses.
We conducted our experiments using the following proto-
cols: (1) 12 video clips, which contain one typical kitchen
action each, are reserved for testing; (2) all other video
frames are used for training; (3) we randomly reserve 10%
of the training data as validation set for training the CNNs.
For training the grasping type, we extended the dataset by
annotating image patches containing hands in the training
videos. The image patches were converted to gray-scale and
then resized to 32×32 pixels. The training set contains 1525
image patches and was labeled with the six grasping types.
We used a GPU based CNN implementation (Jia 2013) to
train the neural network, following the structures described.
For training the object recognition CNN, we first ex-
tracted annotated image patches from the labeled training
videos, and then resized them to 32 ×32 ×3. We used the
same GPU based CNN implementation to train the neural
network, following the structures described above.
For localizing hands on the testing data, we first applied
the hand detector from (Mittal, Zisserman, and Torr 2011)
and picked the top two hand patch proposals (left hand and
right hand, if present). For objects, we trained general object
detectors from labeled training data using techniques from
(Cheng et al. 2014). Furthermore we associated candidate
object patches with the left or right hand, respectively de-
pending on which had the smaller Euclidean distance.
Grasping Type and Object Recognition
On the reserved 10% validation data, the grasping type
recognition module achieved an average precision of 77%
and an average recall of 76%. On the reserved 10% valida-
tion data, the object recognition module achieved an average
precision of 93%, and an average recall of 93%. Figure 2
shows the confusion matrices for grasping type and object
recognition, respectively. From the figure we can see the ro-
bustness of the recognition.
The performance of the object and grasping type recog-
nition modules is also reflected in the commands that our
system generated from the testing videos. We observed an
overall recognition accuracy of 79% on objects, of 91% on
grasping types and of 83% on predicted actions (see Table
4). It is worth mentioning that in the generated commands
the performance in the recognition of object drops, because
some of the objects in the testing sequences do not have
Figure 2: Confusion matrices. Left: grasping type; Right: ob-
ject.
training data, such as “Tofu”. The performance in the clas-
sification of grasping type goes up, because we sum up the
grasping types belief distributions over the frames, which
helps to smooth out wrong labels. The performance metrics
reported here empirically support our hypothesis (a).
Visual Sentence Parsing and Commands
Generation for Robots
Following the probabilistic action grammar from Table 3, we
built upon the implementation of the Viterbi parser from the
Natural Language Processing Kit (Bird, Klein, and Loper
2009) to generate the single most likely parse tree from the
probabilistic visual sentence input. Figure 3 shows the sam-
ple visual processing outputs and final parse trees obtained
using our integrated system. Table 4 lists the commands
generated by our system on the reserved 12 testing videos,
shown together with the ground truth commands. The over-
all percentage of correct commands is 68%. Note, that we
considered a command predicate wrong, if any of the ob-
ject, grasping type or action was recognized incorrectly. The
performance metrics reported here, empirically support our
hypothesis (b).
Discussion
The performance metrics reported in the experiment sec-
tion empirically support our hypotheses that: (1) our system
is able to robustly extract visual sentences with high accu-
racy; (2) our system can learn atomic action commands with
few errors compared to the ground-truth commands. We be-
lieve this preliminary integrated system raises hope towards
a fully intelligent robot for manipulation tasks that can auto-
matically enrich its own knowledge resource by “watching”
recordings from the World Wide Web.
Conclusion and Future Work
In this paper we presented an approach to learn manipulation
action plans from unconstrained videos for cognitive robots.
Two convolutional neural network based recognition mod-
ules (for grasping type and objects respectively), as well as
a language model for action prediction, compose the lower
level of the approach. The probabilistic manipulation action
grammar based Viterbi parsing module is at the higher level,
and its goal is to generate atomic commands in predicate
form. We conducted experiments on a cooking dataset which
consists of unconstrained demonstration videos. From the
HP
AP
HP
AP
O1
Corn
A1
Precision-
Small-
Grasp
H
RightHand
A3
Spread
HP
AP
O1
Brush
A1
Power-
Small-
Grasp
H
LeftHand
HP
AP
O2
Steak
A3
Grip
HP
AP
O1
Tongs
A1
Power-
Small-
Grasp
H
LeftHand
HP
AP
HP
AP
O1
Lemon
A1
Precision-
Small-
Grasp
H
RightHand
A3
Cut
HP
AP
O1
Knife
A1
Power-
Small-
Grasp
H
LeftHand
HP
AP
HP
AP
O1
Bowl
A1
Precision-
Small-
Grasp
H
RightHand
A3
Cut
HP
AP
O1
Knife
A1
Power-
Small-
Grasp
H
LeftHand
Figure 3: Upper row: input unconstrained video frames;
Lower left: color coded (see lengend at the bottom) visual
recognition output frame by frame along timeline; Lower
right: the most likely parse tree generated for each clip.
performance on this challenging dataset, we can conclude
that our system is able to recognize and generate action com-
mands robustly.
We believe that the grasp type is an essential component
for fine grain manipulation action analysis. In future work
we will (1) further extend the list of grasping types to have
a finer categorization; (2) investigate the possibility of using
the grasp type as an additional feature for action recognition;
(3) automatically segment a long demonstration video into
action clips based on the change of grasp type.
Another line of future work lies in the higher level of
Snapshot Ground Truth Commands Learned Commands
Grasp PoS(LH, Knife)
Grasp PrS(RH, Tofu)
Action Cut(Knife, Tofu)
Grasp PoS(LH, Knife)
Grasp PrS(RH, Bowl)
Action Cut(Knife, Bowl)
Grasp PoS(LH, Blender)
Grasp PrL(RH, Bowl)
Action Blend(Blender, Bowl)
Grasp PoS(LH, Bowl)
Grasp PoL(RH, Bowl)
Action Pour(Bowl, Bowl)
Grasp PoS(LH, Tongs)
Action Grip(Tongs, Chicken)
Grasp PoS(LH, Chicken)
Action Cut(Chicken, Chicken)
Grasp PoS(LH, Brush)
Grasp PrS(RH, Corn)
Action Spread(Brush, Corn)
Grasp PoS(LH, Brush)
Grasp PrS(RH, Corn)
Action Spread(Brush, Corn)
Grasp PoS(LH, Tongs)
Action Grip(Tongs, Steak)
Grasp PoS(LH, Tongs)
Action Grip(Tongs, Steak)
Grasp PoS(LH, Spreader)
Grasp PrL(RH, Bread)
Action Spread(Spreader, Bread)
Grasp PoS(LH, Spreader)
Grasp PrL(RH, Bowl)
Action Spread(Spreader, Bowl)
Grasp PoL(LH, Mustard)
Grasp PrS(RH, Bread)
Action Spread(Mustard, Bread)
Grasp PoL(LH, Mustard)
Grasp PrS(RH, Bread)
Action Spread(Mustard, Bread)
Grasp PoS(LH, Spatula)
Grasp PrS(RH, Bowl)
Action Stir(Spatula, Bowl)
Grasp PoS(LH, Spatula)
Grasp PrS(RH, Bowl)
Action Stir(Spatula, Bowl)
Grasp PoL(LH, Pepper)
Grasp PoL(RH, Pepper)
Action Sprinkle(Pepper, Bowl)
Grasp PoL(LH, Pepper)
Grasp PoL(RH, Pepper)
Action Sprinkle(Pepper, Pepper)
Grasp PoS(LH, Knife)
Grasp PrS(RH, Lemon)
Action Cut(Knife, Lemon)
Grasp PoS(LH, Knife)
Grasp PrS(RH, Lemon)
Action Cut(Knife, Lemon)
Grasp PoS(LH, Knife)
Grasp PrS(RH, Broccoli)
Action Cut(Knife, Broccoli)
Grasp PoS(LH, Knife)
Grasp PoL(RH, Broccoli)
Action Cut(Knife, Broccoli)
Grasp PoS(LH, Whisk)
Grasp PrL(RH, Bowl)
Action Stir(Whisk, Bowl)
Grasp PoS(LH, Whisk)
Grasp PrL(RH, Bowl)
Action Stir(Whisk, Bowl)
Overall
Recognition
Accuracy
Object: 79%
Grasping type: 91%
Action: 83%
Overall percentage of
correct commands: 68%
Table 4: LH:LeftHand; RH: RightHand; PoS: Power-Small;
PoL: Power-Large; PoP: Power-Spherical; PrS: Precision-
Small; PrL: Precision-Large. Incorrect entities learned are
marked in red.
the system. The probabilistic manipulation action grammar
used in this work is still a syntax grammar. We are cur-
rently investigating the possibility of coupling manipulation
action grammar rules with semantic rules using lambda ex-
pressions, through the formalism of combinatory categorial
grammar developed by (Steedman 2002).
Acknowledgements This research was funded in part by
the support of the European Union under the Cognitive Sys-
tems program (project POETICON++), the National Sci-
ence Foundation under INSPIRE grant SMA 1248056, and
support by DARPA through U.S. Army grant W911NF-14-
1-0384 under the Project: Shared Perception, Cognition and
Reasoning for Autonomy. NICTA is funded by the Aus-
tralian Government as represented by the Department of
Broadband, Communications and the Digital Economy and
the Australian Research Council through the ICT Centre of
Excellence program.
References
Aksoy, E.; Abramov, A.; D¨
orr, J.; Ning, K.; Dellen, B.; and
W¨
org¨
otter, F. 2011. Learning the semantics of object-action re-
lations by observation. The International Journal of Robotics Re-
search 30(10):1229–1249.
Argall, B. D.; Chernova, S.; Veloso, M.; and Browning, B. 2009.
A survey of robot learning from demonstration. Robotics and Au-
tonomous Systems 57(5):469–483.
Bengio, Y.; Courville, A.; and Vincent, P. 2013. Representation
learning: A review and new perspectives. Pattern Analysis and
Machine Intelligence, IEEE Transactions on 35(8):1798–1828.
Bird, S.; Klein, E.; and Loper, E. 2009. Natural language process-
ing with Python. ” O’Reilly Media, Inc.”.
Cheng, M.-M.; Zhang, Z.; Lin, W.-Y.; and Torr, P. H. S. 2014.
BING: Binarized normed gradients for objectness estimation at
300fps. In IEEE CVPR.
Chomsky, N. 1993. Lectures on government and binding: The Pisa
lectures. Berlin: Walter de Gruyter.
Church, K. W. 1988. A stochastic parts program and noun phrase
parser for unrestricted text. In Proceedings of the second confer-
ence on Applied natural language processing, 136–143. Associa-
tion for Computational Linguistics.
Ciresan, D. C.; Meier, U.; and Schmidhuber, J. 2012. Multi-column
deep neural networks for image classification. In CVPR 2012.
Dalal, N., and Triggs, B. 2005. Histograms of oriented gradients
for human detection. In Computer Vision and Pattern Recogni-
tion, 2005. CVPR 2005. IEEE Computer Society Conference on,
volume 1, 886–893. IEEE.
Das, P.; Xu, C.; Doell, R. F.; and Corso, J. J. 2013. A thousand
frames in just a few words: Lingual description of videos through
latent topics and sparse object stitching. In Proceedings of IEEE
Conference on Computer Vision and Pattern Recognition.
Dunning, T. 1993. Accurate methods for the statistics of surprise
and coincidence. Computational Linguistics 19(1):61–74.
Feix, T.; Romero, J.; Ek, C. H.; Schmiedmayer, H.; and Kragic,
D. 2013. A Metric for Comparing the Anthropomorphic Motion
Capability of Artificial Hands. Robotics, IEEE Transactions on
29(1):82–93.
Graff, D. 2003. English gigaword. In Linguistic Data Consortium,
Philadelphia, PA.
Guerra-Filho, G.; Ferm¨
uller, C.; and Aloimonos, Y. 2005. Dis-
covering a language for human activity. In Proceedings of the
AAAI 2005 Fall Symposium on Anticipatory Cognitive Embodied
Systems. Washington, DC: AAAI.
Guha, A.; Yang, Y.; Ferm¨
uller, C.; and Aloimonos, Y. 2013. Min-
imalist plans for interpreting manipulation actions. In Proceed-
ings of the 2013 International Conference on Intelligent Robots
and Systems, 5908–5914. Tokyo: IEEE.
Jeannerod, M. 1984. The timing of natural prehension movements.
Journal of motor behavior 16(3):235–254.
Jia, Y. 2013. Caffe: An open source convolutional architecture for
fast feature embedding. http://caffe.berkeleyvision.
org/.
Krizhevsky, A.; Sutskever, I.; and Hinton, G. 2013. Imagenet clas-
sification with deep convolutional neural networks. In NIPS 2012.
LeCun, Y., and Bengio, Y. 1998. The handbook of brain theory
and neural networks. Cambridge, MA, USA: MIT Press. chapter
Convolutional networks for images, speech, and time series, 255–
258.
Lee, K.; Su, Y.; Kim, T.-K.; and Demiris, Y. 2013. A syntac-
tic approach to robot imitation learning using probabilistic activity
grammars. Robotics and Autonomous Systems 61(12):1323–1334.
Lenz, I.; Lee, H.; and Saxena, A. 2014. Deep learning for detect-
ing robotic grasps. International Journal of Robotics Research to
appear.
Li, Y.; Ferm¨
uller, C.; Aloimonos, Y.; and Ji, H. 2010. Learning
shift-invariant sparse representation of actions. In Proceedings of
the 2009 IEEE Conference on Computer Vision and Pattern Recog-
nition, 2630–2637. San Francisco, CA: IEEE.
Lowe, D. G. 2004. Distinctive image features from scale-invariant
keypoints. International journal of computer vision 60(2):91–110.
Mittal, A.; Zisserman, A.; and Torr, P. H. 2011. Hand detection
using multiple proposals. In BMVC, 1–11. Citeseer.
Oikonomidis, I.; Kyriazis, N.; and Argyros, A. 2011. Efficient
model-based 3D tracking of hand articulations using Kinect. In
Proceedings of the 2011 British Machine Vision Conference, 1–11.
Dundee, UK: BMVA.
Pastra, K., and Aloimonos, Y. 2012. The minimalist grammar of
action. Philosophical Transactions of the Royal Society: Biological
Sciences 367(1585):103–117.
Saxena, A.; Driemeyer, J.; and Ng, A. Y. 2008. Robotic grasping of
novel objects using vision. The International Journal of Robotics
Research 27(2):157–173.
Shimoga, K. B. 1996. Robot grasp synthesis algorithms: A survey.
The International Journal of Robotics Research 15(3):230–266.
Steedman, M. 2002. Plans, affordances, and combinatory gram-
mar. Linguistics and Philosophy 25(5-6):723–753.
Summers-Stay, D.; Teo, C.; Yang, Y.; Ferm¨
uller, C.; and Aloi-
monos, Y. 2013. Using a minimal action grammar for activity un-
derstanding in the real world. In Proceedings of the 2013 IEEE/RSJ
International Conference on Intelligent Robots and Systems, 4104–
4111. Vilamoura, Portugal: IEEE.
Tenorth, M.; Ziegltrum, J.; and Beetz, M. 2013. Automated align-
ment of specifications of everyday manipulation tasks. In IROS.
IEEE.
Yang, Y.; Teo, C. L.; Daum´
e III, H.; and Aloimonos, Y. 2011.
Corpus-guided sentence generation of natural images. In Proceed-
ings of the Conference on Empirical Methods in Natural Language
Processing, 444–454. Association for Computational Linguistics.
Yang, Y.; Guha, A.; Fermuller, C.; and Aloimonos, Y. 2014. A
cognitive system for understanding human manipulation actions.
Advances in Cognitive Sysytems 3:67–86.
