Content uploaded by Johan Baeten
Author content
All content in this area was uploaded by Johan Baeten on Aug 18, 2014
Content may be subject to copyright.
Autonomous Fruit Picking Machine:
A Robotic Apple Harvester
Johan Baeten1, Kevin Donn´e2, Sven Boedrij2, Wim Beckers2and Eric
Claesen1,2
1Fac. of Industrial Sciences and Technology, Katholieke Hogeschool Limburg,
Belgium Johan.Baeten@iwt.khlim.be
2ACRO: Automation Centre for Research and Education, Katholieke Hogeschool
Limburg, Belgium Eric.Claesen@iwt.khlim.be
Summary. This paper describes the construction and functionality of an Au-
tonomous Fruit Picking Machine (AFPM) for robotic apple harvesting. The key
element for the success of the AFPM is the integrated approach which combines
state of the art industrial components with the newly designed flexible gripper. The
gripper consist of a silicone funnel with a camera mounted inside. The proposed
concepts guarantee adequate control of the autonomous fruit harvesting operation
globally and of the fruit picking cycle particularly. Extensive experiments in the field
validate the functionality of the AFPM.
1 Introduction
The use of robots is no longer strictly limited to industrial environments. Also
for outdoor activities, robotic systems are increasingly combined with new
technologies to automate labour intensive work, such as e.g. apple harvest-
ing [2, 10]. This paper describes the feasibility study for and the development
of an Autonomous Fruit Picking Machine (AFPM)3.
There are two main approaches in robotic apple harvesting being bulk [9,
10] or apple by apple harvesting [3, 11].
Peterson et al. [9] developed a mechanical bulk robotic harvester for apples
grown on narrow, inclined trellises. This type of bulk harvesting requires, in
addition to the canopy-like growth habit, uniform fruit ripeness at harvest,
firm fruit, resistant to damage, and short/stiff limbs [10].
The use of an apple by apple picking system, although inherently slower,
does not suffer from any of the above restrictions. Moreover, only apples of
satisfactory size and maturity are selected for harvesting and can be sorted
out immediately. An apple by apple picking system does, however, require
3Funded by IWT-Vlaanderen under TETRA 40196
inria-00194739, version 1 - 7 Dec 2007
Author manuscript, published in "6th International Conference on Field and Service Robotics - FSR 2007, Chamonix : France
(2007)"
2 J. Baeten, K. Donn´e, S. Boedrij, W. Beckers, E. Claesen
an adequate fruit gripper. The gripper is the key element in the success of
automated apple by apple harvesting. A good gripper ought to preserve the
quality of the apple and should not damage the tree (nor the apple) during
the picking cycle. Setiawan et al. [11] propose a low cost gripper with flexible
inflatable parts. Our gripper consists of a flexible silicone funnel and uses
suction to pick the apple.
An indispensable part of any autonomous apple harvesting machine is the
vision system used to locate the apple [5, 10, 11, 12, 13]. Bulanon et al. [3] use
a (RT) machine vision system to recognize the location of the fruit centre and
the abscission layer of the peduncle. In contrast to most researchers, in our
approach the camera is positioned in the centre of the gripper. This simplifies
the calibration of the set-up and ensures adequate control.
According to the classification given by Hutchinson et al. [4], our image
based control is closer to a look and move strategy than to visual servoing.
There is, however, margin for improvement.
Despite previous developments towards a harvesting platform navigating
autonomously between orchard rows [1], we chose not (yet) to implement an
automated navigation, such as e.g. in [6] or [7], for reasons of both develop-
ment time as well as safety approval.
The aim of this project was to prove the feasibility and demonstrate the
functionality of an Autonomous Fruit Picking Machine (AFPM), by using
existing state of the art (industrial) components. This resulted in the pro-
totype AFPM, described in the following sections. First, section 2 describes
the overall construction of the AFPM. Section 3 presents our new, patented
gripper designed specifically for the apple harvesting task. Section 4 goes into
the control details for one picking cycle. Finally, sections 5 and 6 summarize
the results of the field experiments and conclude this paper.
2 Overall construction of AFPM
The AFPM is built on a platform mounted behind an agriculture tractor. Fig-
ure 1 shows the first version of the AFPM. Figure 2 illustrates schematically
the functional layout and data flow. In order to reduce the development period
of the AFPM, although overkill, an industrial robot (Panasonic VR006L) is
chosen as manipulator. The AFPM further consists of a tractor-driven gen-
erator for power supply, a (2D) horizontal stabilization unit, a 7th external
vertical axis to enlarge the operation range, a safety scanning device, a central
control unit, a touch panel PC with Human Machine Interface, a canopy and
all around curtain to even out light conditions and, finally, a fruit gripper
designed specifically for this task, with a camera mounted in the centre of
the gripper. The flexible gripper (described in section 3) guarantees a firm
grip without damaging the fruit and serves in fact as the mouth of a vacuum
cleaner.
inria-00194739, version 1 - 7 Dec 2007
Autonomous Fruit Picking Machine: A Robotic Apple Harvester 3
High frequency light source
Soft gripper
with camera inside
6 DOF industrial robot
Horizontal
stabilization
External vertical axis Power generation
and transfer
Fig. 1. Construction of the AFPM
uEye
camera
UI2230-C
Central Control Unit
PROFIBUS
USB 2.0
PC
PROFIBUS
PROFIBUS
PLC
VR006L
Robot
Image processing
and HMI
Panasonic
Robot
Horizontal
stabilization Controller
External
vertical
axis
Scanner
Fig. 2. Functional scheme of the AFPM
The 2D horizontal stabilization consists of two hydraulic feet and one turn
over cylinder, configured as a 3-point suspension. Controlled by two level sen-
sors, the system ensures a stable positioning of the robot platform during the
picking cycle.
inria-00194739, version 1 - 7 Dec 2007
4 J. Baeten, K. Donn´e, S. Boedrij, W. Beckers, E. Claesen
3 New flexible gripper
To pick the apple with as little energy as possible, a completely new fruit grip-
per had to be designed. Its patented design has two main functions: catching
the apple and enclosing the camera. Figure 3 gives some examples.
The gripper assumes the shape of the apple and encloses it firmly. After
several prototypes, the optimal funnel shape with respect to edge thickness,
funnel angle and size with a trade off between flexibility and firmness was
designed and tested. The current version has a maximum diameter of 10.5
cm. The gripping function is activated by vacuum suction. It is e.g. possible
the pick up an apple by just pushing the gripper onto the apple and closing
the vacuum port with your finger. Experiments show that even an elongated
exposure of the apple to the (rather small) vacuum levels4used does not
damage the apple in any way.
Fig. 3. Left: two examples of silicone gripper; right: gripper mounted on robot with
camera inside
Placing the camera in the centre of the gripper offers numerous advantages.
First of all, the gripper is always in line with the camera and thus with the
image, which simplifies the (necessary) coordinate transformation from image
to robot. Furthermore, the position of the camera is fully controllable. The
camera can always point its optical axis to the apple (see section 4 and sim-
4The magnitude of underpressure ranges from 150 mbar to 230 mbar with a flow
rate of 200 m3/h.
inria-00194739, version 1 - 7 Dec 2007
Autonomous Fruit Picking Machine: A Robotic Apple Harvester 5
z
cam
z
cam
x
cam
x
cam
y
cam
y
cam
x
p
x
p
y
p
y
p
f
zcam
zcam
TCP
apple
object
plane
image
plane
Rx
Ry
3d-view
ypmp
f
y
front-view
xpmp
f
x
top-view
(-)
Rx
Ry
Fig. 4. 3D, top and front view of the camera model defining the camera frame and
illustrating the computation of the rotation angles θxand θy, needed to centre the
apple in the image
ulation figure 7). This reduces image distortion and eliminates the necessity
for thorough calibration.
4 Approach for fruit picking cycle
The autonomous harvesting operation is hierarchically structured in three
levels. Once the AFPM is stationed in front of the tree with active stabilization
(first level), it scans the tree from 40 look-out positions or sectors (second
level). For each sector, all ripe apples are listed and picked one by one in a
looped task (third level). A picking operation consists of following steps:
1. The position of the apple in the image, possibly after declustering, is
determined. Only ripe apples with qualified size are selected.
2. The camera rotates around x- and y-axes, by θxand θyrespectively, in
order to point the optical axis straight to the apple. Positioning the cam-
era by only rotating the wrist results in a small and therefore fast robot
movement. Figure 4 defines the used set-up. The rotation angles yield:
θx=−arctan(ypµp/f) (1)
for the rotation around the x-axis and
θy= arctan(xpµp/f) (2)
inria-00194739, version 1 - 7 Dec 2007
6 J. Baeten, K. Donn´e, S. Boedrij, W. Beckers, E. Claesen
for the (simultaneous) rotation around the y-axis, with fthe focal length
[mm], µpthe pixel-size [mm/pix] and xp, ypthe measured centre of the
apple in the image plane [pix]. As equations 1 and 2 show, the rotation
angles do not depend on the distance to the apple zcam . Only the focal
length fneeds to be calibrated.
Even if the focal length is not exactly known, the final offset of the centre
of the gripper with respect to the centre of the apple will lie within the
margin of the gripper: e.g. a (rather large) error of 10% on the focal length
causes an error of 1.5oon θx(or θy) resulting in an offset of 1.9 cm for an
apple at 1 m distance. Due to the funnel-like design of our gripper, this
offset will not cause a faulty picking cycle.
x
p2
f
TCP
(1)
f
TCP
(2)
x
p1
ZZ
D
Fig. 5. Illustration of the remaining distance calculation
Robot Vision
PLC, Control
time
x,y
DZ
trigger
trigger
trigger
trigger
start
detection
of
vacuum
start position
for sector
1a
global image
ðall apples
in sector
1b
1st apple ð
position
and size
2a
rotate
TCP/camera
2b
Z
approach
3a
new meas. :
position
and size
3b
distance
to apple
4a
final
approach
4b
detach
apple
5
qq
Fig. 6. Schematic overview of the control flow between vision system, PLC and
robot during one apple-picking cycle
3. While approaching the apple, several images are processed to calculate
by triangulation the remaining distance to the apple. Given the distance
Z, travelled between two images, and xp1,xp2the measured radii of the
(corresponding) apple from those two images, the remaining distance ∆Z
inria-00194739, version 1 - 7 Dec 2007
Autonomous Fruit Picking Machine: A Robotic Apple Harvester 7
to the apple yields
∆Z =xp1Z
xp2−xp1
.(3)
Figure 5 illustrates equation 3. Since the apple remains centred in the
image, the correlation of a given apple in subsequent images is trivial.
4. Once the apple is within a given range of the gripper, the vacuum suction
is activated. The detection of (a small level of ) vacuum triggers the next
step.
5. The apple is picked by rotating it and tilting it softly and is then put
aside. The actual wrist/robot movement differs from section to section. It
is programmed and optimized in advance with a minimum tree intrusion
in mind. A standard apple transporting and collecting system still needs
to be added.
Figure 6 gives a schematic overview of these 5 steps. The indicated num-
bers correspond to the step list for the picking cycle. The image processing
is conducted on an industrial pentium IV 2 GHz PC with 1 GB RAM under
Windows. The actual image processing5is programmed with Halcon soft-
ware [8] and takes about 0.6 seconds (for step 1). Note that the complete
image processing is only needed at the start of a picking cycle. Subsequent
image processing should be limited to a small portion of the image and scaled
down to less functional steps.
5 Field experiments
Field experiments conducted over a period of several weeks in a Jonagold
orchard show that the overall performance of the AFPM is more than satis-
factory. The stabilization experiences no difficulties with possible rapid robot
motion. Thanks to the roof and all-around cover, as shown in figure 7, the
AFPM works even under poor weather conditions, although also here im-
provements are still possible.
The experiments demonstrate that about 80% of the apples (with diam-
eter range from 6 cm to 11 cm) are detected and harvested. However, stem
pulls of 30% are still too high. However, on should take into account the fact
that the apples were treated to prolong the harvest period, causing a more
firm connection of the stems to the limbs. Nevertheless, fine tuning step 5 in
section 4 may lower this fault margin.
Apart from stem pulls, the apples show no bruises or damage at all. Apples
not harvested are either not detectable by the vision system or not reachable
for the robot manipulator. This problem can be partially solved by trimming
the orchard rows into smaller hedges.
5A full description of the image processing, however, falls outside the scope of this
article but will be the subject of future publications.
inria-00194739, version 1 - 7 Dec 2007
8 J. Baeten, K. Donn´e, S. Boedrij, W. Beckers, E. Claesen
Fig. 7. Left: simulation model; right: field experiments
In the current setup the overall cycle time to pick one apple is 8 to 10 sec.
Eliminating the current communication bottleneck between the vision system
and the robot through the central controller ((trigger) signals in figure 6) will
improve the overall picking cycle time.
Extra care should be taken in selecting the first apple in a cluster in order
not to push off other apples. Additional vision information about nearby sharp
limbs could also help to protect the gripper itself from damage.
6 Conclusion
The first results of the AFPM are very promising. All the necessary com-
ponents are fitted together and operate as planned, hereby proving the
feasibility and functionality of the AFPM. There is, however, still margin
for improvement. The bottleneck in communication lies with the connec-
tion/communication between the vision-PC and the central control unit. Fu-
ture work will focus on improving the bandwidth of this connection and on
optimizing the image processing. The aim is to reduce the picking cycle period
from an average of 9 to about 5 seconds (or less). In that case, the produc-
tivity of the AFPM will be close to the work load of about 6 workers, which
makes the machine economically viable.
Special attention will go to improving the actual apple detach movement
by taking into account the relative pose of the apple with respect to the
branch. This should lower the stem pull percentage. Further research is also
planned to automate the navigation trough the orchard and to investigate the
harvesting of pears.
inria-00194739, version 1 - 7 Dec 2007
Autonomous Fruit Picking Machine: A Robotic Apple Harvester 9
References
1. Appeltants W and Ilsbroux W (1995) Automatische sturing van een plukwagen
(Autonomous guidance for a harvester). MA Thesis, Katholieke Hogeschool
Limburg, Belgium
2. Bulanon DM, Kataoka T, Okamoto H, Hata S (2005) Feedback control of ma-
nipulator using machine vision for robotic apple harvesting. In Proceedings of
ASAE, Paper No. 053114, Tampa, USA
3. Bulanon DM, Kataoka T, Ukamoto H, Hata S (2004) Development of a real-
time machine vision system for the apple harvesting robot. In SICE Annual
Conference in Sapporo, pages 595–598, Hokkaido Inst. of Techn., Japan
4. Hutchinson S, Hager G, Corke P (1996) A tutorial on visual servo control. IEEE
Trans. on Robotics and Automation, 12(5):651–670
5. Jimenez A, Ceres R, Pons J (2000) A survey of computer vision methods for
locating fruits on trees. Transactions of the ASAE, 43(6):1911–1920
6. Katypitiya J, Eaton R, Cole A, Meyer C, Rodnay G (2005) Automation of
an agricultural tractor for fruit picking. In IEEE Int. Conf. on Robotics and
Automation, pages 3212–3217, Barcelona, Spain
7. Mateus D, Avi˜na G, Devy M (2005) Robot visual navigation in semi-structured
outdoor environments. In IEEE Int. Conf. on Robotics and Automation, pages
4702–4707, Barcelona, Spain
8. MVTec Software GmbH, M¨unchen, Germany (2006) Halcon 7.1, Quick Guide,
2b edition
9. Peterson DL, Bennedsen BS, Anger WC, Wolford SD (1999) A systems
approach to robotic bulk harvesting of apples. Transactions of the ASAE,
42(4):871–876
10. Peterson DL (2005) Harvest mechanization progress and prospects for fresh
market quality deciduous tree fruits. HortTechnology, 15(1):72–75
11. Setiawan AI, Furukawa T, Preston A (2004) A low-cost gripper for an apple
picking robot. In IEEE Int. Conf. on Robotics and Automation, pages 4448–
4453, New Orleans, LA
12. Stajnko D and `
Emelik Z (2005) Modelling of apple fruit growth by application
of image analysis. Agriculturae Conspectus Scientificus, 70(2):59–64
13. Zhao J, Tow J, and Katupitiya J (2005) On-tree fruit recognition using texture
properties and color data. In Int. Conf. on Intelligent Robots and Systems,
pages 3993–3998, Alberta, Canada
inria-00194739, version 1 - 7 Dec 2007
