ArticlePDF Available

Autonomous Fruit Picking Machine: A Robotic Apple Harvester



This paper describes the construction and functionality of an Autonomous Fruit Picking Machine (AFPM) for robotic apple harvesting. The key element for the success of the AFPM is the integrated approach which combines state of the art industrial components with the newly designed °exible gripper. The gripper consist of a silicone funnel with a camera mounted inside. The proposed concepts guarantee adequate control of the autonomous fruit harvesting operation globally and of the fruit picking cycle particularly. Extensive experiments in the ¯eld validate the functionality of the AFPM.
Autonomous Fruit Picking Machine:
A Robotic Apple Harvester
Johan Baeten1, Kevin Donn´e2, Sven Boedrij2, Wim Beckers2and Eric
1Fac. of Industrial Sciences and Technology, Katholieke Hogeschool Limburg,
2ACRO: Automation Centre for Research and Education, Katholieke Hogeschool
Limburg, Belgium
Summary. This paper describes the construction and functionality of an Au-
tonomous Fruit Picking Machine (AFPM) for robotic apple harvesting. The key
element for the success of the AFPM is the integrated approach which combines
state of the art industrial components with the newly designed flexible gripper. The
gripper consist of a silicone funnel with a camera mounted inside. The proposed
concepts guarantee adequate control of the autonomous fruit harvesting operation
globally and of the fruit picking cycle particularly. Extensive experiments in the field
validate the functionality of the AFPM.
1 Introduction
The use of robots is no longer strictly limited to industrial environments. Also
for outdoor activities, robotic systems are increasingly combined with new
technologies to automate labour intensive work, such as e.g. apple harvest-
ing [2, 10]. This paper describes the feasibility study for and the development
of an Autonomous Fruit Picking Machine (AFPM)3.
There are two main approaches in robotic apple harvesting being bulk [9,
10] or apple by apple harvesting [3, 11].
Peterson et al. [9] developed a mechanical bulk robotic harvester for apples
grown on narrow, inclined trellises. This type of bulk harvesting requires, in
addition to the canopy-like growth habit, uniform fruit ripeness at harvest,
firm fruit, resistant to damage, and short/stiff limbs [10].
The use of an apple by apple picking system, although inherently slower,
does not suffer from any of the above restrictions. Moreover, only apples of
satisfactory size and maturity are selected for harvesting and can be sorted
out immediately. An apple by apple picking system does, however, require
3Funded by IWT-Vlaanderen under TETRA 40196
inria-00194739, version 1 - 7 Dec 2007
Author manuscript, published in "6th International Conference on Field and Service Robotics - FSR 2007, Chamonix : France
2 J. Baeten, K. Donn´e, S. Boedrij, W. Beckers, E. Claesen
an adequate fruit gripper. The gripper is the key element in the success of
automated apple by apple harvesting. A good gripper ought to preserve the
quality of the apple and should not damage the tree (nor the apple) during
the picking cycle. Setiawan et al. [11] propose a low cost gripper with flexible
inflatable parts. Our gripper consists of a flexible silicone funnel and uses
suction to pick the apple.
An indispensable part of any autonomous apple harvesting machine is the
vision system used to locate the apple [5, 10, 11, 12, 13]. Bulanon et al. [3] use
a (RT) machine vision system to recognize the location of the fruit centre and
the abscission layer of the peduncle. In contrast to most researchers, in our
approach the camera is positioned in the centre of the gripper. This simplifies
the calibration of the set-up and ensures adequate control.
According to the classification given by Hutchinson et al. [4], our image
based control is closer to a look and move strategy than to visual servoing.
There is, however, margin for improvement.
Despite previous developments towards a harvesting platform navigating
autonomously between orchard rows [1], we chose not (yet) to implement an
automated navigation, such as e.g. in [6] or [7], for reasons of both develop-
ment time as well as safety approval.
The aim of this project was to prove the feasibility and demonstrate the
functionality of an Autonomous Fruit Picking Machine (AFPM), by using
existing state of the art (industrial) components. This resulted in the pro-
totype AFPM, described in the following sections. First, section 2 describes
the overall construction of the AFPM. Section 3 presents our new, patented
gripper designed specifically for the apple harvesting task. Section 4 goes into
the control details for one picking cycle. Finally, sections 5 and 6 summarize
the results of the field experiments and conclude this paper.
2 Overall construction of AFPM
The AFPM is built on a platform mounted behind an agriculture tractor. Fig-
ure 1 shows the first version of the AFPM. Figure 2 illustrates schematically
the functional layout and data flow. In order to reduce the development period
of the AFPM, although overkill, an industrial robot (Panasonic VR006L) is
chosen as manipulator. The AFPM further consists of a tractor-driven gen-
erator for power supply, a (2D) horizontal stabilization unit, a 7th external
vertical axis to enlarge the operation range, a safety scanning device, a central
control unit, a touch panel PC with Human Machine Interface, a canopy and
all around curtain to even out light conditions and, finally, a fruit gripper
designed specifically for this task, with a camera mounted in the centre of
the gripper. The flexible gripper (described in section 3) guarantees a firm
grip without damaging the fruit and serves in fact as the mouth of a vacuum
inria-00194739, version 1 - 7 Dec 2007
Autonomous Fruit Picking Machine: A Robotic Apple Harvester 3
High frequency light source
Soft gripper
with camera inside
6 DOF industrial robot
External vertical axis Power generation
and transfer
Fig. 1. Construction of the AFPM
Central Control Unit
USB 2.0
Image processing
and HMI
stabilization Controller
Fig. 2. Functional scheme of the AFPM
The 2D horizontal stabilization consists of two hydraulic feet and one turn
over cylinder, configured as a 3-point suspension. Controlled by two level sen-
sors, the system ensures a stable positioning of the robot platform during the
picking cycle.
inria-00194739, version 1 - 7 Dec 2007
4 J. Baeten, K. Donn´e, S. Boedrij, W. Beckers, E. Claesen
3 New flexible gripper
To pick the apple with as little energy as possible, a completely new fruit grip-
per had to be designed. Its patented design has two main functions: catching
the apple and enclosing the camera. Figure 3 gives some examples.
The gripper assumes the shape of the apple and encloses it firmly. After
several prototypes, the optimal funnel shape with respect to edge thickness,
funnel angle and size with a trade off between flexibility and firmness was
designed and tested. The current version has a maximum diameter of 10.5
cm. The gripping function is activated by vacuum suction. It is e.g. possible
the pick up an apple by just pushing the gripper onto the apple and closing
the vacuum port with your finger. Experiments show that even an elongated
exposure of the apple to the (rather small) vacuum levels4used does not
damage the apple in any way.
Fig. 3. Left: two examples of silicone gripper; right: gripper mounted on robot with
camera inside
Placing the camera in the centre of the gripper offers numerous advantages.
First of all, the gripper is always in line with the camera and thus with the
image, which simplifies the (necessary) coordinate transformation from image
to robot. Furthermore, the position of the camera is fully controllable. The
camera can always point its optical axis to the apple (see section 4 and sim-
4The magnitude of underpressure ranges from 150 mbar to 230 mbar with a flow
rate of 200 m3/h.
inria-00194739, version 1 - 7 Dec 2007
Autonomous Fruit Picking Machine: A Robotic Apple Harvester 5
Fig. 4. 3D, top and front view of the camera model defining the camera frame and
illustrating the computation of the rotation angles θxand θy, needed to centre the
apple in the image
ulation figure 7). This reduces image distortion and eliminates the necessity
for thorough calibration.
4 Approach for fruit picking cycle
The autonomous harvesting operation is hierarchically structured in three
levels. Once the AFPM is stationed in front of the tree with active stabilization
(first level), it scans the tree from 40 look-out positions or sectors (second
level). For each sector, all ripe apples are listed and picked one by one in a
looped task (third level). A picking operation consists of following steps:
1. The position of the apple in the image, possibly after declustering, is
determined. Only ripe apples with qualified size are selected.
2. The camera rotates around x- and y-axes, by θxand θyrespectively, in
order to point the optical axis straight to the apple. Positioning the cam-
era by only rotating the wrist results in a small and therefore fast robot
movement. Figure 4 defines the used set-up. The rotation angles yield:
θx=arctan(ypµp/f) (1)
for the rotation around the x-axis and
θy= arctan(xpµp/f) (2)
inria-00194739, version 1 - 7 Dec 2007
6 J. Baeten, K. Donn´e, S. Boedrij, W. Beckers, E. Claesen
for the (simultaneous) rotation around the y-axis, with fthe focal length
[mm], µpthe pixel-size [mm/pix] and xp, ypthe measured centre of the
apple in the image plane [pix]. As equations 1 and 2 show, the rotation
angles do not depend on the distance to the apple zcam . Only the focal
length fneeds to be calibrated.
Even if the focal length is not exactly known, the final offset of the centre
of the gripper with respect to the centre of the apple will lie within the
margin of the gripper: e.g. a (rather large) error of 10% on the focal length
causes an error of 1.5oon θx(or θy) resulting in an offset of 1.9 cm for an
apple at 1 m distance. Due to the funnel-like design of our gripper, this
offset will not cause a faulty picking cycle.
Fig. 5. Illustration of the remaining distance calculation
Robot Vision
PLC, Control
start position
for sector
global image
ðall apples
in sector
1st apple ð
and size
new meas. :
and size
to apple
Fig. 6. Schematic overview of the control flow between vision system, PLC and
robot during one apple-picking cycle
3. While approaching the apple, several images are processed to calculate
by triangulation the remaining distance to the apple. Given the distance
Z, travelled between two images, and xp1,xp2the measured radii of the
(corresponding) apple from those two images, the remaining distance ∆Z
inria-00194739, version 1 - 7 Dec 2007
Autonomous Fruit Picking Machine: A Robotic Apple Harvester 7
to the apple yields
∆Z =xp1Z
Figure 5 illustrates equation 3. Since the apple remains centred in the
image, the correlation of a given apple in subsequent images is trivial.
4. Once the apple is within a given range of the gripper, the vacuum suction
is activated. The detection of (a small level of ) vacuum triggers the next
5. The apple is picked by rotating it and tilting it softly and is then put
aside. The actual wrist/robot movement differs from section to section. It
is programmed and optimized in advance with a minimum tree intrusion
in mind. A standard apple transporting and collecting system still needs
to be added.
Figure 6 gives a schematic overview of these 5 steps. The indicated num-
bers correspond to the step list for the picking cycle. The image processing
is conducted on an industrial pentium IV 2 GHz PC with 1 GB RAM under
Windows. The actual image processing5is programmed with Halcon soft-
ware [8] and takes about 0.6 seconds (for step 1). Note that the complete
image processing is only needed at the start of a picking cycle. Subsequent
image processing should be limited to a small portion of the image and scaled
down to less functional steps.
5 Field experiments
Field experiments conducted over a period of several weeks in a Jonagold
orchard show that the overall performance of the AFPM is more than satis-
factory. The stabilization experiences no difficulties with possible rapid robot
motion. Thanks to the roof and all-around cover, as shown in figure 7, the
AFPM works even under poor weather conditions, although also here im-
provements are still possible.
The experiments demonstrate that about 80% of the apples (with diam-
eter range from 6 cm to 11 cm) are detected and harvested. However, stem
pulls of 30% are still too high. However, on should take into account the fact
that the apples were treated to prolong the harvest period, causing a more
firm connection of the stems to the limbs. Nevertheless, fine tuning step 5 in
section 4 may lower this fault margin.
Apart from stem pulls, the apples show no bruises or damage at all. Apples
not harvested are either not detectable by the vision system or not reachable
for the robot manipulator. This problem can be partially solved by trimming
the orchard rows into smaller hedges.
5A full description of the image processing, however, falls outside the scope of this
article but will be the subject of future publications.
inria-00194739, version 1 - 7 Dec 2007
8 J. Baeten, K. Donn´e, S. Boedrij, W. Beckers, E. Claesen
Fig. 7. Left: simulation model; right: field experiments
In the current setup the overall cycle time to pick one apple is 8 to 10 sec.
Eliminating the current communication bottleneck between the vision system
and the robot through the central controller ((trigger) signals in figure 6) will
improve the overall picking cycle time.
Extra care should be taken in selecting the first apple in a cluster in order
not to push off other apples. Additional vision information about nearby sharp
limbs could also help to protect the gripper itself from damage.
6 Conclusion
The first results of the AFPM are very promising. All the necessary com-
ponents are fitted together and operate as planned, hereby proving the
feasibility and functionality of the AFPM. There is, however, still margin
for improvement. The bottleneck in communication lies with the connec-
tion/communication between the vision-PC and the central control unit. Fu-
ture work will focus on improving the bandwidth of this connection and on
optimizing the image processing. The aim is to reduce the picking cycle period
from an average of 9 to about 5 seconds (or less). In that case, the produc-
tivity of the AFPM will be close to the work load of about 6 workers, which
makes the machine economically viable.
Special attention will go to improving the actual apple detach movement
by taking into account the relative pose of the apple with respect to the
branch. This should lower the stem pull percentage. Further research is also
planned to automate the navigation trough the orchard and to investigate the
harvesting of pears.
inria-00194739, version 1 - 7 Dec 2007
Autonomous Fruit Picking Machine: A Robotic Apple Harvester 9
1. Appeltants W and Ilsbroux W (1995) Automatische sturing van een plukwagen
(Autonomous guidance for a harvester). MA Thesis, Katholieke Hogeschool
Limburg, Belgium
2. Bulanon DM, Kataoka T, Okamoto H, Hata S (2005) Feedback control of ma-
nipulator using machine vision for robotic apple harvesting. In Proceedings of
ASAE, Paper No. 053114, Tampa, USA
3. Bulanon DM, Kataoka T, Ukamoto H, Hata S (2004) Development of a real-
time machine vision system for the apple harvesting robot. In SICE Annual
Conference in Sapporo, pages 595–598, Hokkaido Inst. of Techn., Japan
4. Hutchinson S, Hager G, Corke P (1996) A tutorial on visual servo control. IEEE
Trans. on Robotics and Automation, 12(5):651–670
5. Jimenez A, Ceres R, Pons J (2000) A survey of computer vision methods for
locating fruits on trees. Transactions of the ASAE, 43(6):1911–1920
6. Katypitiya J, Eaton R, Cole A, Meyer C, Rodnay G (2005) Automation of
an agricultural tractor for fruit picking. In IEEE Int. Conf. on Robotics and
Automation, pages 3212–3217, Barcelona, Spain
7. Mateus D, Avi˜na G, Devy M (2005) Robot visual navigation in semi-structured
outdoor environments. In IEEE Int. Conf. on Robotics and Automation, pages
4702–4707, Barcelona, Spain
8. MVTec Software GmbH, M¨unchen, Germany (2006) Halcon 7.1, Quick Guide,
2b edition
9. Peterson DL, Bennedsen BS, Anger WC, Wolford SD (1999) A systems
approach to robotic bulk harvesting of apples. Transactions of the ASAE,
10. Peterson DL (2005) Harvest mechanization progress and prospects for fresh
market quality deciduous tree fruits. HortTechnology, 15(1):72–75
11. Setiawan AI, Furukawa T, Preston A (2004) A low-cost gripper for an apple
picking robot. In IEEE Int. Conf. on Robotics and Automation, pages 4448–
4453, New Orleans, LA
12. Stajnko D and `
Emelik Z (2005) Modelling of apple fruit growth by application
of image analysis. Agriculturae Conspectus Scientificus, 70(2):59–64
13. Zhao J, Tow J, and Katupitiya J (2005) On-tree fruit recognition using texture
properties and color data. In Int. Conf. on Intelligent Robots and Systems,
pages 3993–3998, Alberta, Canada
inria-00194739, version 1 - 7 Dec 2007
... So far, many different highly integrated robotic systems have been developed to harvest apples fruit-by-fruit (Baeten et al., 2008;Bu et al., 2022;Bulanon & Kataoka, 2010;Hohimer et al., 2019;Silwal et al., 2017;Zhang et al., 2022Zhao et al., 2011). For example, Baeten et al. (2008) reported an apple harvesting robot, which consisted of an internal camera, a seven-degree-of-freedom (7-DOF) industrial manipulator, and a vacuum activated, funnel shaped gripper. ...
... So far, many different highly integrated robotic systems have been developed to harvest apples fruit-by-fruit (Baeten et al., 2008;Bu et al., 2022;Bulanon & Kataoka, 2010;Hohimer et al., 2019;Silwal et al., 2017;Zhang et al., 2022Zhao et al., 2011). For example, Baeten et al. (2008) reported an apple harvesting robot, which consisted of an internal camera, a seven-degree-of-freedom (7-DOF) industrial manipulator, and a vacuum activated, funnel shaped gripper. ...
... The RGB-D camera is mounted on a horizontal frame that is above the manipulator. Different from the other robotic harvesting systems (e.g., Baeten et al., 2008) that attach the camera to the manipulator or the end-effector, our installation scheme ensures that the RGB-D camera can provide a global view of the scene, which facilitates the use of multiple manipulators planned in our future versions. Since the depth measurement of consumer RGB-D camera is not stable under leaf/branch occlusions and/or challenging lighting conditions (see Neupane et al., 2021, for an in-depth review on localization performance of commercial RGB-D sensors), we design a new laser-camera unit to address this issue. ...
Full-text available
Decreased availability and rising cost in labor poses a serious threat to the long‐term profitability and sustainability of the apple industry in the United States and many other countries. Harvest automation is thus urgently needed. In this paper, we present the unified system design and field evaluation of a new apple harvesting robot. The robot is mainly composed of a specially designed perception component, a four‐degree‐of‐freedom manipulator, an improved vacuum‐based soft end‐effector, and a dropping/catching component to receive and transport picked fruits. Software algorithms are developed to enable synergistic coordination of the hardware components for efficient, automated harvesting of apples in challenging orchard environments. Specifically, by integrating modified triangulation and image processing and analysis algorithms, a novel perception strategy is developed to achieve robust apple detection and precise localization. Improved planning and control algorithms are developed to guide the robot to the target positions. The performance of the robotic system was evaluated through field tests in two apple orchards with different tree architectures and foliage conditions. In the orchard where trees were young and well‐pruned, the robot achieved 82.4% successful harvesting rate. In a second, older orchard with dense, clustered branches and foliage, the robot had 65.2% successful rate. The average cycle time to harvest a fruit was approximately 6 s, which included software algorithm processing and hardware execution. Moreover, through an in‐depth analysis of the obtained results, limitations and planned future works are discussed.
... The Beijing Institute of Intelligent Agriculture Equipment utilizes a monocular camera to calculate the accurate coordinates of peripheral fruits by measuring the deviation between the top fruit center and the image center [20] (as shown in Figure 2A). The University of KU Leuven in Belgium has designed an apple harvesting robot [21] that uses a monocular camera mounted at the center of a flexible gripper, ensuring the alignment between the gripper and the camera, simplifying the image-to-robot-end-effector coordinate transformation and reducing image distortion (as shown in Figure 2B). When approaching an apple, the remaining distance to the apple is calculated through triangulation. ...
... [20]. (B) Apple picking robot [21]. (C) Image processing of Lychee clusters in the orchard. ...
Full-text available
Intelligent agriculture imposes higher requirements on the recognition and localization of fruit and vegetable picking robots. Due to its unique visual information and relatively low hardware cost, machine vision is widely applied in the recognition and localization of fruit and vegetable picking robots. This article provides an overview of the application of machine vision in the recognition and localization of fruit and vegetable picking robots. Firstly, the advantages, disadvantages, and the roles of different visual sensors and machine vision algorithms in the recognition and localization of fruit and vegetable picking robots are introduced, including monocular cameras, stereo cameras, structured light cameras, multispectral cameras, image segmentation algorithms, object detection algorithms, and 3D reconstruction algorithms. Then, the current status and challenges faced by machine vision in the recognition and localization of fruit and vegetable picking robots are summarized. These challenges include the stability of fast recognition under complex background interference, stability of recognition under different lighting environments for the same crop, the reliance of recognition and localization on prior information in the presence of fruit overlap and occlusions caused by leaves and branches, and the uncertainty of picking caused by complex working environments. In current research on algorithms dealing with complex background interference and various occlusion disturbances, good results have been achieved. Different lighting environments have a significant impact on the recognition and positioning of fruits and vegetables, with a minimum accuracy of 59.2%. Finally, this article outlines future research directions to address these challenges.
... However, it was the 2010s that saw a true revolution in gripper design. Adaptive grippers emerged, their flexible fingers and suction cups adapting seamlessly to a variety of fruit shapes [33][34][35][36]. Soft grippers, employing compliant materials, further improved interactions, reducing the risk of damage. ...
Full-text available
Promotion of research and development in advanced technology must be implemented in agriculture to increase production in the current challenging environment where the demand for manual farming is decreasing due to the unavailability of skilled labor, high cost, and shortage of labor. In the last two decades, the demand for fruit harvester technologies, i.e., mechanized harvesting, manned and unmanned aerial systems, and robotics, has increased. However, several industries are working on the development of industrial-scale production of advanced harvesting technologies at low cost, but to date, no commercial robotic arm has been developed for selective harvesting of valuable fruits and vegetables, especially within controlled strictures, i.e., greenhouse and hydroponic contexts. This research article focused on all the parameters that are responsible for the development of automated robotic arms. A broad review of the related research works from the past two decades (2000 to 2022) is discussed, including their limitations and performance. In this study, data are obtained from various sources depending on the topic and scope of the review. Some common sources of data for writing this review paper are peer-reviewed journals, book chapters, and conference proceedings from Google Scholar. The entire requirement for a fruit harvester contains a manipulator for mechanical movement, a vision system for localizing and recognizing fruit, and an end-effector for detachment purposes. Performance, in terms of harvesting time, harvesting accuracy, and detection efficiency of several developments, has been summarized in this work. It is observed that improvement in harvesting efficiency and custom design of end-effectors is the main area of interest for researchers. The harvesting efficiency of the system is increased by the implementation of optimal techniques in its vision system that can acquire low recognition error rates.
... In the yield prediction of any fruit detection and counting is the prime need. Some traditional approaches like thresholding (Fernandez-Maloigne et al., 1993), morphological operations (Baeten et al., 2008), circular Hough transform (Grasso et al., 1996), and filtering (Ceres et al., 1998), edge detection (Ceres et al., 1998) etc. were used for fruit detection purpose. There are so many special methods available to extract the region of interest (ROI), which is nothing but fruit from the total image. ...
Full-text available
Traditional techniques for estimating the weight of clusters in a winery, generally consist of manually counting the variety of clusters per vine, and scaling by means of the entire variety of vines. This method can be arduous, costly, and its accuracy is dependent on the scale of the sample. To overcome these problems, hybrid approaches of Computer Vision (CV), Deep Learning (DL) and Machine Learning (ML) based vineyard yield prediction systems are proposed. Self-prepared datasets are used for comparative analysis of 2D and 3D yield prediction systems for vineyards. DL-based approach for segmentation operation on an RGB-D image dataset created with the D435I camera is used along with the ML-based weight prediction technique of grape clusters present in the single image is employed using these datasets. A comparative analysis of the DL-based Keras regression model and various ML-based regression models for the weight prediction task is taken into account, and finally a prediction model is proposed to estimate the yield of the entire vineyard. The analysis shows improved performance with the 3D vineyard yield prediction system compared to the 2D vineyard yield prediction system with grape cluster segmentation pixel accuracy upto 94.81% and yield prediction accuracy upto 99.58%.
... The designed configuration achieves better control of manipulator motion by minimizing the centripetal and coupling torques. A 6-DOF industrial robot is taken as the manipulator for fruit plucking applications, which include a new design of suction gripper (Baeten et al., 2008). The chosen robot can execute the plucking cycle period in an average of 5 seconds, as verified experimentally. ...
Full-text available
In the development of automated manipulators for fruit and vegetable picking technologies, the challenge of ensuring an efficient, stable, and loss-free picking process has been a complex problem. In such an environment, manipulators require the most efficient and robust control for effective operations. In this paper, a serial 9-DOF redundant manipulator (1P8R) is proposed with various controllers for trajectory tracking problems in agricultural applications. The dynamic analysis of redundant manipulator has been carried out using the Recursive Newton-Euler method. The joint configurations of the robot are determined using optimization techniques for specific Task Space Locations (TSLs) by avoiding obstacles. The process of generating joint trajectories has been implemented by considering the cubic polynomial function. For the task of controlling the robot trajectory tracking in the virtual agricultural environment, different combinations of Proportional (P), Integrative (I), Derivative (D), and Feed-Forward (FF) controllers are employed, and a comparative analysis has been performed among these controllers. To verify the performance of the manipulator, simulations are carried out in a virtual environment using Simulink software. Results show that the robot is able to reach specific TSL accurately with better control and it is found that the implementation of Feed-Forward and PID-CTC controllers has better performance in a complex environment.
The end effector characterizing a flipping–cutting way is designed firstly for picking spherical fruits with a good universality, which is mainly composed of three parts such as a clamping mechanism, a flipping mechanism, and a cutting mechanism. Subsequently, the kinematics simulation for the clamping mechanism, flipping mechanism, and cutting mechanism will be implemented by mathematical models and the ADAMS software, and results reflected through them are consistent with each other, thus presenting the feasibility of the structural scheme and two models. On the basis of scheme and results of models, the sample of the end effector is manufactured. Finally, picking experiments in a lab will be also conducted in view of diverse indexes, such as the successful grabbing action, flipping–cutting action, flipping–breaking action, average fruit picking time, success rate, and damage rate. It is drawn from experimental results that the success rate of grabbing is 100%, and a success rate for picking is over 80%. An average picking time is about 9.6s, and very few fruits are damaged. Therefore, such a scheme is feasible, further presenting a good reference for designing the entire robot in a field of picking diverse spherical fruits.
Full-text available
The idea of applying machine technology for fruit and vegetable harvesting has been around for 50 years. Various prototypes have been developed through the joint efforts of scholars worldwide. However, the existing prototypes of harvesting robots are still in the experimental research stage due to their low harvesting efficiency. With the help of information technology, related research has reached a milestone, which is full of opportunities and challenges for harvesting robot researchers. This paper briefly introduces the composition and operation process of harvesting robots, so that readers have a clear understanding of harvesting robots and their harvesting principles. Then the research results of harvesting robots at home and abroad are systematically summarized, and the research progress of harvesting robots is analyzed in detail from three aspects: rapid and accurate identification and positioning of target fruits, end-effector of harvesting robots, and application of harvesting path planning in harvesting robots. The results show that improving harvesting efficiency is the focus and hot spot of harvesting robot research. With the impetus of information technology, how to achieve fast and accurate recognition of multiple environments and multi-information outcomes, obtain reasonable path planning, and further optimize control strategies are all important research directions.
Full-text available
The possibility of RGB image processing and analysis for modelling of the development and growth of apple fruits was investigated during the two seasons under the orchards experiment in the four-years old ‘Golden Delicious’ and ‘Gala’ variety. The fruit detection depended significantly on the size and fruit’s colour of each growing stage, thus the correlation coefficients were continuously increasing from r=0.71 (2001) and r=0.73 (2002) after fruit tinning in June, up to r=0.88 (2001) and r=0.89 (2002) at harvesting in September respectively. The yield at harvest was estimated with the accuracy of 94% and 101% for ‘Golden Delicious’ and with 106% and 92% for ‘Gala’ respectively, whenever based on images captured on June 22 2001 and June 26 2002. Therefore, the image algorithm was proved to be equal or even better method for estimating the yield at harvest than the common ‘Prognosefruit’ method (accuracy 101% and 77% for ‘Golden Delicious’ and 72% and 63% for ‘Gala’ respectively).
Full-text available
This article provides a tutorial introduction to visual servo control of robotic manipulators. Since the topic spans many disciplines our goal is limited to providing a basic conceptual framework. We begin by reviewing the prerequisite topics from robotics and computer vision, including a brief review of coordinate transformations, velocity representation, and a description of the geometric aspects of the image formation process. We then present a taxonomy of visual servo control systems. The two major classes of systems, position-based and image-based systems, are then discussed in detail. Since any visual servo system must be capable of tracking image features in a sequence of images, we also include an overview of feature-based and correlation-based methods for tracking. We conclude the tutorial with a number of observations on the current directions of the research field of visual servo control
A unique robotic bulk harvester was conceived and developed to remove apples grown on narrow inclined trellises. The system combined mechanical harvesting technology with sensors and intelligent adaptive technology to identify an individual branch, determine fruit locations, position a Rapid Displacement Actuator (RDA) and a catching surface under the apples, and execute the RDA. Detachment was effected by rapidly displacing the limb away from the fruit. Requirements for a compatible tree training system were developed. Field testing demonstrated feasibility of the complete system. Fruit removal averaged 95% and detached fruit graded 99% U.S. Extra Fancy. Factors were identified to improve all aspects of the system and will require additional research.
Conference Paper
The challenges in developing a fruit harvesting robot are recognizing the fruit in the foliage, detaching the fruit from the tree, and the coordination of the vision system and the hand system. This paper presents the development of a feedback control of manipulator using machine vision. To measure the distance of the fruit from the camera, the laser ranging sensor is used. The detected fruit should be in the center of the image for its distance to be measured, so the camera should be positioned using the manipulator to position the fruit in the center of the image. A feedback control using machine vision is used for this positioning procedure. Two controllers are considered; threeposition on-off controller and proportional controller with variable gain. Both controllers were implemented in a simulation. Simulation results showed that both controllers were able to guide the manipulator to position the fruit in the center of the image.
Deciduous tree fruit crops such as apple (Malus domestica), peach (Prunus persica), and sweet cherry (Prunus avium) are not mechanically harvested for the fresh market. Attempts to mechanically harvest these fruits by mass removal techniques have not been successful due to excessive fruit damage caused during detachment, fall through the canopy, and collection. Robotic harvesters have not been commercially accepted due to insufficient fruit recovery. A U.S. Department of Agriculture-Agricultural Research Service (USDA-ARS) harvesting concept shows promise for harvesting both fresh market quality apples and sweet cherries. Successful mechanical harvesting of fresh market quality deciduous tree fruit will only occur when plant characteristics and machine designs are integrated into a compatible system. Cultivar characteristics that would facilitate machine harvesting are uniform fruit maturity at harvest, firm fruit that are resistant to mechanical damage, and compact growth habit that produces fruit in narrow canopies and on short/stiff limbs. Engineers must develop new detachment principles that minimize the energy input to effect fruit detachment, and develop durable energy-absorbing catching surfaces/conveyors to eliminate damage during collection of the fruit. As technology advances, sorting and sizing systems might be developed that can be operating on the harvester to eliminate culls in the field and deliver only fresh market quality fruit to the packers.
Conference Paper
This paper presents the hardware and software architectures of a compact agricultural tractor, that is currently being developed as a generic mobile platform for agricultural tasks. The specific task that is being addressed is fruit picking. Such systems require precision outdoor maneuvering and coordination with the implement attached to the vehicle. The system described also includes a loader attached to the front of the tractor which carries the fruit picking robot. First the hardware associated with the safety subsystem, steering, traction and loader control systems are described. Then the real-time software that coordinates the control of all subsystems while maintaining a functional radio link to a remote station is described.
Conference Paper
As a prelude to using stereo vision to accurately locate apples in an orchard, this paper presents a vision based algorithm to locate apples in a single image. On-tree situations of contrasting red and green apples as well as green apples in the orchard with poor contrast have been considered. The study found out that the redness in both cases of red and green apples can be used to differentiate apples from the rest of the orchard. Texture based edge detection has been combined with redness measures, and area thresholding followed by circle fitting, to determine the location of apples in the image plane. In the case of severely cluttered environments, Laplacian filters have been used to further clutter the foliage arrays by edge enhancement so that texture differences between the foliage and the apples increased thereby facilitating the separation of apples from the foliage. Results are presented that show the recognition of red and green apples in a number of situations as well as apples that are clustered together and/or occluded.
Conference Paper
This work describes a navigation framework for robots in semi-structured outdoor environments which enables planning of semantic tasks by chaining of elementary visual-based movement primitives. Navigation is achieved by understanding the underlying world behind the image and using these results as a guideline to control the robot. As retrieving semantic information from vision is computationally demanding, short-term tasks are planned and executed while new vision information is processed. Thanks to learning techniques, the methods are adapted to different environment conditions. Fusion and filtering techniques provide reliability and stability to the system. The procedures have been fully integrated and tested with a real robot in an experimental environment. Results are discussed.
Conference Paper
This research describes the development of a real-time machine vision system to guide a harvesting robotic manipulator for the red Fuji apples. The machine vision system is composed of a color CCD video camera to acquire Fuji apple images at the orchard and a PC to process the acquired images. The machine vision system was able to recognize the fruit under the different lighting conditions and it could locate the fruit in less than one second.
Conference Paper
A special gripper has been designed to pick apples from trees in an apple orchard. It is a low cost gripper with highly capability to pick an apple without scratching its skin. The gripper has been designed in accordance to the apple orchard environment and robot specification. Furthermore, the experiments show that the gripper can complete all tasks properly. The gripper was mounted on a manipulator and tested in an indoor laboratory environment. Experimental results show that the gripper could successfully pick apples.