Content uploaded by Ahmad R. Shahid
Author content
All content in this area was uploaded by Ahmad R. Shahid on Oct 31, 2019
Content may be subject to copyright.
Available via license: CC BY-NC-ND 4.0
Content may be subject to copyright.
Detection and Counting of On-Tree Citrus Fruit for
Crop Yield Estimation
Zeeshan Malik, Sheikh Ziauddin, Ahmad R. Shahid, and Asad Safi
Department of Computer Science,
COMSATS Institute of Information Technology,
Park Road, Islamabad
Abstract—In this paper, we present a technique to estimate
citrus fruit yield from the tree images. Manually counting the
fruit for yield estimation for marketing and other managerial
tasks is time consuming and requires human resources, which do
not always come cheap. Different approaches have been used for
the said purpose, yet separation of fruit from its background poses
challenges, and renders the exercise inaccurate. In this paper,
we use k-means segmentation for recognition of fruit, which
segments the image accurately thus enabling more accurate yield
estimation. We created a dataset containing 83 tree images with
4001 citrus fruits from three different fields. We are able to detect
the on-tree fruits with an accuracy of 91.3%. In addition, we find
a strong correlation between the manual and the automated fruit
count by getting coefficients of determination R2up to 0.99.
Keywords—Precision agriculture; yield estimation; k-means
segmentation; leaf occlusion;, illumination; morphology
I. INT ROD UC TI ON
In citrus groves, yield estimation is typically carried out
a few weeks earlier to fruitage to estimate the resource
requirement. The manual process of crop assessment is oc-
casionally done by hand pawns. Preferably, yield would be
estimated at numerous periods during crop development but
it requires enormous labor cost and time. Precise, low-cost
yield estimation is important for cultivators, especially if it
can be done timely in the growing season. As orange juice
needs to be processed within 48 hours of harvesting, orange
juice manufacturers need suppliers to provide accurate yield
estimates to guarantee that their juice plants can run at full
capacity given the time constraints. Additionally, more accu-
rate yield estimates will allow farmers to plan more precisely
for their harvesting labor and other logistical needs. Image
processing can help in improving the decision making process
for irrigation, fruit sorting and yield estimation [1]. Detection
of fruit is important as the subsequent fruit counting depends
on accurate detection.
Citrus fruit have different properties that can be used for
detection purposes. The most natural property to be used
for such purpose is the color. In the past, color has been
extensively used [2], [3], [4], [5], [6], [7], [8], [9]. Color
on is own may not provide enough information as it may
change depending upon how ripe the fruit is. For instance,
unripe oranges may be greenish, while over-ripe oranges may
be brownish. That poses a challenge when detection is based
on filtering of orange color only.
Lighting may pose another challenge as the oranges may
appear differently, under varying lighting conditions, such as
bright sunlight, cloudy, and evening. Figure 1 shows a tree
image under two varying light conditions. The image on the
left is captured on a cloudy day and the one on the right is
captured under direct sunlight. Under cloudy conditions, the
images are more consistent in terms of brightness and intensity
changes. While in the latter case, things may look different if
they are exposed to direct sunlight as opposed to when they
are under a shadow.
Another problem is that of occlusion. Occlusion may be
caused either by leaves or by the neighboring oranges, as can
be seen from Figure 2. In the first case, a single orange may
be counted as more than one, as leaves may affect the shape
of the orange. In the later case, may fruits may appear as one
larger fruit with irregular shape. In the first case, a need arises
for a way to count two or more smaller citrus blobs as one,
while in the later case we need to break down a larger blob
into smaller units, where each unit is counted as a separate
fruit.
II. RE LATE D WOR K
Despite the above-mentioned challenges involved, image
processing has been used with good results for automated
fruit counting and yield estimation. In the past, a variety of
techniques have been used across diverse fruits. Stajnko et
al. [10] used a combination of normalized chromaticity coor-
dinates (RGB) and thermal images to estimate on-tree count
of apple fruit. In order to separate pixels more precisely, they
also used the normalized difference index (NDI). Using their
technique, they were able to achieve correlation coefficient R2
of 0.83 to 0.88 at different times during the vegetation period.
They observed that the values of R2were improved during
the fruit ripening period. In a subsequent work [11], they used
surface texture and color features to segment apples from the
background. HSI color space was used for segmentation using
a threshold which was empirically selected. Red color intensity
(3∗R−G−B), contrast, uniformity and entropy were the main
features used. They achieved a detection accuracy of 89% with
a false detection of 2.2%.
Zhou et al. [12] also used color features for on-tree apple
counting. Fifty tree images were captured at two different times
during the season. The images were captured under normal
daylight conditions. For the images captured earlier in the
season, they used color RGB difference (R−Band G−R)
while two different color models were used for the images
captured during ripening season. They achieved R2scores of
0.8 and 0.85 for the two seasons, respectively. Regunathan
and Lee [13] presented a technique for citrus detection and
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 7, No. 5, 2016
519 | P a g e
www.ijacsa.thesai.org
Fig. 1: Citrus tree images captured under two different sunlight
conditions.
size finding. Their technique relies on an untrasonic sensor
to determine the distance between the camera and the fruit.
They used hue and saturation features in combination with
three different classifiers namely Beyesian, neural network and
Fischer’s linear discriminant to segment and differentiate fruit
from the background.
Hannan et al. [14] used red chromaticity coefficient for
image enhancement which helped in better segmentation in
variable light orange tree images. They used shape analysis
technique to detect overlapped oranges achieving a detection
rate of 90% with 4% false detection. Yet another citrus fruit
counting scheme was presented by Chinchuluun et al. [15].
They used Bayesian classification in addition to morphological
operations and watershed segmentation for citrus segmentation
and detection. They reported a correlation coefficient value
of 0.89 between the manual and the automated fruit count.
Billingsley [16] presented a machine vision system for count-
ing Macadamia nuts on a harvester roller. RGB color features
were used to separate the nuts and the leaves from the roller.
To segment the nuts from the leaves, they used a modified
version of circular Hough transform. Cakir et al. [17] used
color histograms and shape analysis to detect and count orange
fruit on tree images. They were able to achieve a detection
accuracy of 76.5%.
Annamalai and Lee [18] developed an algorithm for citrus
counting. They used hue and saturation color planes and
utilized histogram information to segment the fruit from the
background and the leaves. Erosion and dilation operations
were used to remove noise. They found R2of 0.76 between
the manual count and the count obtained using their algorithm.
In this paper, we present a technique for automated yield
estimation using k-means segmentation. The proposed tech-
nique is able to detect and count citrus fruit by catering for
changes in color both due to lighting conditions as well as the
state of ripeness of the fruit. Our system also takes care of the
cases where the fruit is occluded either by the leaves or by
other fruit. The experimental evaluation shows the accuracy
and robustness of the proposed scheme as described in the
subsequent sections.
The rest of this paper is organized as follows: The proposed
scheme is presented in Section 2. The results of our experi-
ments are described in Section 3 while Section 4 concludes
this paper.
Fig. 2: Orange fruit occluded by leaves (left). Orange fruit
overlapping each other (right).
Fig. 3: An overview of the proposed technique.
III. PROP OS ED TE CH NI QU E
The technique we proposed here uses k-means segmenta-
tion algorithm on orange tree images. First, a few preprocess-
ing steps are performed including noise reduction and image
enhancement. Next, we minimize shadow effects from the
image. Then, we extract oranges using blob detection and size
calculation which is then followed by the final yield estimation.
Figure 3 gives an overview of the proposed technique by
showing its different steps in a sequential order. The details
of these steps are given in the following sub-sections.
1) Image Preprocessing: In general, images contain a lot of
redundant information not needed for the application in hand.
The image may contain noise which makes the edge detection
and the segmentation tasks prone to errors. Therefore, it is
often necessary to perform certain type of noise reduction
and image enhancement before any meaningful processing of
the images. In this paper, we use the Perona-Malik model for
image enhancement and noise reduction [19]. It smoothes the
image without effecting significant features of the image such
as lines, edges or other important details that are necessary for
analysis and interpretation of the images. In this model, image
is smoothed using the following mathematical relation.
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 7, No. 5, 2016
520 | P a g e
www.ijacsa.thesai.org
Fig. 4: Shadow reduction step. The image on left is before the
shadow reduction while that on right is the same image after
shadow reduction step is applied.
∂I
∂t =div(c(x, y, t)∇I) = c(x, y , t)∆I+∇c.∆I(1)
where div denotes the divergence operator, while ∇and ∆
represents gradient and Laplacian, respectively. The diffusivity
function in Perona-Malik model is given by the following
equation.
c(x, y, t) = g(||∇I(x, y, t)||)(2)
where gis a nonnegative, monotonically decreasing func-
tion with g(0) = 1.
A. Shadow Reduction
Lighting conditions play a very important role in the per-
formance of many computer vision algorithms. In particular,
shadows in an image can cause problems in recognition,
segmentation and tracking algorithms to produce desirable
outcomes. Distinct objects can be combined through shadows
and they can obscure object recognition systems. When dealing
with outdoor images, shadow minimization is an important
task. In order to minimize shadow, we adjust the luminosity of
the image. First, we convert the RGB image to L∗a∗bimage,
where Lis the luminosity layer, while aand brepresents
color-opponent dimensions. Next, we increase the luminosity
of the image which results in reduced shadow effect in the
image. Finally, the image is converted to RGB color space after
replacing the luminosity layer with the processed data. Figure 4
shows an image after shadow reduction step is applied.
B. Object Separation
One of the main challenges in orange counting is that of
overlapped oranges. Due to overlapping, multiple fruits may
be counted as a single fruit which negatively impacts the
fruit counts and the yield estimates. In order to overcome this
challenge and to separate the overlapping fruit, we convolve
the image with a variance mask. After the convolution, each
pixel in the output RGB image contains the neighbor variance
of the R, G and B channels, respectively [20]. The image is
then transformed to gray scale by taking the mean of the three
color channels. Finally, a threshold is applied on the image.
These steps not only separate overlapping fruit, they also help
in reducing undesired edges such as those within the foliage
or the grass. Figure 5 shows an image after object separation
step is applied.
Fig. 5: Object separation step. The image on left is before the
object separation while that on right is the same image after
object separation step is applied.
C. K-Means Segmentation and Orange Extraction
Image Segmentation is the most important part of the
whole process of yield estimation. We use k-means clustering
algorithm for orange segmentation. K-means clustering is an
unsupervised classification technique which deals with finding
a pattern in a collection of unlabelled data [21]. The k-means
algorithm aims at minimizing a squared error function by
iteratively reorganizing the clusters. The iterations continue
until the cluster means do not move further than a certain
cut-off value.
K-means algorithm is popular due to its simplicity and
relatively low computational complexity. It is suitable in our
scenario as the number of clusters (K) is easy to choose.
An image of an orange tree generally consists of regions
representing the oranges, leaves, branches and sky. Therefore,
we select K to be 4 corresponding to these 4 regions. After the
clustering, we apply thresholding to extract the oranges from
the tree images. Each object in the image is segmented using
a particular RGB value.
In many cases, the fruit is visually fragmented because of
the obstructions caused by the leaves. This makes the counting
error prone as one fruit may be counted as two or more
fruits. In order to remove the smaller fragments of these fruits,
we converted the image into binary and applied the erosion
operation. The erosion of a binary image Aby the structuring
element Bis defined by [22]:
AB={z|(B)z⊆A}(3)
i.e., the output of erosion functions contains all pixels zin
Asuch that Bis in Awhen origin of B=z.
Figure 6 shows two images. On the left, there is input
image while the output of k-means segmentation is shown on
the right.
D. Blob Detection and Orange Counting
The final step of the proposed orange counting technique
is to detect monolithic fruit regions, which are also known as
blobs or objects. After the erosion operation, we find the con-
nected components in the binary image using 8-connectivity.
Each connected component corresponds to one orange. We
count the number of connected components which gives us
the number of oranges in a particular tree image.
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 7, No. 5, 2016
521 | P a g e
www.ijacsa.thesai.org
Fig. 6: K-means segmenation step. The image on left is the
input image while that on right is the output after k-means
segmentation is applied.
Fig. 7: Comparison of citrus fruit count per tree counted
manually against that calculated using the proposed technique
for Khanpur dataset.
IV. EXP ER IM EN TAL EVALUATIO N
A. Dataset
We have collected a total of 83 tree images taken from
Khapur and National Agricultural Research Council (NARC)
fields. The images were taken at three times with an average
gap of around one month. In the rest of this paper, we will
call these three datasets Khanpur, NARC 1 and NARC 2,
respectively. Khanpur dataset contains 23 tree images while
NARC 1 and NARC 2 contain 16 and 44 tree images, re-
spectively. The images contain variable illumination including
shadows, bright sunlight and dusk. We prepared a ground truth
by manually counting and marking all the orange fruit in all
the input images.
B. Results
We performed experiments by getting automated citrus
counting using the proposed scheme and comparing the results
with the ground truth. The experimental results show a correct
detection rate of 91.3%, as shown in Table I. In addition to
finding detection rate, we used linear regression to model the
relationship between the automated results and the ground
truth. Figures 7, 8,and 9, show the result of plotting manual and
automated orange counts for Khanpur, NARC 1 and NARC
2 datasets, respectively. The figures also show the regression
equations as well as the coefficient of determination R2. As
you can see, neither the accuracy in Table I, nor the coefficients
of determination in Figures 7, 8, 9 vary significantly among
different datasets. As the datasets were created at different
Fig. 8: Comparison of citrus fruit count per tree counted
manually against that calculated using the proposed technique
for NARC 1 dataset.
Fig. 9: Comparison of citrus fruit count per tree counted
manually against that calculated using the proposed technique
for NARC 2 dataset.
times with varying lighting conditions, the consistency in
results shows the robustness of the proposed technique.
In Table II, we compare our results with the past work.
Some of the past work has presented results in terms of
detection accuracy while some has presented it in terms of
coefficient of determination (R2). Table II shows that our
technique is capable of detecting and counting the fruits with
more accuracy as compared to others.
V. CO NC LU SI ON
In this paper, we have presented a technique for the
segmentation, detection and yield measurement of citrus fruit.
The proposed approach gives very good results under varying
lighting conditions, occlusion of leaves and overlapping of
TABLE I: Orange detection and counting results of the pro-
posed scheme.
Dataset No of Images No of Fruits Machine Count Accuracy (%)
Khanpur 23 1662 1532 92.2
NARC 1 16 543 501 92.3
NARC 2 44 1796 1621 90.3
Overall 83 4001 3654 91.30%
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 7, No. 5, 2016
522 | P a g e
www.ijacsa.thesai.org
TABLE II: Comparison with the previous work.
Technique Accuracy (%) R2Value
Cakir et al. [17] 76.5 -
Hannan et al. [14] 90 -
Chinchuluunet al. [15] - 0.89
Annamalai and Lee [18] - 0.76
Proposed Technique 91.3 0.94 to 0.99
fruits on the images taken from varying distances from the
orange trees. Our experiments on three different datasets
showed an accuracy of 91.3% with R2values of up to 0.99.
In the future, we aim to collect larger datasets for further
experiments. In addition, instead of taking images manually
from the citrus trees, we plan to use a camera-mounted robot
for image acquisition.
REF ER EN CE S
[1] C. Pohl and J. Van Genderen, “Review article multisensor image fusion
in remote sensing: concepts, methods and applications,” International
journal of remote sensing, vol. 19, no. 5, pp. 823–854, 1998.
[2] D. Bulanon, T. Kataoka, S. Zhang, Y. Ota, and T. Hiroma, “Optimal
thresholding for the automatic recognition of apple fruits,” ASAE, no.
01-3033, 2001.
[3] T. Burks, “A real-time machine vision algorithm for robotic citrus
harvesting,” 2007.
[4] M. W. Hannan and T. F. Burks, “Current developments in automated
citrus harvesting,” in ASABE Annual International Meeting, 2004.
[5] R. Harrell, D. Slaughter, and P. Adsit, “A fruit-tracking system for
robotic harvesting,” Machine Vision and Applications, vol. 2, no. 2,
pp. 69–80, 1989.
[6] R. Harrell, P. Adsit, T. Pool, R. Hoffman et al., “The florida robotic
grove-lab.” Transactions of the ASAE, vol. 33, no. 2, pp. 391–399, 1990.
[7] A. Jimenez, R. Ceres, J. Pons et al., “A survey of computer vision
methods for locating fruit on trees,” Transactions of the ASAE-American
Society of Agricultural Engineers, vol. 43, no. 6, pp. 1911–1920, 2000.
[8] F. Pla, F. Juste, and F. Ferri, “Feature extraction of spherical objects in
image analysis: an application to robotic citrus harvesting,” Computers
and Electronics in Agriculture, vol. 8, no. 1, pp. 57–72, 1993.
[9] E. Tutle, “Image controlled robotics in agricultural environments,” 1984.
[10] D. Stajnko, M. Lakota, and M. Hoˇ
cevar, “Estimation of number and
diameter of apple fruits in an orchard during the growing season by
thermal imaging,” Computers and Electronics in Agriculture, vol. 42,
no. 1, pp. 31–42, 2004.
[11] D. Stajnko, J. Rakun, M. Blanke et al., “Modelling apple fruit yield
using image analysis for fruit colour, shape and texture.” European
journal of horticultural science, vol. 74, no. 6, pp. 260–267, 2009.
[12] R. Zhou, L. Damerow, Y. Sun, and M. M. Blanke, “Using colour
features of cv. ’gala’ apple fruits in an orchard in image processing
to predict yield,” Precision Agriculture, vol. 13, no. 5, pp. 568–580,
2012.
[13] M. Regunathan and W. S. Lee, “Citrus fruit identification and size
determination using machine vision and ultrasonic sensors,” in ASABE
Annual International Meeting, 2005.
[14] M. Hannan, T. Burks, and D. M. Bulanon, “A machine vision algo-
rithm combining adaptive segmentation and shape analysis for orange
fruit detection,” Agricultural Engineering International: CIGR Journal,
2010.
[15] R. Chinchuluun, W. Lee, R. Ehsani et al., “Machine vision system
for determining citrus count and size on a canopy shake and catch
harvester,” Applied Engineering in Agriculture, vol. 25, no. 4, pp. 451–
458, 2009.
[16] J. Billingsley, “More machine vision applications in the NCEA,” in
Mechatronics and Machine Vision in Practice. Springer, 2008, pp.
333–343.
[17] Y. Cakir, M. Kirci, E. O. Gunes, and B. B. Ustundag, “Detection
of oranges in outdoor conditions,” in Agro-Geoinformatics (Agro-
Geoinformatics), 2013 Second International Conference on. IEEE,
2013, pp. 500–503.
[18] P. Annamalai and W. Lee, “Citrus yield mapping system using machine
vision,” in ASAE Annual International Meeting, 2003.
[19] P. Perona and J. Malik, “Scale-space and edge detection using
anisotropic diffusion,” Pattern Analysis and Machine Intelligence, IEEE
Transactions on, vol. 12, no. 7, pp. 629–639, 1990.
[20] A. B. Payne, K. B. Walsh, P. Subedi, and D. Jarvis, “Estimation
of mango crop yield using image analysis–segmentation method,”
Computers and Electronics in Agriculture, vol. 91, pp. 57–64, 2013.
[21] D. Fradkin and I. Muchnik, “A study of k-means clustering for im-
proving classification accuracy of multi-class SVM,” Technical Report.
Rutgers University, New Brunswick, New Jersey 08854, Tech. Rep.,
2004.
[22] R. C. Gonzalez and R. E. Woods, Digital Image Processing. Prentice
Hall, 2007.
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 7, No. 5, 2016
523 | P a g e
www.ijacsa.thesai.org