ArticlePDF Available

Abstract and Figures

In this paper, we present a technique to estimate citrus fruit yield from the tree images. Manually counting the fruit for yield estimation for marketing and other managerial tasks is time consuming and requires human resources, which do not always come cheap. Different approaches have been used for the said purpose, yet separation of fruit from its background poses challenges, and renders the exercise inaccurate. In this paper, we use k-means segmentation for recognition of fruit, which segments the image accurately thus enabling more accurate yield estimation. We created a dataset containing 83 tree images with 4001 citrus fruits from three different fields. We are able to detect the on-tree fruits with an accuracy of 91.3%. In addition, we find a strong correlation between the manual and the automated fruit count by getting coefficients of determination R2 up to 0.99.
Content may be subject to copyright.
Detection and Counting of On-Tree Citrus Fruit for
Crop Yield Estimation
Zeeshan Malik, Sheikh Ziauddin, Ahmad R. Shahid, and Asad Safi
Department of Computer Science,
COMSATS Institute of Information Technology,
Park Road, Islamabad
Abstract—In this paper, we present a technique to estimate
citrus fruit yield from the tree images. Manually counting the
fruit for yield estimation for marketing and other managerial
tasks is time consuming and requires human resources, which do
not always come cheap. Different approaches have been used for
the said purpose, yet separation of fruit from its background poses
challenges, and renders the exercise inaccurate. In this paper,
we use k-means segmentation for recognition of fruit, which
segments the image accurately thus enabling more accurate yield
estimation. We created a dataset containing 83 tree images with
4001 citrus fruits from three different fields. We are able to detect
the on-tree fruits with an accuracy of 91.3%. In addition, we find
a strong correlation between the manual and the automated fruit
count by getting coefficients of determination R2up to 0.99.
KeywordsPrecision agriculture; yield estimation; k-means
segmentation; leaf occlusion;, illumination; morphology
I. INT ROD UC TI ON
In citrus groves, yield estimation is typically carried out
a few weeks earlier to fruitage to estimate the resource
requirement. The manual process of crop assessment is oc-
casionally done by hand pawns. Preferably, yield would be
estimated at numerous periods during crop development but
it requires enormous labor cost and time. Precise, low-cost
yield estimation is important for cultivators, especially if it
can be done timely in the growing season. As orange juice
needs to be processed within 48 hours of harvesting, orange
juice manufacturers need suppliers to provide accurate yield
estimates to guarantee that their juice plants can run at full
capacity given the time constraints. Additionally, more accu-
rate yield estimates will allow farmers to plan more precisely
for their harvesting labor and other logistical needs. Image
processing can help in improving the decision making process
for irrigation, fruit sorting and yield estimation [1]. Detection
of fruit is important as the subsequent fruit counting depends
on accurate detection.
Citrus fruit have different properties that can be used for
detection purposes. The most natural property to be used
for such purpose is the color. In the past, color has been
extensively used [2], [3], [4], [5], [6], [7], [8], [9]. Color
on is own may not provide enough information as it may
change depending upon how ripe the fruit is. For instance,
unripe oranges may be greenish, while over-ripe oranges may
be brownish. That poses a challenge when detection is based
on filtering of orange color only.
Lighting may pose another challenge as the oranges may
appear differently, under varying lighting conditions, such as
bright sunlight, cloudy, and evening. Figure 1 shows a tree
image under two varying light conditions. The image on the
left is captured on a cloudy day and the one on the right is
captured under direct sunlight. Under cloudy conditions, the
images are more consistent in terms of brightness and intensity
changes. While in the latter case, things may look different if
they are exposed to direct sunlight as opposed to when they
are under a shadow.
Another problem is that of occlusion. Occlusion may be
caused either by leaves or by the neighboring oranges, as can
be seen from Figure 2. In the first case, a single orange may
be counted as more than one, as leaves may affect the shape
of the orange. In the later case, may fruits may appear as one
larger fruit with irregular shape. In the first case, a need arises
for a way to count two or more smaller citrus blobs as one,
while in the later case we need to break down a larger blob
into smaller units, where each unit is counted as a separate
fruit.
II. RE LATE D WOR K
Despite the above-mentioned challenges involved, image
processing has been used with good results for automated
fruit counting and yield estimation. In the past, a variety of
techniques have been used across diverse fruits. Stajnko et
al. [10] used a combination of normalized chromaticity coor-
dinates (RGB) and thermal images to estimate on-tree count
of apple fruit. In order to separate pixels more precisely, they
also used the normalized difference index (NDI). Using their
technique, they were able to achieve correlation coefficient R2
of 0.83 to 0.88 at different times during the vegetation period.
They observed that the values of R2were improved during
the fruit ripening period. In a subsequent work [11], they used
surface texture and color features to segment apples from the
background. HSI color space was used for segmentation using
a threshold which was empirically selected. Red color intensity
(3RGB), contrast, uniformity and entropy were the main
features used. They achieved a detection accuracy of 89% with
a false detection of 2.2%.
Zhou et al. [12] also used color features for on-tree apple
counting. Fifty tree images were captured at two different times
during the season. The images were captured under normal
daylight conditions. For the images captured earlier in the
season, they used color RGB difference (RBand GR)
while two different color models were used for the images
captured during ripening season. They achieved R2scores of
0.8 and 0.85 for the two seasons, respectively. Regunathan
and Lee [13] presented a technique for citrus detection and
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 7, No. 5, 2016
519 | P a g e
www.ijacsa.thesai.org
Fig. 1: Citrus tree images captured under two different sunlight
conditions.
size finding. Their technique relies on an untrasonic sensor
to determine the distance between the camera and the fruit.
They used hue and saturation features in combination with
three different classifiers namely Beyesian, neural network and
Fischer’s linear discriminant to segment and differentiate fruit
from the background.
Hannan et al. [14] used red chromaticity coefficient for
image enhancement which helped in better segmentation in
variable light orange tree images. They used shape analysis
technique to detect overlapped oranges achieving a detection
rate of 90% with 4% false detection. Yet another citrus fruit
counting scheme was presented by Chinchuluun et al. [15].
They used Bayesian classification in addition to morphological
operations and watershed segmentation for citrus segmentation
and detection. They reported a correlation coefficient value
of 0.89 between the manual and the automated fruit count.
Billingsley [16] presented a machine vision system for count-
ing Macadamia nuts on a harvester roller. RGB color features
were used to separate the nuts and the leaves from the roller.
To segment the nuts from the leaves, they used a modified
version of circular Hough transform. Cakir et al. [17] used
color histograms and shape analysis to detect and count orange
fruit on tree images. They were able to achieve a detection
accuracy of 76.5%.
Annamalai and Lee [18] developed an algorithm for citrus
counting. They used hue and saturation color planes and
utilized histogram information to segment the fruit from the
background and the leaves. Erosion and dilation operations
were used to remove noise. They found R2of 0.76 between
the manual count and the count obtained using their algorithm.
In this paper, we present a technique for automated yield
estimation using k-means segmentation. The proposed tech-
nique is able to detect and count citrus fruit by catering for
changes in color both due to lighting conditions as well as the
state of ripeness of the fruit. Our system also takes care of the
cases where the fruit is occluded either by the leaves or by
other fruit. The experimental evaluation shows the accuracy
and robustness of the proposed scheme as described in the
subsequent sections.
The rest of this paper is organized as follows: The proposed
scheme is presented in Section 2. The results of our experi-
ments are described in Section 3 while Section 4 concludes
this paper.
Fig. 2: Orange fruit occluded by leaves (left). Orange fruit
overlapping each other (right).
Fig. 3: An overview of the proposed technique.
III. PROP OS ED TE CH NI QU E
The technique we proposed here uses k-means segmenta-
tion algorithm on orange tree images. First, a few preprocess-
ing steps are performed including noise reduction and image
enhancement. Next, we minimize shadow effects from the
image. Then, we extract oranges using blob detection and size
calculation which is then followed by the final yield estimation.
Figure 3 gives an overview of the proposed technique by
showing its different steps in a sequential order. The details
of these steps are given in the following sub-sections.
1) Image Preprocessing: In general, images contain a lot of
redundant information not needed for the application in hand.
The image may contain noise which makes the edge detection
and the segmentation tasks prone to errors. Therefore, it is
often necessary to perform certain type of noise reduction
and image enhancement before any meaningful processing of
the images. In this paper, we use the Perona-Malik model for
image enhancement and noise reduction [19]. It smoothes the
image without effecting significant features of the image such
as lines, edges or other important details that are necessary for
analysis and interpretation of the images. In this model, image
is smoothed using the following mathematical relation.
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 7, No. 5, 2016
520 | P a g e
www.ijacsa.thesai.org
Fig. 4: Shadow reduction step. The image on left is before the
shadow reduction while that on right is the same image after
shadow reduction step is applied.
∂I
∂t =div(c(x, y, t)I) = c(x, y , t)∆I+c.I(1)
where div denotes the divergence operator, while and
represents gradient and Laplacian, respectively. The diffusivity
function in Perona-Malik model is given by the following
equation.
c(x, y, t) = g(||∇I(x, y, t)||)(2)
where gis a nonnegative, monotonically decreasing func-
tion with g(0) = 1.
A. Shadow Reduction
Lighting conditions play a very important role in the per-
formance of many computer vision algorithms. In particular,
shadows in an image can cause problems in recognition,
segmentation and tracking algorithms to produce desirable
outcomes. Distinct objects can be combined through shadows
and they can obscure object recognition systems. When dealing
with outdoor images, shadow minimization is an important
task. In order to minimize shadow, we adjust the luminosity of
the image. First, we convert the RGB image to Labimage,
where Lis the luminosity layer, while aand brepresents
color-opponent dimensions. Next, we increase the luminosity
of the image which results in reduced shadow effect in the
image. Finally, the image is converted to RGB color space after
replacing the luminosity layer with the processed data. Figure 4
shows an image after shadow reduction step is applied.
B. Object Separation
One of the main challenges in orange counting is that of
overlapped oranges. Due to overlapping, multiple fruits may
be counted as a single fruit which negatively impacts the
fruit counts and the yield estimates. In order to overcome this
challenge and to separate the overlapping fruit, we convolve
the image with a variance mask. After the convolution, each
pixel in the output RGB image contains the neighbor variance
of the R, G and B channels, respectively [20]. The image is
then transformed to gray scale by taking the mean of the three
color channels. Finally, a threshold is applied on the image.
These steps not only separate overlapping fruit, they also help
in reducing undesired edges such as those within the foliage
or the grass. Figure 5 shows an image after object separation
step is applied.
Fig. 5: Object separation step. The image on left is before the
object separation while that on right is the same image after
object separation step is applied.
C. K-Means Segmentation and Orange Extraction
Image Segmentation is the most important part of the
whole process of yield estimation. We use k-means clustering
algorithm for orange segmentation. K-means clustering is an
unsupervised classification technique which deals with finding
a pattern in a collection of unlabelled data [21]. The k-means
algorithm aims at minimizing a squared error function by
iteratively reorganizing the clusters. The iterations continue
until the cluster means do not move further than a certain
cut-off value.
K-means algorithm is popular due to its simplicity and
relatively low computational complexity. It is suitable in our
scenario as the number of clusters (K) is easy to choose.
An image of an orange tree generally consists of regions
representing the oranges, leaves, branches and sky. Therefore,
we select K to be 4 corresponding to these 4 regions. After the
clustering, we apply thresholding to extract the oranges from
the tree images. Each object in the image is segmented using
a particular RGB value.
In many cases, the fruit is visually fragmented because of
the obstructions caused by the leaves. This makes the counting
error prone as one fruit may be counted as two or more
fruits. In order to remove the smaller fragments of these fruits,
we converted the image into binary and applied the erosion
operation. The erosion of a binary image Aby the structuring
element Bis defined by [22]:
AB={z|(B)zA}(3)
i.e., the output of erosion functions contains all pixels zin
Asuch that Bis in Awhen origin of B=z.
Figure 6 shows two images. On the left, there is input
image while the output of k-means segmentation is shown on
the right.
D. Blob Detection and Orange Counting
The final step of the proposed orange counting technique
is to detect monolithic fruit regions, which are also known as
blobs or objects. After the erosion operation, we find the con-
nected components in the binary image using 8-connectivity.
Each connected component corresponds to one orange. We
count the number of connected components which gives us
the number of oranges in a particular tree image.
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 7, No. 5, 2016
521 | P a g e
www.ijacsa.thesai.org
Fig. 6: K-means segmenation step. The image on left is the
input image while that on right is the output after k-means
segmentation is applied.
Fig. 7: Comparison of citrus fruit count per tree counted
manually against that calculated using the proposed technique
for Khanpur dataset.
IV. EXP ER IM EN TAL EVALUATIO N
A. Dataset
We have collected a total of 83 tree images taken from
Khapur and National Agricultural Research Council (NARC)
fields. The images were taken at three times with an average
gap of around one month. In the rest of this paper, we will
call these three datasets Khanpur, NARC 1 and NARC 2,
respectively. Khanpur dataset contains 23 tree images while
NARC 1 and NARC 2 contain 16 and 44 tree images, re-
spectively. The images contain variable illumination including
shadows, bright sunlight and dusk. We prepared a ground truth
by manually counting and marking all the orange fruit in all
the input images.
B. Results
We performed experiments by getting automated citrus
counting using the proposed scheme and comparing the results
with the ground truth. The experimental results show a correct
detection rate of 91.3%, as shown in Table I. In addition to
finding detection rate, we used linear regression to model the
relationship between the automated results and the ground
truth. Figures 7, 8,and 9, show the result of plotting manual and
automated orange counts for Khanpur, NARC 1 and NARC
2 datasets, respectively. The figures also show the regression
equations as well as the coefficient of determination R2. As
you can see, neither the accuracy in Table I, nor the coefficients
of determination in Figures 7, 8, 9 vary significantly among
different datasets. As the datasets were created at different
Fig. 8: Comparison of citrus fruit count per tree counted
manually against that calculated using the proposed technique
for NARC 1 dataset.
Fig. 9: Comparison of citrus fruit count per tree counted
manually against that calculated using the proposed technique
for NARC 2 dataset.
times with varying lighting conditions, the consistency in
results shows the robustness of the proposed technique.
In Table II, we compare our results with the past work.
Some of the past work has presented results in terms of
detection accuracy while some has presented it in terms of
coefficient of determination (R2). Table II shows that our
technique is capable of detecting and counting the fruits with
more accuracy as compared to others.
V. CO NC LU SI ON
In this paper, we have presented a technique for the
segmentation, detection and yield measurement of citrus fruit.
The proposed approach gives very good results under varying
lighting conditions, occlusion of leaves and overlapping of
TABLE I: Orange detection and counting results of the pro-
posed scheme.
Dataset No of Images No of Fruits Machine Count Accuracy (%)
Khanpur 23 1662 1532 92.2
NARC 1 16 543 501 92.3
NARC 2 44 1796 1621 90.3
Overall 83 4001 3654 91.30%
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 7, No. 5, 2016
522 | P a g e
www.ijacsa.thesai.org
TABLE II: Comparison with the previous work.
Technique Accuracy (%) R2Value
Cakir et al. [17] 76.5 -
Hannan et al. [14] 90 -
Chinchuluunet al. [15] - 0.89
Annamalai and Lee [18] - 0.76
Proposed Technique 91.3 0.94 to 0.99
fruits on the images taken from varying distances from the
orange trees. Our experiments on three different datasets
showed an accuracy of 91.3% with R2values of up to 0.99.
In the future, we aim to collect larger datasets for further
experiments. In addition, instead of taking images manually
from the citrus trees, we plan to use a camera-mounted robot
for image acquisition.
REF ER EN CE S
[1] C. Pohl and J. Van Genderen, “Review article multisensor image fusion
in remote sensing: concepts, methods and applications,” International
journal of remote sensing, vol. 19, no. 5, pp. 823–854, 1998.
[2] D. Bulanon, T. Kataoka, S. Zhang, Y. Ota, and T. Hiroma, “Optimal
thresholding for the automatic recognition of apple fruits,” ASAE, no.
01-3033, 2001.
[3] T. Burks, “A real-time machine vision algorithm for robotic citrus
harvesting,” 2007.
[4] M. W. Hannan and T. F. Burks, “Current developments in automated
citrus harvesting,” in ASABE Annual International Meeting, 2004.
[5] R. Harrell, D. Slaughter, and P. Adsit, A fruit-tracking system for
robotic harvesting,” Machine Vision and Applications, vol. 2, no. 2,
pp. 69–80, 1989.
[6] R. Harrell, P. Adsit, T. Pool, R. Hoffman et al., “The florida robotic
grove-lab. Transactions of the ASAE, vol. 33, no. 2, pp. 391–399, 1990.
[7] A. Jimenez, R. Ceres, J. Pons et al., “A survey of computer vision
methods for locating fruit on trees,” Transactions of the ASAE-American
Society of Agricultural Engineers, vol. 43, no. 6, pp. 1911–1920, 2000.
[8] F. Pla, F. Juste, and F. Ferri, “Feature extraction of spherical objects in
image analysis: an application to robotic citrus harvesting,” Computers
and Electronics in Agriculture, vol. 8, no. 1, pp. 57–72, 1993.
[9] E. Tutle, “Image controlled robotics in agricultural environments,” 1984.
[10] D. Stajnko, M. Lakota, and M. Hoˇ
cevar, “Estimation of number and
diameter of apple fruits in an orchard during the growing season by
thermal imaging,” Computers and Electronics in Agriculture, vol. 42,
no. 1, pp. 31–42, 2004.
[11] D. Stajnko, J. Rakun, M. Blanke et al., “Modelling apple fruit yield
using image analysis for fruit colour, shape and texture. European
journal of horticultural science, vol. 74, no. 6, pp. 260–267, 2009.
[12] R. Zhou, L. Damerow, Y. Sun, and M. M. Blanke, “Using colour
features of cv. ’gala’ apple fruits in an orchard in image processing
to predict yield,” Precision Agriculture, vol. 13, no. 5, pp. 568–580,
2012.
[13] M. Regunathan and W. S. Lee, “Citrus fruit identification and size
determination using machine vision and ultrasonic sensors,” in ASABE
Annual International Meeting, 2005.
[14] M. Hannan, T. Burks, and D. M. Bulanon, “A machine vision algo-
rithm combining adaptive segmentation and shape analysis for orange
fruit detection,” Agricultural Engineering International: CIGR Journal,
2010.
[15] R. Chinchuluun, W. Lee, R. Ehsani et al., “Machine vision system
for determining citrus count and size on a canopy shake and catch
harvester, Applied Engineering in Agriculture, vol. 25, no. 4, pp. 451–
458, 2009.
[16] J. Billingsley, “More machine vision applications in the NCEA,” in
Mechatronics and Machine Vision in Practice. Springer, 2008, pp.
333–343.
[17] Y. Cakir, M. Kirci, E. O. Gunes, and B. B. Ustundag, “Detection
of oranges in outdoor conditions,” in Agro-Geoinformatics (Agro-
Geoinformatics), 2013 Second International Conference on. IEEE,
2013, pp. 500–503.
[18] P. Annamalai and W. Lee, “Citrus yield mapping system using machine
vision,” in ASAE Annual International Meeting, 2003.
[19] P. Perona and J. Malik, “Scale-space and edge detection using
anisotropic diffusion, Pattern Analysis and Machine Intelligence, IEEE
Transactions on, vol. 12, no. 7, pp. 629–639, 1990.
[20] A. B. Payne, K. B. Walsh, P. Subedi, and D. Jarvis, “Estimation
of mango crop yield using image analysis–segmentation method,”
Computers and Electronics in Agriculture, vol. 91, pp. 57–64, 2013.
[21] D. Fradkin and I. Muchnik, “A study of k-means clustering for im-
proving classification accuracy of multi-class SVM, Technical Report.
Rutgers University, New Brunswick, New Jersey 08854, Tech. Rep.,
2004.
[22] R. C. Gonzalez and R. E. Woods, Digital Image Processing. Prentice
Hall, 2007.
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 7, No. 5, 2016
523 | P a g e
www.ijacsa.thesai.org
... However, remote sensing usually measures canopy vigor or stress indirectly, rather than directly observing grains, fruits, nuts, or vegetables. Canopy occlusion is a primary obstacle that obscures the relationship between visible plant components and measured yield values (Malik et al., 2016;Liu et al., 2020;Wang et al., 2021). Ground-and remotely-sensed yield estimation approaches have their respective strengths and weaknesses, and they offer complementary information. ...
... For lower altitude UAVs and ground-based imagery, DNNs have been primarily applied for detection and counting of fruit, nuts, or vegetables, which are then calibrated/validated against ground-truth yield measurements (Nuske et al., 2014;Anderson et al., 2019). In terms of object detection and counting, DNN models have significantly improved fruit detection and localization, but still face challenges with occluded fruit, which affects their performance in predicting ground yield (Malik et al., 2016;Wang et al., 2021). ...
Article
Crop yield forecasting is essential for informed farm management decisions. However, most yield forecasting models have low spatial resolution, late-season predictions, and lack validation for unseen years or locations. These limitations likely stem from the scarcity of large-scale, high-resolution yield measurement data collected over multiple years, which is uncommon in commercial specialty crop operations. Additionally, these limitations raise concerns about the models’ utility and generalizability under new environmental and management conditions within or across farms. In this study, we develop a spatio-temporal deep learning model to forecast wine grape yield early in the season, utilizing a large dataset with high spatio-temporal resolution (i.e., yield data from million grapevines of eight cultivars observed over four years). The model is trained on weekly 10 m RGB-NIR time-series satellite data from Sentinel 2A-B, fused with categorical and continuous management inputs, including cultivar type, trellis type, row spacing, and canopy spacing. We assess the model’s generalizability by examining its performance on data from unseen years and/or locations and at multiple spatial resolutions. Our results show that combining management data with satellite imagery significantly improves model performance on entirely unseen vineyard blocks at 10 m resolution, achieving an of 0.76, a mean absolute error (MAE) of 4.21 tonnes/hectare, and a mean absolute percent error (MAPE) of 13%. We find that cultivars with considerable year-to-year yield variability tend to exhibit lower predictive performance and may benefit from longer time-series observations for model training to encompass a wide range of environmental and management conditions. We also observe improved estimations from the early season in April to the middle of the growing season in June. In conclusion, the yield forecasting and model validation framework established in this study lays the foundation for training spatio-temporally aware deep learning models on exceptionally large yield datasets with high spatio-temporal resolution. We anticipate that such models will become increasingly prevalent as yield monitors are more frequently deployed in specialty crop operations.
... Dorj et al. [6] developed a watershed algorithm to detect and segment citrus fruits after converting RGB images into an HSV color space, achieving an R2 value of 0.93. Some studies utilized size as a criterion for detecting object boundaries [7][8][9]. Yasar and Akdemir [10] devised an artificial neural network (ANN) method for orange detection by extracting color features from an HSV color space, achieving an accuracy rate of 89.80% on the test set. Another study proposed by Zhao et al. [11] applied the sum of absolute transformed difference (SATD) method to detect fruit pixels of immature green citrus, employing a support vector machine classifier to eliminate false positives based on texture features, resulting in accuracy and recall rates of 0.88 and 0.80, respectively. ...
Article
Full-text available
The global agriculture industry is encountering challenges due to labor shortages and the demand for increased efficiency. Currently, fruit yield estimation in guava orchards primarily depends on manual counting. Machine vision is an essential technology for enabling automatic yield estimation in guava production. To address the detection of guava in complex natural environments, this paper proposes an improved lightweight and efficient detection model, V-YOLO (VanillaNet-YOLO). By utilizing the more lightweight and efficient VanillaNet as the backbone network and modifying the head part of the model, we enhance detection accuracy, reduce the number of model parameters, and improve detection speed. Experimental results demonstrate that V-YOLO and YOLOv10n achieve the same mean average precision (mAP) of 95.0%, but V-YOLO uses only 43.2% of the parameters required by YOLOv10n, performs calculations at 41.4% of the computational cost, and exhibits a detection speed that is 2.67 times that of YOLOv10n. These findings indicate that V-YOLO can be employed for rapid detection and counting of guava, providing an effective method for visually estimating fruit yield in guava orchards.
... Fruit yield can be reflected laterally by the quantity (Teixidoé t al., 2012;Payne et al., 2013). Dorj et al. (2017) and Malik et al. (2016) implemented distinct image thresholding techniques to segregate citrus fruits from the background. This approach was used for detecting and counting citrus fruits in single images. ...
Article
Full-text available
Introduction Green pepper yield estimation is crucial for establishing harvest and storage strategies. Method This paper proposes an automatic counting method for green pepper fruits based on object detection and multi-object tracking algorithm. Green pepper fruits have colors similar to leaves and are often occluded by each other, posing challenges for detection. Based on the YOLOv5s, the CS_YOLOv5s model is specifically designed for green pepper fruit detection. In the CS_YOLOv5s model, a Slim-Nick combined with GSConv structure is utilized in the Neck to reduce model parameters while enhancing detection speed. Additionally, the CBAM attention mechanism is integrated into the Neck to enhance the feature perception of green peppers at various locations and enhance the feature extraction capabilities of the model. Result According to the test results, the CS_YOLOv5s model of mAP, Precision and Recall, and Detection time of a single image are 98.96%, 95%, 97.3%, and 6.3 ms respectively. Compared to the YOLOv5s model, the Detection time of a single image is reduced by 34.4%, while Recall and mAP values are improved. Additionally, for green pepper fruit tracking, this paper combines appearance matching algorithms and track optimization algorithms from SportsTrack to optimize the DeepSort algorithm. Considering three different scenarios of tracking, the MOTA and MOTP are stable, but the ID switch is reduced by 29.41%. Based on the CS_YOLOv5s model, the counting performance before and after DeepSort optimization is compared. For green pepper counting in videos, the optimized DeepSort algorithm achieves ACP (Average Counting Precision), MAE (Mean Absolute Error), and RMSE (Root Mean Squared Error) values of 95.33%, 3.33, and 3.74, respectively. Compared to the original algorithm, ACP increases by 7.2%, while MAE and RMSE decrease by 6.67 and 6.94, respectively. Additionally, Based on the optimized DeepSort, the fruit counting results using YOLOv5s model and CS_YOLOv5s model were compared, and the results show that using the better object detector CS_YOLOv5s has better counting accuracy and robustness.
... Accurate yield prediction is crucial for efficient resource management. DL-based systems have been used to estimate crop yield by counting fruits, flowers, or seeds (Bargoti & Underwood, 2017;Braguy et al., 2021;Chen et al., 2019;Feng et al., 2021;Gan et al., 2018;Malik et al., 2016;Zhou et al., 2021) a task that was previously labor-intensive. These models leverage object detection and image segmentation techniques to provide insights for optimizing harvesting schedules and resource allocation. ...
... This topic is one of the challenging fields where Machine Vision Technology is used in various research fields, including agriculture. As the initial stage in crop yield estimation. Crop yields estimation on plantations, using object recognition and image processing methods for detection and counting the number of fruits [1][2]. ...
... The experimental results of three different datasets have displayed 91.3% accuracy. In addition to this, [11]authors have found a correlation between automated and manual fruit count by getting R² values of up to 0.99. ...
Conference Paper
In this research work, the deep learning CNN approach had been used for the detection of diseases of citrus fruits. For the conduction of this experiment, the deep CNN model was trained by using real-time citrus fruit images that have been collected from the fields of the Punjab province. Before splitting the dataset into training and validation, the dataset was initially augmented and then preprocessed with the help of the MATLAB tool. While conducting this experiment, 80:20 was the ratio in which the dataset had been split for the training and validation proposed. Further, the deep CNN model was trained and tested over the collected dataset. After the successful implementation of the deep CNN model, the highest average accuracy that had been recorded was 98.40%. This increases quality of life and research prospects in the industry. The main reason behind the conduction of this experiment was to increase awareness of the severity of citrus fruit diseases and to develop an efficacious model for the classification of diseases of citrus fruits.
... However, at present, there are not many publications on research, the application of technologies in estimating and counting coconut fruit. Studies often stop at counting the number of fruits on the tree image (Wijethunga et al., 2008;Malik et al., 2016;Chi Cuong et al., 2017) or estimate crop yield (Dorj et al., 2013;Behera et al., 2019). Some experimental systems have relatively high investment costs and are only suitable for large, 55 clearly planned farming areas. ...
Article
Full-text available
In our country today, the counting of dried coconuts at the production facilities is done manually, takes a lot of time and is not accurate. The goal of this study is to build an automatic, fast and accurate coconut counting system. The study was conducted on the peeled dried coconut fruit with a diameter of 15 cm to 20 cm using image processing technology and open-source computer vision library - OpenCV library. The algorithm includes four main steps. First, determine the object and the background using the Otsu segmentation method. Next, estimate the distance between the background and the object to determine the closest area to the center of the object. Then, find the contour, determine the center and area of the object to reduce the noise. The watershed segmentation algorithm is used to separate overlapping and stacking objects. Finally, count the number of objects contained in the image. In the initial experimental results, the counting system has had an accuracy of over 95% with processing time per image about 75 ms and the counting capacity of the system is over 2000 fruits/hour has confirmed the efficiency of the proposed method.
... Recently, several studies have reported flower and fruit detection using automated systems. Automated counting techniques have been developed for several plants, such as apples [3][4][5], citrus [6,7], dragon fruits [8], mangoes [9], peppers [10], and strawberries [11]. Some of these studies employed image processing technologies to detect and count flowers and fruits using color spaces such as hue-saturation-value (HSV) [4], RGB, and YCbCr [8]. ...
Article
Full-text available
Automated crop monitoring using image analysis is commonly used in horticulture. Image-processing technologies have been used in several studies to monitor growth, determine harvest time, and estimate yield. However, accurate monitoring of flowers and fruits in addition to tracking their movements is difficult because of their location on an individual plant among a cluster of plants. In this study, an automated clip-type Internet of Things (IoT) camera-based growth monitoring and harvest date prediction system was proposed and designed for tomato cultivation. Multiple clip-type IoT cameras were installed on trusses inside a greenhouse, and the growth of tomato flowers and fruits was monitored using deep learning-based blooming flower and immature fruit detection. In addition, the harvest date was calculated using these data and temperatures inside the greenhouse. Our system was tested over three months. Harvest dates measured using our system were comparable with the data manually recorded. These results suggest that the system could accurately detect anthesis, number of immature fruits, and predict the harvest date within an error range of ±2.03 days in tomato plants. This system can be used to support crop growth management in greenhouses.
Preprint
Full-text available
Crop yield prediction is essential for agricultural planning but remains challenging due to the complex interactions between weather, climate, and management practices. To address these challenges, we introduce a deep learning-based multi-model called Climate-Management Aware Vision Transformer (CMAViT), designed for pixel-level vineyard yield predictions. CMAViT integrates both spatial and temporal data by leveraging remote sensing imagery and short-term meteorological data, capturing the effects of growing season variations. Additionally, it incorporates management practices, which are represented in text form, using a cross-attention encoder to model their interaction with time-series data. This innovative multi-modal transformer tested on a large dataset from 2016-2019 covering 2,200 hectares and eight grape cultivars including more than 5 million vines, outperforms traditional models like UNet-ConvLSTM, excelling in spatial variability capture and yield prediction, particularly for extreme values in vineyards. CMAViT achieved an R2 of 0.84 and a MAPE of 8.22% on an unseen test dataset. Masking specific modalities lowered performance: excluding management practices, climate data, and both reduced R2 to 0.73, 0.70, and 0.72, respectively, and raised MAPE to 11.92%, 12.66%, and 12.39%, highlighting each modality's importance for accurate yield prediction. Code is available at https://github.com/plant-ai-biophysics-lab/CMAViT.
Article
Full-text available
Accurate fruit counting is one of the significant phenotypic traits for crucial fruit harvesting decision making. Existing approaches perform counting through detection or regression-based approaches. Detection of fruit instances is very challenging because of the very small fruit size compared to the whole size image of a tree. At the same time, regression-based counting techniques contributes impressive results but presents inaccurate results while number of instances increases. Moreover, most approaches lack scalability and are applicable only on one or two fruit types. This paper proposes a fruit counting mechanism that combines loose segmentation and regression counting that works on six fruit types: Apple, Orange, Tomato, Peach, Pomegranate and Almond. Through relaxed segmentation, fruit clusters are segmented to extract the small image regions which contain the small cluster of fruits. Extracted regions are forwarded for the regression counting of fruits. Relaxed segmentation is achieved through a state-of-the-art deconvolutional network, while modified Inception Residual Networks (ResNet) based nonlinear regression module is proposed for fruit counting. For segmentation, 4,820 original images, including corresponding mask images, of all six fruit types are augmented to 32,412 images through different augmentation techniques, while 21,450 extracted patches are augmented to 89,120 images used for the regression module training. The proposed approach attained a counting accuracy of 94.71% for individual fruit types higher than techniques reported in literature.
Article
Full-text available
With the availability of multisensor, multitemporal, multiresolution and multifrequency image data from operational Earth observation satellites the fusion of digital image data has become a valuable tool in remote sensing image evaluation. Digital image fusion is a relatively new research ® eld at the leading edge of available technology. It forms a rapidly developing area of research in remote sensing. This review paper describes and explains mainly pixel based image fusion of Earth observation satellite data as a contribution to multisensor integration oriented data processing.
Book
In contrast with previous books on mechatronics and machine vision in practice, a significant number of chapters focus on systems designed for human interaction and deciphering human motion. Examples illustrate assistive actuation of hip joints, the augmentation of touch sense in artificial hand prostheses and helping stroke survivors in repetitive motion therapy. Interactive mechatronics and the experience of developing machine interfaces has enabled an examination of how we use mechatronics in the service of training, and even to consider why computer games perhaps appear to capture attention so much more readily than a human instructor! Mechatronics continues to be an exciting and developing field. It is now an essential part of our world and living experience. This and the previous books in this series illustrate the journey in developing the use of mechatronics so far. We anticipate that you will find the chapters here an equal source of inspiration for new devices to solve the challenges of new applications, and of course as a resource for teaching and inspiring the new generation of mechatronics engineers.
Article
A mobile grove-lab was developed to study the use of robotic technology for picking oranges under actual production conditions. The design and operation of the citrus picking robot developed for this facility is described. The sensor system developed to identify and locate fruit in real-time and the state network programming technique used to develop a task-level control program for the citrus picking robot are discussed. The suitability of the vision system and state network programming technique for real-time vision-servo robotic picking is demonstrated. It was concluded that the technical and economic practicality of robotic citrus harvesting can only be demonstrated with an operational multiple-arm harvesting system. Multiple usage of robotic harvesting equipment and acquisition of detailed production data by a robotic harvester were identified as intangible benefits of robotic harvesting which should encourage the commercial development of a multiple-arm machine.
Article
Over the last several years there has been a renewed interest in the automation of harvesting of fruits and vegetables. The two major challenges in the automation of harvesting are the recognition of the fruit and its detachment from the tree. In this paper, a machine vision algorithm for the recognition of citrus fruits is presented. The algorithm consists of segmentation, region labeling, size filtering, perimeter extraction and circle detection. Evaluation of the algorithm consisted of images taken inside the canopy (varying lighting condition) and on the canopy surface. Results showed that more than 90% of the fruits were detected in the 110 images tested. In addition, the proposed segmentation was able to deal with varying lighting condition and the circle detection method proved to be effective in detecting fruits in clusters. The development of this algorithm with its capability of detecting fruits in varying lighting condition and occlusion would enhance the overall performance of robotic fruit harvesting.
Article
This work describes a new, computerised vision-based model to estimate the diameter and number of apple fruit on a tree and hence its yield autonomously under natural weather conditions in a fruit orchard. A charge-coupled device (CCD) camera acquired images of cvs 'Golden Delicious' and 'Jonagold' apple (Malus domestica L. Borkh.) trees seven times in the vegeta-tion period, i.e. every two weeks from June to Septem-ber 2007 for modelling the tree volume, fruit diameter and yield at harvest time. Images were processed off-line using image analysis for fruit colour, shape and texture. The fruit detection algorithm was successfully tested on trees bearing from 15 to 42 apple fruits and missed or mis-classified 1 to 3 apple fruits per tree. Fruit detection was sufficiently accurate with an 89 % rate and an overall error rate of 2.2 %. Fruit diameter was underestimated at the beginning of fruit growth in June, but the data corresponded closely (R2 = 0.96) with orchard measurements from July onwards, which enabled accurate modelling of the expected yield per tree. The proposed autonomous model hence has a large potential for forecasting the yield in June/July of harvest in September/October initially of apple in Europe and in the Northern hemisphere.
Conference Paper
This research was conducted to develop an image processing algorithm to identify and count the number of citrus fruits in an image. Once this algorithm is completed it will be incorporated into a machine vision system consisting of a GPS receiver and distance measuring devices in a pick-up truck to estimate yield of a citrus grove on-the-go. A total of 90 images were acquired in an experimental citrus grove. Images of the citrus grove were analyzed and histogram & pixel distribution of various classes (citrus, leaf, and background) were developed. The threshold of segmentation of the images to recognize citrus fruits was estimated from the pixel distribution of hue and saturation color plane. A computer vision algorithm was developed to enhance and extract information from the images. Preprocessing steps for removing noise and identifying properly the number of citrus fruits were carried out using a combination of erosion and dilation. Finally the number of fruits was counted using blob analysis. The total time for processing an image was 283 ms excluding image acquisition time. The algorithm was tested on 59 validation images and the R2 value between the number of fruits counted by the machine vision algorithm and the average number of fruits by manual counting was 0.76.
Conference Paper
The area of intelligent automated citrus harvesting has become a renewed area of research in recent years. The renewed interest is a result of increased economic demand for better solutions for selective automated citrus harvesting than are currently available by purely mechanical harvesters. Throughout this paper the main challenges facing intelligent automated citrus harvesting are addressed: fruit detection and robotic harvesting. The area of fruit detection is discussed, and incorporates the important properties of citrus that can be used for detection. Robotic harvesting is covered, and involves the discussion of the main mechanical design needs as well as the use of visual servoing for the control of the robotic harvester. A description of our proposed intelligent citrus harvesting system as well as our current prototype is presented.
Conference Paper
The aging and the decreasing number of farm workers in Japan have been a potential problem. That is why research on the automation of agricultural operations is conducted in recent years. One of these operations is the harvesting of fruit trees such as the apples. A robotic hand that could harvest the apple fruit similar to the human picker has been developed, however the visual guidance of the developed hand has not yet been made. In this paper, a machine vision system that would guide the robotic harvesting hand was studied. The machine vision system consisted of a digital video camera and a personal computer. Images of a Fuji apple tree were analyzed and histograms of its luminance and color difference of red were developed. The threshold for segmentation of the images to recognize the fruit portion was estimated from the histograms using the optimal thresholding method. The estimated threshold effectively recognized the fruit portion. The threshold calculated from the luminance histogram using the optimal thresholding method was not effective in recognizing the Fuji apple while the threshold selected from the color difference of red histogram was effective in recognizing the Fuji apple.
Conference Paper
In this study, a method is proposed to detect orange fruits on tree using image processing techniques in order to develop software for an auto-orange collector robot system. Fruit detection based on computer vision includes some problems. The main problem is variable lighting conditions for the environment in outdoor. The additional problem is the occlusion of fruits by leaves, branches, and the other fruits. In the proposed method these problems are also examined.