No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Deep learning based 3d reconstruction for phenotyping of wheat seeds: a dataset, challenge, and baseline method
Asexual reproduction results in offspring that are genetically identical to the mother. Among apomictic plants (reproducing asexually through seeds) many require paternal genetic contribution for proper endosperm development (pseudogamous endosperm). We examined phenotypic diversity in seed traits using a diverse panel of sexual and apomictic accessions from the genus Boechera. While genetic uniformity resulting from asexual reproduction is expected to reduce phenotypic diversity in seeds produced by apomictic individuals, pseudogamous endosperm, variable endosperm ploidy, and the deviations from 2:1 maternal:paternal genome ratio in endosperm can all contribute to increased phenotypic diversity among apomictic offspring. We characterized seed size variation in 64 diploid sexual and apomictic (diploid and triploid) Boechera lineages. In order to find out whether individual seed size was related to endosperm ploidy we performed individual seed measurements (projected area and mass) using the phenoSeeder robot system and flow cytometric seed screen. In order to test whether individual seed size had an effect on resulting fitness we performed a controlled growth experiment and recorded seedling life history traits (germination success, germination timing, and root growth rate). Seeds with triploid embryos were 33% larger than those with diploid embryos, but no average size difference was found between sexual and apomictic groups. We identified a maternal effect whereby chloroplast lineage 2 had 30% larger seeds than lineage 3, despite having broad and mostly overlapping geographic ranges. Apomictic seeds were not more uniform in size than sexual seeds, despite genetic uniformity of the maternal gametophyte in the former. Among specific embryo/endosperm ploidy combinations, seeds with tetraploid (automomous) endosperm were on average smaller, and the proportion of such seeds was highest in apomicts. Larger seeds germinated more quickly than small seeds, and lead to higher rates of root growth in young seedlings. Seed mass is under balancing selection in Boechera, and it is an important predictor of several traits, including germination probability and timing, root growth rates, and developmental abnormalities in apomictic accessions.
Attribution License (CC BY 4.0). Storage roots of cassava plants crops are one of the main providers of starch in many South American, African, and Asian countries. Finding varieties with high yields is crucial for growing and breeding. This requires a better understanding of the dynamics of storage root formation, which is usually done by repeated manual evaluation of root types, diameters, and their distribution in excavated roots. We introduce a newly developed method that is capable to analyze the distribution of root diameters automatically, even if root systems display strong variations in root widths and clustering in high numbers. An application study was conducted with cassava roots imaged in a video acquisition box. The root diameter distribution was quantified automatically using an iterative ridge detection approach, which can cope with a wide span of root diameters and clustering. The approach was validated with virtual root models of known geometries and then tested with a time-series of excavated root systems. Based on the retrieved diameter classes, we show plausibly that the dynamics of root type formation can be monitored qualitatively and quantitatively. We conclude that this new method reliably determines important phenotypic traits from storage root crop images. The method is fast and robustly analyses complex root systems and thereby applicable in high-throughput phenotyping and future breeding.
Grapevine is one of the economically most important quality crops. The monitoring of the plant performance during the growth period is, therefore, important to ensure a high quality end-product. This includes the observation, detection, and respective reduction of unhealthy berries (physically damaged, or diseased). At harvest, it is not necessary to know the exact cause of the damage, but rather if the damage is apparent or not. Since a manual screening and selection before harvest is time-consuming and expensive, we propose an automatic, image-based machine learning approach, which can lead observers directly to anomalous areas without the need to monitor every plant manually. Specifically, we train a fully convolutional variational autoencoder with a feature perceptual loss on images with healthy berries only and consider image areas with deviations from this model as damaged berries. We use heatmaps which visualize the results of the trained neural network and, therefore, support the decision making for farmers. We compare our method against a convolutional autoencoder that was successfully applied to a similar task and show that our approach outperforms it.
Seedling establishment is the first stage of crop productivity, and root phenotypes at seed emergence are critical to a successful start of shoot growth as well as for water and nutrient uptake. In this study, we investigate seedling establishment in winter wheat utilizing a newly developed workflow based on magnetic resonance imaging (MRI). Using the eight parents of the MAGIC (multi-parent advanced generation inter-cross) population we analysed the 4D root architecture of 288 individual seedlings grown in natural soils with plant neighbors over 3 d of development. Time of root and shoot emergence, total length, angle, and depth of the axile roots varied significantly among these genotypes. The temporal data resolved rates of elongation of primary roots and first and second seminal root pairs. Genotypes with slowly elongating primary roots had rapidly elongating first and second seminal root pairs and vice versa, resulting in variation in root system architecture mediated not only by root angle but also by initiation and relative elongation of axile roots. We demonstrated that our novel MRI workflow with a unique planting design and automated measurements allowed medium throughput phenotyping of wheat roots in 4D and could give new insights into regulation of root system architecture.
Revealing the contributions of genes to plant phenotype is frequently challenging because loss‐of‐function effects may be subtle or masked by varying degrees of genetic redundancy. Such effects can potentially be detected by measuring plant fitness, which reflects the cumulative effects of genetic changes over the lifetime of a plant. However, fitness is challenging to measure accurately, particularly in species with high fecundity and relatively small propagule sizes such as Arabidopsis thaliana.
An image segmentation‐based method using the software ImageJ and an object detection‐based method using the Faster Region‐based Convolutional Neural Network (R‐CNN) algorithm were used for measuring two Arabidopsis fitness traits: seed and fruit counts.
The segmentation‐based method was error‐prone (correlation between true and predicted seed counts, r² = 0.849) because seeds touching each other were undercounted. By contrast, the object detection‐based algorithm yielded near perfect seed counts (r² = 0.9996) and highly accurate fruit counts (r² = 0.980). Comparing seed counts for wild‐type and 12 mutant lines revealed fitness effects for three genes; fruit counts revealed the same effects for two genes.
Our study provides analysis pipelines and models to facilitate the investigation of Arabidopsis fitness traits and demonstrates the importance of examining fitness traits when studying gene functions.
High-throughput phenotyping involves many samples and diverse trait types. For the goal of automatic measurement and batch data processing, a novel method for high-throughput legume seed phenotyping is proposed. A pipeline of automatic data acquisition and processing, including point cloud acquisition, single-seed extraction, pose normalization, three-dimensional (3D) reconstruction, and trait estimation, is proposed. First, a handheld laser scanner is used to obtain the legume seed point clouds in batches. Second, a combined segmentation method using the RANSAC method, the Euclidean segmentation method, and the dimensionality of the features is proposed to conduct single-seed extraction. Third, a coordinate rotation method based on PCA and the table normal is proposed to conduct pose normalization. Fourth, a fast symmetry-based 3D reconstruction method is built to reconstruct a 3D model of the single seed, and the Poisson surface reconstruction method is used for surface reconstruction. Finally, 34 traits, including 11 morphological traits, 11 scale factors, and 12 shape factors, are automatically calculated. A total of 2500 samples of five kinds of legume seeds are measured. Experimental results show that the average accuracies of scanning and segmentation are 99.52% and 100%, respectively. The overall average reconstruction error is 0.014 mm. The average morphological trait measurement accuracy is submillimeter, and the average relative percentage error is within 3%. The proposed method provides a feasible method of batch data acquisition and processing, which will facilitate the automation in high-throughput legume seed phenotyping.
JURECA is a Pre-Exascale Modular Supercomputer operated by Jülich Supercomputing Centre at Forschungszentrum Jülich. The system combines a flexible Data Centric (DC) module, based on the Atos BullSequana XH2000 with a selection of best-of-its-kind components, and a scalability-focused Booster module, delivered by Intel and Dell Technologies based on the Xeon Phi many-core processor. With its novel architecture, it supports a wide variety of high-performance computing and data analytics workloads.
The Global Wheat Head Detection (GWHD) dataset was created in 2020 and has assembled 193,634 labelled wheat heads from 4700 RGB images acquired from various acquisition platforms and 7 countries/institutions. With an associated competition hosted in Kaggle, GWHD_2020 has successfully attracted attention from both the computer vision and agricultural science communities. From this first experience, a few avenues for improvements have been identified regarding data size, head diversity, and label reliability. To address these issues, the 2020 dataset has been reexamined, relabeled, and complemented by adding 1722 images from 5 additional countries, allowing for 81,553 additional wheat heads. We now release in 2021 a new version of the Global Wheat Head Detection dataset, which is bigger, more diverse, and less noisy than the GWHD_2020 version.
Changes in canopy architecture traits have been shown to contribute to yield increases. Optimizing both light interception and light interception efficiency of agricultural crop canopies will be essential to meeting the growing food needs. Canopy architecture is inherently three-dimensional (3D), but many approaches to measuring canopy architecture component traits treat the canopy as a two-dimensional (2D) structure to make large scale measurement, selective breeding, and gene identification logistically feasible. We develop a high throughput voxel carving strategy to reconstruct 3D representations of sorghum from a small number of RGB photos. Our approach builds on the voxel carving algorithm to allow for fully automatic reconstruction of hundreds of plants. It was employed to generate 3D reconstructions of individual plants within a sorghum association population at the late vegetative stage of development. Light interception parameters estimated from these reconstructions enabled the identification of known and previously unreported loci controlling light interception efficiency in sorghum. The approach is generalizable and scalable, and it enables 3D reconstructions from existing plant high throughput phenotyping datasets. We also propose a set of best practices to increase 3D reconstructions' accuracy.
The phenotypic analysis of root system growth is important to inform efforts to enhance plant resource acquisition from soils. However, root phenotyping still remains challenging due to soil opacity, requiring systems that facilitate root system visibility and image acquisition. Previously reported systems require costly or bespoke materials not available in most countries, where breeders need tools to select varieties best adapted to local soils and field conditions. Here, we report an affordable soil‐based growth (rhizobox) and imaging system to phenotype root development in greenhouses or shelters. All components of the system are made from locally available commodity components, facilitating the adoption of this affordable technology in low‐income countries. The rhizobox is large enough (~6000 cm² visible soil) to not restrict vertical root system growth for most if not all of the life cycle, yet light enough (∼21 kg when filled with soil) for routine handling. Support structures and an imaging station, with five cameras covering the whole soil surface, complement the rhizoboxes. Images are acquired via the Phenotiki sensor interface, collected, stitched and analysed. Root system architecture (RSA) parameters are quantified without intervention. RSA of a dicot (chickpea, Cicer arietinum L.) and a monocot (barley, Hordeum vulgare L.) species, which exhibit contrasting root systems, were analysed. Insights into root system dynamics during vegetative and reproductive stages of the chickpea lifecycle were obtained. This affordable system is relevant for efforts in Ethiopia and other low‐ and middle‐income countries to sustainably enhance crop yields and climate resilience.
Efficient seed germination and establishment are important traits for field and glasshouse crops. Large‐scale germination experiments are laborious and prone to observer errors, leading to the necessity for automated methods. We experimented with five crop species, including tomato, pepper, Brassica, barley, and maize, and concluded an approach for large‐scale germination scoring.
Here, we present the SeedGerm system, which combines cost‐effective hardware and open‐source software for seed germination experiments, automated seed imaging, and machine‐learning based phenotypic analysis. The software can process multiple image series simultaneously and produce reliable analysis of germination‐ and establishment‐related traits, in both comma‐separated values (CSV) and processed images (PNG) formats.
In this article, we describe the hardware and software design in detail. We also demonstrate that SeedGerm could match specialists’ scoring of radicle emergence. Germination curves were produced based on seed‐level germination timing and rates rather than a fitted curve. In particular, by scoring germination across a diverse panel of Brassica napus varieties, SeedGerm implicates a gene important in abscisic acid (ABA) signalling in seeds.
We compared SeedGerm with existing methods and concluded that it could have wide utilities in large‐scale seed phenotyping and testing, for both research and routine seed technology applications.
In order to train the neural network for plant phenotyping, a sufficient amount of training data must be prepared, which requires time-consuming manual data annotation process that often becomes the limiting step. Here, we show that an instance segmentation neural network aimed to phenotype the barley seed morphology of various cultivars, can be sufficiently trained purely by a synthetically generated dataset. Our attempt is based on the concept of domain randomization, where a large amount of image is generated by randomly orienting the seed object to a virtual canvas. The trained model showed 96% recall and 95% average Precision against the real-world test dataset. We show that our approach is effective also for various crops including rice, lettuce, oat, and wheat. Constructing and utilizing such synthetic data can be a powerful method to alleviate human labor costs for deploying deep learning-based analysis in the agricultural domain. Toda et al. train a neural network algorithm for crop seed segmentation using synthetically generated datasets. The model achieves very high precision and is effective for a variety of seeds like barley, rice, and lettuce. Their approach will reduce human labor costs needed to prepare training datasets for similar algorithms for agricultural applications.
The timing and duration of flowering are key agronomic traits that are often associated with the ability of a variety to escape abiotic stress such as heat and drought. Flowering information is valuable in both plant breeding and agricultural production management. Visual assessment, the standard protocol used for phenotyping flowering, is a low-throughput and subjective method. In this study, we evaluated multiple imaging sensors (RGB and multiple multispectral cameras), image resolution (proximal/remote sensing at 1.6 to 30 m above ground level/AGL), and image processing (standard and unsupervised learning) techniques in monitoring flowering intensity of four cool-season crops (canola, camelina, chickpea, and pea) to enhance the accuracy and efficiency in quantifying flowering traits. The features (flower area, percentage of flower area with respect to canopy area) extracted from proximal (1.6–2.2 m AGL) RGB and multispectral (with near infrared, green and blue band) image data were strongly correlated (r up to 0.89) with visual rating scores, especially in pea and canola. The features extracted from unmanned aerial vehicle integrated RGB image data (15–30 m AGL) could also accurately detect and quantify large flowers of winter canola (r up to 0.84), spring canola (r up to 0.72), and pea (r up to 0.72), but not camelina or chickpea flowers. When standard image processing using thresholds and unsupervised machine learning such as k-means clustering were utilized for flower detection and feature extraction, the results were comparable. In general, for applicability of imaging for flower detection, it is recommended that the image data resolution (i.e., ground sampling distance) is at least 2–3 times smaller than that of the flower size. Overall, this study demonstrates the feasibility of utilizing imaging for monitoring flowering intensity in multiple varieties of evaluated crops.
We introduce a novel technique to measure volumes of any shaped objects based on acoustic components. The focus is on small objects with rough surfaces, such as plant seeds. The method allows measurement of object volumes more than 1000 times smaller than the volume of the sensor chamber with both high precision and high accuracy. The method is fast, noninvasive, and easy to produce and use. The measurement principle is supported by theory, describing the behavior of the measured data for objects of known volumes in a range of 1 to 800 µL. In addition to single-frequency, we present frequency-dependent measurements that provide supplementary information about pores on the surface of a measured object, such as the total volume of pores and, in the case of cylindrical pores, their average radius-to-length ratio. We demonstrate the usefulness of the method for seed phenotyping by measuring the volume of irregularly shaped seeds and showing the ability to “look” under the husk and inside pores, which allows us to assess the true density of seeds.
The reconstruction of fruit shape, and particularly, that of strawberry, is needed for a variety of purposes such as the assessing of market class and informing cultivar release documentation. In this work, we focus on the use of fruit shape for phenotyping and selection in breeding programs [6]. We describe a portable, low-cost turntable system for fruit shape estimation in which the fruit rotates on a spindle , as shown in Figure 1. The spindle's rotation speed is controlled by an Arduino board. The design allows the user to alter the number of cameras used for capture, so one or more cameras capture images as the fruit rotates. A complete rotation takes 9 seconds per sample, capturing 56 − 60 frames per camera, per rotation. The total cost for a prototype was $1, 600. Currently, we use one camera and remove the constraint that the angle of rotation be perfectly known [2, 8] by calibrating for the external camera calibration parameters for each sample. Two cubes offset from each other by 45 degrees were three-dimensionally printed and attached to the spindle. On top of each visible cube face, charuco tags, [1, 3] or aruco tags embedded in a chessboard patterns, were added. External camera calibration parameters for each image acquisition timepoint were computed using the method of Tabb and Medeiros 2019 [10]. Figure 1. Left: Strawberry fruit mounted on top of spindle with charuco patterns. Right: Cyan pyramids illustrate the estimated camera positions relative to strawberry; blue region represents reconstructed strawberry. Once the camera calibration parameters are computed, background removal is performed and a shape from inconsistent silhouette method [9] is used for shape estimation. We show the reconstructed camera positions and fruit shape of a strawberry in Figure 1. While initially designed for strawberry, the system is also generalizable to other fruits and will also be demonstrated on at least potato in our poster. Shape from silhouette methods are able to capture convex and saddle regions, but not concavities, so will be limited to situations where concavities are not plentiful. This system computes calibration information while the samples for reconstruction are acquired, in contrast to multi-view stereo systems for plant phenotyping [4, 5, 7].
Volume carving is a well established method for visual hull reconstruction and has been successfully applied in plant phenotyping, especially for 3d reconstruction of small plants and seeds. When imaging larger plants at still relatively high spatial resolution (≤1 mm), well known implementations become slow or have prohibitively large memory needs. Here we present and evaluate a computationally efficient algorithm for volume carving, allowing e.g., 3D reconstruction of plant shoots. It combines a well-known multi-grid representation called “Octree” with an efficient image region integration scheme called “Integral image.” Speedup with respect to less efficient octree implementations is about 2 orders of magnitude, due to the introduced refinement strategy “Mark and refine.” Speedup is about a factor 1.6 compared to a highly optimized GPU implementation using equidistant voxel grids, even without using any parallelization. We demonstrate the application of this method for trait derivation of banana and maize plants.
Generation of 3D data by deep neural network has been attracting increasing attention in the research community. The majority of extant works resort to regular representations such as volumetric grids or collection of images; however, these representations obscure the natural invariance of 3D shapes under geometric transformations and also suffer from a number of other issues. In this paper we address the problem of 3D reconstruction from a single image, generating a straight-forward form of output -- point cloud coordinates. Along with this problem arises a unique and interesting issue, that the groundtruth shape for an input image may be ambiguous. Driven by this unorthodox output form and the inherent ambiguity in groundtruth, we design architecture, loss function and learning paradigm that are novel and effective. Our final solution is a conditional shape sampler, capable of predicting multiple plausible 3D point clouds from an input image. In experiments not only can our system outperform state-of-the-art methods on single image based 3d reconstruction benchmarks; but it also shows a strong performance for 3d shape completion and promising ability in making multiple plausible predictions.
The enormous diversity of seed traits is an intriguing feature and critical for the overwhelming success of higher plants. In particular, seed mass is generally regarded to be key for seedling development but is mostly approximated by using scanning methods delivering only two-dimensional data, often termed seed size. However, three-dimensional traits, such as the volume or mass of single seeds, are very rarely determined in routine measurements. Here, we introduce a device named phenoSeeder, which enables the handling and phenotyping of individual seeds of very different sizes. The system consists of a pick-and-place robot and a modular setup of sensors that can be versatilely extended. Basic biometric traits detected for individual seeds are two-dimensional data from projections, three-dimensional data from volumetric measures, and mass, from which seed density is also calculated. Each seed is tracked by an identifier and, after phenotyping, can be planted, sorted, or individually stored for further evaluation or processing (e.g. in routine seed-to-plant tracking pipelines). By investigating seeds of Arabidopsis (Arabidopsis thaliana), rapeseed (Brassica napus), and barley (Hordeum vulgare), we observed that, even for apparently roundshaped seeds of rapeseed, correlations between the projected area and the mass of seeds were much weaker than between volume and mass. This indicates that simple projections may not deliver good proxies for seed mass. Although throughput is limited, we expect that automated seed phenotyping on a single-seed basis can contribute valuable information for applications in a wide range of wild or crop species, including seed classification, seed sorting, and assessment of seed quality. © 2016 American Society of Plant Biologists. All Rights Reserved.
We describe a method for 3D reconstruction of plant seed surfaces, focusing on small seeds with diameters as small as 200 μm. The method considers robotized systems allowing single seed handling in order to rotate a single seed in front of a camera. Even though such systems feature high position repeatability, at sub-millimeter object scales, camera pose variations have to be compensated. We do this by robustly estimating the tool center point from each acquired image. 3D reconstruction can then be performed by a simple shape-from-silhouette approach. In experiments we investigate runtimes, theoretically achievable accuracy, experimentally achieved accuracy, and show as a proof of principle that the proposed method is well sufficient for 3D seed phenotyping purposes.
Image-based approaches to plant phenotyping are gaining momentum providing fertile ground for several interesting vision tasks where fine-grained categorization is necessary, such as leaf segmentation among a variety of cultivars, and cultivar (or mutant) identification. However, benchmark data focusing on typical imaging situations and vision tasks are still lacking, making it difficult to compare existing methodologies. This paper describes a collection of benchmark datasets of raw and annotated top-view color images of rosette plants. We briefly describe plant material, imaging setup and procedures for different experiments: one with various cultivars of Arabidopsis and one with tobacco undergoing different treatments. We proceed to define a set of computer vision and classification tasks and provide accompanying datasets and annotations based on our raw data. We describe the annotation process performed by experts and discuss appropriate evaluation criteria. We also offer exemplary use cases and results on some tasks obtained with parts of these data. We hope with the release of this rigorous dataset collection to invigorate the development of algorithms in the context of plant phenotyping but also provide new interesting datasets for the general computer vision community to experiment on. Data are publicly available at http://www.plant-phenotyping.org/datasets.
Background
Measuring grain characteristics is an integral component of cereal breeding and research into genetic control of seed development. Measures such as thousand grain weight are fast, but do not give an indication of variation within a sample. Other methods exist for detailed analysis of grain size, but are generally costly and very low throughput. Grain colour analysis is generally difficult to perform with accuracy, and existing methods are expensive and involved.
Results
We have developed a software method to measure grain size and colour from images captured with consumer level flatbed scanners, in a robust, standardised way. The accuracy and precision of the method have been demonstrated through screening wheat and Brachypodium distachyon populations for variation in size and colour.
Conclusion
By using GrainScan, cheap and fast measurement of grain colour and size will enable plant research programs to gain deeper understanding of material, where limited or no information is currently available.
Arabidopsis thaliana is a useful model organism for deciphering the genetic determinants of seed size; however the small size of its seeds makes measurements difficult. Bulk seed weights are often used as an indicator of average seed size, but details of individual seed is obscured. Analysis of seed images is possible but issues arise from variations in seed pigmentation and shadowing making analysis laborious. We therefore investigated the use of a consumer level scanner to facilitate seed size measurements in conjunction with open source image-processing software.
By using the transmitted light from the slide scanning function of a flatbed scanner and particle analysis of the resulting images, we have developed a method for the rapid and high throughput analysis of seed size and seed size distribution. The technical variation due to the approach was negligible enabling us to identify aspects of maternal plant growth that contribute to biological variation in seed size. By controlling for these factors, differences in seed size caused by altered parental genome dosage and mutation were easily detected. The method has high reproducibility and sensitivity, such that a mutant with a 10% reduction in seed size was identified in a screen of endosperm-expressed genes. Our study also generated average seed size data for 91 Arabidopsis accessions and identified a number of quantitative trait loci from two recombinant inbred line populations, generated from Cape Verde Islands and Burren accessions crossed with Columbia.
This study describes a sensitive, high-throughput approach for measuring seed size and seed size distribution. The method provides a low cost and robust solution that can be easily implemented into the workflow of studies relating to various aspects of seed development.
Many algorithms for both identifying and reconstructing a 3-D object are based on the 2-D silhouettes of the object. In general, identifying a nonconvex object using a silhouette-based approach implies neglecting some features of its surface as identification clues. The same features cannot be reconstructed by volume intersection techniques using multiple silhouettes of the object. This paper addresses the problem of finding which parts of a nonconvex object are relevant for silhouette-based image understanding. For this purpose, the geometric concept of visual hull of a 3-D object is introduced. This is the closest approximation of object S that can be obtained with the volume intersection approach; it is the maximal object silhouette-equivalent to S, i.e., which can be substituted for S without affecting any silhouette. Only the parts of the surface of S that also lie on the surface of the visual hull can be reconstructed or identified using silhouette-based algorithms. The visual hull depends not only on the object but also on the region allowed to the viewpoint. Two main viewing regions result in the external and internal visual hull. In the former case the viewing region is related to the convex hull of S, in the latter it is bounded by S. The internal visual hull also admits an interpretation not related to silhouettes. Algorithms for computing visual hulls are presented and their complexity analyzed. In general, the visual hull of a 3-D planar face object turns out to be bounded by planar and curved patches
Use of imaging data has become more profound with the advent of digital cameras, the internet, and automation. With advances of computer vision, actionable information can now be extracted from images to advance the study of plants and their phenotype. In this chapter we document the experience of advancing the state of the art in developing methods that extract such information motivated and orchestrated via data challenges. We identify several examples and conclude that this can be a fruitful bridge between the communities offering problems and data with those that can offer the solutions. We offer suggestions on how to get started and conclude by providing advice for shaping data challenges of the future.
Weeds are mainly spread by weed seeds being mixed with agricultural and forestry crop seeds, grain, animal hair, and other plant products, and disturb the growing environment of target plants such as crops and wild native plants. The accurate and efficient classification of weed seeds is important for the effective management and control of weeds. However, classification remains mainly dependent on destructive sampling-based manual inspection, which has a high cost and rather low flux. We considered that this problem could be solved using a nondestructive intelligent image recognition method. First, on the basis of the establishment of the image acquisition system for weed seeds, images of single weed seeds were rapidly and completely segmented, and a total of 47 696 samples of 140 species of weed seeds and foreign materials remained. Then, six popular and novel deep Convolutional Neural Network (CNN) models are compared to identify the best method for intelligently identifying 140 species of weed seeds. Of these samples, 33 600 samples are randomly selected as the training dataset for model training, and the remaining 14 096 samples are used as the testing dataset for model testing. AlexNet and GoogLeNet emerged from the quantitative evaluation as the best methods. AlexNet has strong classification accuracy and efficiency (low time consumption), and GoogLeNet has the best classification accuracy. A suitable CNN model for weed seed classification could be selected according to specific identification accuracy requirements and time costs of applications. This research is beneficial for developing a detection system for weed seeds in various applications. The resolution of taxonomic issues and problems associated with the identification of these weed seeds may allow for more effective management and control.
Seeds image analysis has become essential to preserve biodiversity. This is why recognition and classification of plant species on the earth’s planet is nowadays a great challenge. The paper focuses on this purpose by studying two plant seeds datasets to classify their families or species through deep learning techniques. SeedNet, a novel CNN has been proposed to face the depicted issue, and several state-of-the-art convolutional neural networks have been exploited for an exhaustive comparison of most adequate for the considered scenario. In detail, promising results in seed classification for both analysed datasets, reaching accuracy values of 95.65% for the first one and 97.47% for the second one, have been obtained. The retrieval problem with the deep learning approach was also addressed, achieving satisfying performances. We consider the obtained results for both the tasks as an excellent starting point to develop a complete seeds recognition, classification and retrieval system to offer impressive support in agriculture and botany fields.
In phenotyping experiments plants are often germinated in high numbers, and in a manual transplantation step selected and moved to single pots. Selection is based on visually derived germination date, visual size, or health inspection. Such values are often inaccurate, as evaluating thousands of tiny seedlings is tiring. We address these issues by quantifying germination detection with an automated, imaging-based device, and by a visual support system for inspection and transplantation. While this is a great help and reduces the need for visual inspection, accuracy of seedling detection is not yet sufficient to allow skipping the inspection step. We therefore present a new dataset and challenge containing 19.5k images taken by our germination detection system and manually verified labels. We describe in detail the involved automated system and handling setup. As baseline we report the performances of the currently applied color-segmentation based algorithm and of five transfer-learned deep neural networks.
Recently, several high-throughput phenotyping facilities have been established that allow for an automated collection of multiple view images of a large number of plants over time. One of the key problems in phenotyping is identifying individual plant organs such as leaves, stems, or roots. We introduced a novel algorithm that uses a 3D segmented plant on its input by using a voxel carving algorithm, and separates the plant into leaves and stems. Our algorithm first uses voxel thinning that generates a first approximation of the plant 3D skeleton. The skeleton is transformed into a mathematical tree by comparing and assessing paths from each leaf or stem tip to the plant root and pruned by using biologically inspired features, fed into a machine learning classifier, leading to a skeleton that corresponds to the input plant. The final skeleton is then used to identify the plant organs and segment voxels. We validated our system on 20 different plants, each represented in a voxel array of a resolution , and the segmentation was executed in under one minute, making our algorithm suitable for the processing of large amounts of plants.
We propose an end-to-end deep learning architecture that produces a 3D shape in triangular mesh from a single color image. Limited by the nature of deep neural network, previous methods usually represent a 3D shape in volume or point cloud, and it is non-trivial to convert them to the more ready-to-use mesh model. Unlike the existing methods, our network represents 3D mesh in a graph-based convolutional neural network and produces correct geometry by progressively deforming an ellipsoid, leveraging perceptual features extracted from the input image. We adopt a coarse-to-fine strategy to make the whole deformation procedure stable, and define various of mesh related losses to capture properties of different levels to guarantee visually appealing and physically accurate 3D geometry. Extensive experiments show that our method not only qualitatively produces mesh model with better details, but also achieves higher 3D shape estimation accuracy compared to the state-of-the-art.
Commercial and scientific plant breeding programs require the phenotyping of large populations. Phenotyping is typically a manual task (costly, time-consuming and sometimes arbitrary). The use of computer vision techniques is a potential solution to some of these specific tasks. In the last years, Deep Learning, and in particular Convolutional Neural Networks (CNNs), have shown a number of advantages over traditional methods in the area. In this work we introduce a computer vision method that estimates the number of seeds into soybean pods, a difficult task that usually requires the intervention of human experts. To this end we developed a classic approach, based on tailored features extraction (FE) followed by a Support Vector Machines (SVM) classification model, and also the referred CNNs. We show how standard CNNs can be easily configured and how a simple method can be used to visualize the key features learned by the model in order to infer the correct class. We processed different seasons batches with both methods obtaining 50.4% (FE + SVM) and 86.2% (CNN) of accuracy in test, highlighting the particularly high increase in generalization capabilities of a deep learning approach over a classic machine vision approach in this task. Dataset and code are publicly available.
We propose an end-to-end deep learning architecture that produces a 3D shape in triangular mesh from a single color image. Limited by the nature of deep neural network, previous methods usually represent a 3D shape in volume or point cloud, and it is non-trivial to convert them to the more ready-to-use mesh model. Unlike the existing methods, our network represents 3D mesh in a graph-based convolutional neural network and produces correct geometry by progressively deforming an ellipsoid, leveraging perceptual features extracted from the input image. We adopt a coarse-to-fine strategy to make the whole deformation procedure stable, and define various of mesh related losses to capture properties of different levels to guarantee visually appealing and physically accurate 3D geometry. Extensive experiments show that our method not only qualitatively produces mesh model with better details, but also achieves higher 3D shape estimation accuracy compared to the state-of-the-art.
Training modern deep learning models requires large amounts of computation, often provided by GPUs. Scaling computation from one GPU to many can enable much faster training and research progress but entails two complications. First, the training library must support inter-GPU communication. Depending on the particular methods employed, this communication may entail anywhere from negligible to significant overhead. Second, the user must modify his or her training code to take advantage of inter-GPU communication. Depending on the training library's API, the modification required may be either significant or minimal. Existing methods for enabling multi-GPU training under the TensorFlow library entail non-negligible communication overhead and require users to heavily modify their model-building code, leading many researchers to avoid the whole mess and stick with slower single-GPU training. In this paper we introduce Horovod, an open source library that improves on both obstructions to scaling: it employs efficient inter-GPU communication via ring reduction and requires only a few lines of modification to user code, enabling faster, easier distributed training in TensorFlow. Horovod is available under the Apache 2.0 license at https://github.com/uber/horovod.
Reproduction of spatial properties of recorded sound scenes is increasingly recognised as a crucial element of all emerging immersive applications, with domestic or cinema-oriented audiovisual reproduction for entertainment, telepresence and immersive teleconferencing, and augmented and virtual reality being key examples. Such applications benefit from a general spatial audio processing framework, being able to exploit spatial information from a variety of recording formats in order to reproduce the original sound scene in a perceptually transparent way.
Directional Audio Coding (DirAC) is a recent parametric spatial sound reproduction method that fulfils many of the requirements of such a framework. It is based on a universal 3D audio format known as B-format and achieves flexible and effective perceptual reproduction for loudspeakers or headphones.
Part of this work focuses on the model of DirAC and aims to extend it. Firstly, it is shown that by taking into account information of the four-channel recording array that generates the B-format signals, it is possible to improve both analysis of the sound scene and reproduction. Secondly, these findings are generalised for various recording configurations.
A further generalisation of DirAC is attempted in a spatial transform domain, the spherical harmonic domain (SHD), with higher-order B-format signals. Formulating the DirAC model in the SHD combines the perceptual effectiveness of DirAC with the increased resolution of higher-order B-format and overcomes most limitations of traditional DirAC.
Some novel applications of parametric processing of spatial sound are demonstrated for sound and music engineering. The first shows the potential of modifying the spatial information in the recording for creative manipulation of sound scenes, while the second shows improvement of music reproduction captured with established surround recording techniques.The effectiveness of parametric techniques in conveying distance and externalisation cues over headphones, led to research in controlling the perceived distance using loudspeakers in a room. This is achieved by manipulating the direct-to-reverberant energy ratio using a compact loudspeaker array with a variable directivity pattern.
Lastly, apart from reproduction of recorded sound scenes, auralisation of the spatial properties of acoustical spaces are of interest. We demonstrate that this problem is well-suited to parametric spatial analysis. The nature of room impulse responses captured with a large microphone array allows very high-resolution approaches, and such approaches for detection and localisation of multiple reflections in a single short observation window are applied and compared.
Inspired by the recent success of methods that employ shape priors to achieve robust 3D reconstructions, we propose a novel recurrent neural network architecture that we call the 3D Recurrent Reconstruction Neural Network (3D-R2N2). The network learns a mapping from images of objects to their underlying 3D shapes from a large collection of synthetic data [13]. Our network takes in one or more images of an object instance from arbitrary viewpoints and outputs a reconstruction of the object in the form of a 3D occupancy grid. Unlike most of the previous works, our network does not require any image annotations or object class labels for training or testing. Our extensive experimental analysis shows that our reconstruction framework (i) outperforms the state-of-the-art methods for single view reconstruction, and (ii) enables the 3D reconstruction of objects in situations when traditional SFM/SLAM methods fail (because of lack of texture and/or wide baseline).
Deeper neural networks are more difficult to train. We present a residual
learning framework to ease the training of networks that are substantially
deeper than those used previously. We explicitly reformulate the layers as
learning residual functions with reference to the layer inputs, instead of
learning unreferenced functions. We provide comprehensive empirical evidence
showing that these residual networks are easier to optimize, and can gain
accuracy from considerably increased depth. On the ImageNet dataset we evaluate
residual nets with a depth of up to 152 layers---8x deeper than VGG nets but
still having lower complexity. An ensemble of these residual nets achieves
3.57% error on the ImageNet test set. This result won the 1st place on the
ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100
and 1000 layers.
The depth of representations is of central importance for many visual
recognition tasks. Solely due to our extremely deep representations, we obtain
a 28% relative improvement on the COCO object detection dataset. Deep residual
nets are foundations of our submissions to ILSVRC & COCO 2015 competitions,
where we also won the 1st places on the tasks of ImageNet detection, ImageNet
localization, COCO detection, and COCO segmentation.
Spherical Fibonacci point sets yield nearly uniform point distributions on the unit sphere S² ⊂ R³. The forward generation of these point sets has been widely researched and is easy to implement, such that they have been used in various applications.
Unfortunately, the lack of an efficient mapping from points on the unit sphere to their closest spherical Fibonacci point set neighbors rendered them impractical for a wide range of applications, especially in computer graphics. Therefore, we introduce an inverse mapping from points on the unit sphere which yields the nearest neighbor in an arbitrarily sized spherical Fibonacci point set in constant time, without requiring any precomputations or table lookups.
We show how to implement this inverse mapping on GPUs while addressing arising floating point precision problems. Further, we demonstrate the use of this mapping and its variants, and show how to apply it to fast unit vector quantization. Finally, we illustrate the means by which to modify this inverse mapping for texture mapping with smooth filter kernels and showcase its use in the field of procedural modeling.
In this work we investigate the effect of the convolutional network depth on
its accuracy in the large-scale image recognition setting. Our main
contribution is a thorough evaluation of networks of increasing depth, which
shows that a significant improvement on the prior-art configurations can be
achieved by pushing the depth to 16-19 weight layers. These findings were the
basis of our ImageNet Challenge 2014 submission, where our team secured the
first and the second places in the localisation and classification tracks
respectively.
With increasing demand to support and accelerate progress in breeding for novel traits, the plant research community faces the need to accurately measure increasingly large numbers of plants and plant parameters. The goal is to provide quantitative analyses of plant structure and function relevant for traits that help plants better adapt to lowinput agriculture and resource-limited environments. We provide an overview of the inherently multidisciplinary research in plant phenotyping, focusing on traits that will assist in selecting genotypes with increased resource use efficiency. We highlight opportunities and challenges for integrating noninvasive or minimally invasive technologies into screening protocols to characterize plant responses to environmental challenges for both controlled and field experimentation. Although technology evolves rapidly, parallel efforts are still required because large-scale phenotyping demands accurate reporting of at least a minimum set of information concerning experimental protocols, data management schemas, and integration with modeling. The journey toward systematic plant phenotyping has only just begun. Expected final online publication date for the Annual Review of Plant Biology Volume 64 is April 29, 2013. Please see http://www.annualreviews.org/catalog/pubdates.aspx for revised estimates.
Occluding contours from an image sequence with view-point specifications determine a bounding volume approximating the object generating the contours. The initial creation and continual refinement of the approximation requires a volumetric representation that facilitates modification yet is descriptive of surface detail. The ``volume segment'' representation presented in this paper is one such representation.
An important computer vision task in robotics is modeling of 3D objects in a robot's workspace. This paper presents a method for generating octree models of 3D solid objects from their silhouettes obtained in a sequence of images. The silhouettes of objects which are projected into an image and the center of projection of the image generate 3D conic volumes. A 3D model of the objects is iteratively constructed by intersecting such conic volumes obtained from a sequence of images. Hierarchical octree structures are used to represent and to process 3D volume data efficiently. The volumes of individual objects are labeled by a connectivity-labeling algorithm, and surface-normal vectors are added to their surface volume elements. The volume of the workspace not yet seen in any image is also included in the model.
Transferring pointnet++ segmentation from virtual to real plants
- A Chaudhury
- P Hanappe
- R Azaïs
- C Godin
- D Colliaux