Technical Report

Automated Analysis of Underwater Imagery: Accomplishments, Products, and Vision A Report on the NOAA Fisheries Strategic Initiative on Automated Image Analysis 2014-2018

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Recent developments in low-cost autonomous underwater vehicles (AUVs), stationary camera arrays, and towed vehicles have made it possible for fishery scientists to begin using optical data streams (e.g. still and video imagery) to generate species-specific, size-structured abundance estimates for different species of marine organisms. Increasingly, NOAA Fisheries and other agencies are employing camera-based surveys to estimate size-structured abundance for key stocks. While there are many benefits to optical surveys, including reduced inter-observer error as well as the ability to audit the observations and generate high sample sizes with reduced personnel and days at sea, the volume of optical data generated quickly exceeds the capabilities of human analysis. Automated image processing methods have been developed and utilized in the human surveillance, biomedical, and defense domains for some time (LeCun et al. 2015; Szeliski 2010) and there are currently many open-source computer vision libraries and packages available on the internet. In the marine science environment, however, computer vision has yet to reach its full potential. Techniques for automated detection, identification, measurement, tracking, and counting fish in underwater optical data streams do exist (Chuang et al. 2014a, 2014b, 2013, 2011; Williams et al. 2016), however, few of these systems are fully automated, with all of the functions required to produce highly successful and accurate results. Marine scientists rarely possess formal programming and development experience. Hence, existing solutions typically exist as one-off, localized applications, specific to particular analysis tasks. As such, they are generally non-transferrable as functional applications with utility across the domain. Consequently, with few exceptions (Huang et al. 2012; Williams et al. 2012; Chuang et al. 2014b; Chuang et al. 2014a; National Research Council 2014; Fisher et al. 2016; and Williams et al. 2016) there has been little operational use of automated analysis within the marine science community. In response to this need, in 2011, the NOAA Fisheries OST initiated a Strategic Initiative on Automated Image Analysis (SI). The mission of this SI was to develop guidelines, set priorities, and fund projects to develop broad-scale, standardized, and efficient automated tools for the analysis of optical data for use in stock assessment. The goal is to create an end-to-end open source software toolkit that allows for the automated analysis of optical data streams and in turn provide fishery-independent abundance estimates for use in stock assessment.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Scallop stock assessment data (abundance, size and age) is usually collected through a combination of fishery independent dredge surveys and fishery dependent surveys of landed catch. Attempts to use in-situ underwater surveys using still or video imagery captured by diver, Remotely Operated Vehicle's (ROV) and benthic sledges have been undertaken but these require manual analysis of the images which is both time consuming and expensive (Richards et al., 2019). ...
Article
King Scallop (Pecten maximus) is the third most valuable species landed by UK fishing vessels. This research assesses the potential to use a Convolutional Neural Network (CNN) detector to identify P. maximus in images of the seabed, recorded using low cost camera technology. A ground truth annotated dataset of images of P. maximus captured in situ was collated. Automatic scallop detectors built into the Video and Image Analytics for Marine Environments (VIAME) toolkit were evaluated on the ground truth dataset. The best performing CNN (NetHarn_1_class) was then trained on the annotated training dataset (90% of the ground truth set) to produce a new detector specifically for P. maximus. The new detector was evaluated on a subset of 208 images (10% of the ground truth set) with the following results: Precision 0.97, Recall 0.95, F1 Score of 0.96, mAP 0.91, with a confidence threshold of 0.5. These results strongly suggest that application of machine learning and optimisation of the low cost imaging approach is merited with a view to expanding stock assessment and scientific survey methods using this non-destructive and more cost-effective approach.
... oach to address the need to streamline large volumes of imagery data collected from underwater fish surveys. NOAA Fisheries and Kitware Computer Vision Inc. worked collaboratively to develop the open source VIAME toolkit for the scientific community to streamline the post-processing of imagery data collected from fish surveys (Dawkins et. al. 2017;Richards et. al. 2019). ...
Article
Full-text available
Global warming is rapidly emerging as a universal threat to ecological integrity and function, highlighting the urgent need for a better understanding of the impact of heat exposure on the resilience of ecosystems and the people who depend on them 1 . Here we show that in the aftermath of the record-breaking marine heatwave on the Great Barrier Reef in 2016 2 , corals began to die immediately on reefs where the accumulated heat exposure exceeded a critical threshold of degree heating weeks, which was 3-4 °C-weeks. After eight months, an exposure of 6 °C-weeks or more drove an unprecedented, regional-scale shift in the composition of coral assemblages, reflecting markedly divergent responses to heat stress by different taxa. Fast-growing staghorn and tabular corals suffered a catastrophic die-off, transforming the three-dimensionality and ecological functioning of 29% of the 3,863 reefs comprising the world's largest coral reef system. Our study bridges the gap between the theory and practice of assessing the risk of ecosystem collapse, under the emerging framework for the International Union for Conservation of Nature (IUCN) Red List of Ecosystems 3 , by rigorously defining both the initial and collapsed states, identifying the major driver of change, and establishing quantitative collapse thresholds. The increasing prevalence of post-bleaching mass mortality of corals represents a radical shift in the disturbance regimes of tropical reefs, both adding to and far exceeding the influence of recurrent cyclones and other local pulse events, presenting a fundamental challenge to the long-term future of these iconic ecosystems.
Article
Full-text available
Combining state-of-the art digital imaging technology with different kinds of marine exploration techniques such as modern AUV (autonomous underwater vehicle), ROV (remote operating vehicle) or other monitoring platforms enables marine imaging on new spatial and/or temporal scales. A comprehensive interpretation of such image collections requires the detection, classification and quantification of objects of interest in the images usually performed by domain experts. However, the data volume and the rich content of the images makes the support by software tools inevitable. We define some requirements for marine image annotation and present our new online tool Biigle 2.0. It is developed with a special focus on annotating benthic fauna in marine image collections with tools customized to increase efficiency and effectiveness in the manual annotation process. The software architecture of the system is described and the special features of Biigle 2.0 are illustrated with different use-cases and future developments are discussed.
Conference Paper
Full-text available
Fish abundance estimation with the aid of visual analysis has drawn increasing attention based on the underwater videos from a remotely-operated vehicle (ROV). We build a novel fish tracking and counting system followed by tracking-by-detection framework. Since fish may keep entering or leaving the field of view (FOV), an offline trained deformable part model (DPM) fish detector is adopted to detect live fish from video data. Besides that, a multiple kernel tracking approach is used to associate the same object across consecutive frames for fish counting purpose. However, due to the diversity of fish poses, the deformation of fish body shape and the color similarity between fish and background, the detection performance greatly decreases, resulting in a large error in tracking and counting. To deal with such issue, we propose a closed-loop mechanism between tracking and detection. First, we arrange detection results into tracklets and extract motion features from arranged tracklets. A Bayesian classifier is then applied to remove unreliable detections. Finally, the tracking results are modified based on the reliable detections. This proposed strategy effectively addresses the false detection problem and largely decreases the tracking error. Favorable performance is achieved by our proposed closed-loop between tracking and detection on the real-world ROV videos.
Article
Full-text available
This paper explores the use of a Bayesian non-parametric topic modeling technique for the purpose of anomaly detection in video data. We present results from two experiments. The first experiment shows that the proposed technique is automatically able characterize the underlying terrain, and detect anomalous flora in image data collected by an underwater robot. The second experiment shows that the same technique can be used on images from a static camera in a dynamic unstructured environment. The second dataset consisting of video data from a static seafloor camera, capturing images of a busy coral reef. The proposed technique was able to detect all three instances of an underwater vehicle passing in front of the camera, amongst many other observations of fishes, debris, lighting changes due to surface waves, and benthic flora.
Article
Full-text available
Global climate change and other anthropogenic stressors have heightened the need to rapidly characterize ecological changes in marine benthic communities across large scales. Digital photography enables rapid collection of survey images to meet this need, but the subsequent image annotation is typically a time consuming, manual task. We investigated the feasibility of using automated point-annotation to expedite cover estimation of the 17 dominant benthic categories from survey-images captured at four Pacific coral reefs. Inter- and intra- annotator variability among six human experts was quantified and compared to semi- and fully- automated annotation methods, which are made available at coralnet.ucsd.edu. Our results indicate high expert agreement for identification of coral genera, but lower agreement for algal functional groups, in particular between turf algae and crustose coralline algae. This indicates the need for unequivocal definitions of algal groups, careful training of multiple annotators, and enhanced imaging technology. Semi-automated annotation, where 50% of the annotation decisions were performed automatically, yielded cover estimate errors comparable to those of the human experts. Furthermore, fully-automated annotation yielded rapid, unbiased cover estimates but with increased variance. These results show that automated annotation can increase spatial coverage and decrease time and financial outlay for image-based reef surveys.
Article
Full-text available
Deep learning allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction. These methods have dramatically improved the state-of-the-art in speech recognition, visual object recognition, object detection and many other domains such as drug discovery and genomics. Deep learning discovers intricate structure in large data sets by using the backpropagation algorithm to indicate how a machine should change its internal parameters that are used to compute the representation in each layer from the representation in the previous layer. Deep convolutional nets have brought about breakthroughs in processing images, video, speech and audio, whereas recurrent nets have shone light on sequential data such as text and speech.
Conference Paper
Full-text available
Automated fish species identification in open aquatic habitats based on video analytics is the primary area of research in camera-based fisheries surveys. Finding informative features for these analyses, however, is fundamentally challenging due to poor quality of underwater imagery and strong visual similarity among species. In this paper, we compare two different fish feature extraction methods, namely the supervised and unsupervised approaches, which are then applied to a hierarchical partial classification framework. Several specified anatomical parts of fish are automatically located to generate the supervised feature descriptors. For unsupervised feature extraction, a scale-invariant object part learning algorithm is proposed to discover common shape of body parts and then extract appearance, location and size information of each part. Experiments show that the unsupervised approach achieves better recognition performance on live fish images collected by trawl-based cameras.
Conference Paper
Full-text available
Live fish recognition in open aquatic habitats suffers from the high uncertainty in many of the data. To alleviate this problem without discarding those data, the system should learn a species hierarchy so that high-level labels can be assigned to ambiguous data. In this paper, a systematic hierarchical partial classification algorithm is therefore proposed for underwater fish species recognition. Partial classification is applied at each level of the species hierarchy so that the coarse-to-fine categorization stops once the decision confidence is low. By defining the exponential benefit function, we formulate the selection of decision threshold as an optimization problem. Also, attributes from important fish anatomical parts are focused to generate discriminative feature descriptors. Experiments show that the proposed method achieves an accuracy up to 94%, with partial decision rate less than 5%, on underwater fish images with high uncertainty and class imbalance.
Article
Full-text available
Capture-recapture methods are frequently employed to estimate abundance of cetaceans using photographic techniques and a variety of statistical models. However, there are many unresolved issues regarding the selection and manipulation of images that can potentially impose bias on resulting estimates. To examine the potential impact of these issues we circulated a test data set of dorsal fin images from bottlenose dolphins to several independent research groups. Photo-identification methods were generally similar, but the selection, scoring, and matching of images varied greatly amongst groups. Based on these results we make the following recommendations. Researchers should: (1) determine the degree of marking, or level of distinctiveness, and use images of sufficient quality to recognize animals of that level of distinctiveness; (2) ensure that markings are sufficiently distinct to eliminate the potential for “twins” to occur; (3) stratify data sets by distinctiveness and generate a series of abundance estimates to investigate the influence of including animals of varying degrees of markings; and (4) strive to examine and incorporate variability among analysts into capture-recapture estimation. In this paper we summarize these potential sources of bias and provide recommendations for best practices for using natural markings in a capture-recapture framework.
Conference Paper
Full-text available
Live fish recognition in the open sea is a challenging multi-class classification task. We propose a novel method to recognize fish in an unrestricted natural environment recorded by underwater cameras. This method extracts 66 types of features, which are a combination of color, shape and texture properties from different parts of the fish and reduce the feature dimensions with forward sequential feature selection (FSFS) procedure. The selected features of the FSFS are used by an SVM. We present a Balance-Guaranteed Optimized Tree (BGOT) to control the error accumulation in hierarchical classification and, therefore, achieve better performance. A BGOT of 10 fish species is automatically constructed using the inter-class similarities and a heuristic method. The proposed BGOT-based hierarchical classification method achieves about 4% better accuracy compared to state-of-the-art techniques on a live fish image dataset.
Conference Paper
Full-text available
Non-extractive fish abundance estimation with the aid of visual analysis has drawn increasing attention. Low frame rate and variable illumination in the underwater environment, however, makes conventional tracking methods unreliable. In this paper, a robust multiple fish tracking system for low-frame-rate underwater stereo cameras is proposed. With the result of fish segmentation, a computationally efficient block-matching method is applied to perform successful stereo matching. A multiple-feature matching cost function is utilized to give a simple but effective metric for finding the temporal match of each target. Built upon reliable stereo matching, a multiple-target tracking algorithm via the Viterbi data association is developed to overcome the poor motion continuity of targets. Experimental results show that an accurate underwater live fish tracking result with stereo cameras is achieved.
Article
Full-text available
Specular reflection of solar radiation on non-flat water surfaces is a serious confounding factor for benthic remote sensing in shallow-water environments. This problem was recently overcome by Hochberg et al., who provided an effective method for the removal of 'sun glint' from remotely sensed images by utilization of the brightness in a near-infrared (NIR) band. Application of the technique was shown to give an increase in the accuracy of benthic habitat classification. However, as presented, the method is sensitive to outlier pixels, requires a time-consuming masking of land and cloud, and is not formulated in a manner leading to ease of implementation. We present a revised version of the method, which is more robust, does not require masking and can be implemented very simply. The practical approach described here will hopefully expedite the routine adoption of this effective and simple technique throughout the aquatic remote sensing community.
Article
Full-text available
Reproductive data for 2 pelagic dolphin subspecies in the eastern tropical Pacific, the eastern spinner (ES) dolphin Stenella longirostris orientalis and northeastern pantropical spotted (NEPS) dolphin S. attenuata attenuata, were obtained from aerial photographs taken between 1987 and 2003. Two measures of reproductive output were estimated: proportion of adult dolphins with calves ('proportion with calves') and length at which calves disassociated from their mothers ('length at disassociation'). Trends in length at disassociation were investigated to determine if the proportion with calves was affected by possible changes in calving interval, but no changes were found. Propor- tion with calves for ES dolphins was stable from 1987 to 1993, then declined from 1993 to 2003; pro- portion with calves for NEPS dolphins decreased steadily from 1987 to 2003. For both species, propor- tion with calves was related to number of dolphins in the school of the focal species and/or proportion of the school made up of the focal species. For NEPS dolphins, annual number of purse-seine sets on dolphins was a predictor of both proportion with calves and length at disassociation. Because NEPS dolphins are the main species targeted by the fishery, the link between fishing activity and both mea- sures of reproductive output indicates that the fishery has population-level effects beyond reported direct kill. Decline in reproductive output is the proximate cause or one of the proximate causes of the failure of dolphin populations to recover at rates expected after reduction of high bycatch levels.
Article
Full-text available
Abstract Marine ecosystems are facing increasing exposure to a range of stressors and declines in critical ecological functions. The likelihood of further loss of functions and resilience is dependent, in part, on the extent of functional redundancy (i.e. the capacity of one species to functionally compensate for the loss of another species) within critical functional groups. We used multiple metrics; species richness, generic richness, abundance and reserve capacity (i.e. the relative number of individuals available to fulfil the function if the numerically dominant species is lost), as indicators to assess the potential functional redundancy of four functional groups of herbivorous fishes (browsers, excavators, grazers and scrapers) in two of the worlds’ most intact coral reef ecosystems: the Great Barrier Reef (GBR) and Ningaloo Reef in Western Australia. We found marked variations in potential redundancy among habitats within each reef system and functional groups. Despite negligible fishing of herbivorous fishes, coastal habitats in both reef systems had lower functional redundancy compared to offshore locations for all herbivorous fishes collectively and the four functional groups independently. This pattern was consistent in all four indicators of redundancy. The potential vulnerability of these coastal habitats is highlighted by recent shifts from coral to macroalgal dominance on several coastal reefs of the GBR. Our approach provides a simple yet revealing evaluation of potential functional redundancy. Moreover, it highlights the spatial variation in potential vulnerability and resilience of reef systems.
Article
Full-text available
The world's coral reefs are being degraded, and the need to reduce local pressures to offset the effects of increasing global pressures is now widely recognized. This study investigates the spatial and temporal dynamics of coral cover, identifies the main drivers of coral mortality, and quantifies the rates of potential recovery of the Great Barrier Reef. Based on the world's most extensive time series data on reef condition (2,258 surveys of 214 reefs over 1985-2012), we show a major decline in coral cover from 28.0% to 13.8% (0.53% y(-1)), a loss of 50.7% of initial coral cover. Tropical cyclones, coral predation by crown-of-thorns starfish (COTS), and coral bleaching accounted for 48%, 42%, and 10% of the respective estimated losses, amounting to 3.38% y(-1) mortality rate. Importantly, the relatively pristine northern region showed no overall decline. The estimated rate of increase in coral cover in the absence of cyclones, COTS, and bleaching was 2.85% y(-1), demonstrating substantial capacity for recovery of reefs. In the absence of COTS, coral cover would increase at 0.89% y(-1), despite ongoing losses due to cyclones and bleaching. Thus, reducing COTS populations, by improving water quality and developing alternative control measures, could prevent further coral decline and improve the outlook for the Great Barrier Reef. Such strategies can, however, only be successful if climatic conditions are stabilized, as losses due to bleaching and cyclones will otherwise increase.
Conference Paper
Full-text available
This paper describes an automatic segmentation algorithm for fish sampled by a trawl-based underwater camera system. To overcome the problem caused by very low brightness contrast between fish and their underwater background with dynamically changing luminance, our proposed algorithm adopts an innovative histogram backprojection procedure on double local-thresholded images to ensure a reliable segmentation on the fish shape boundaries. The thresholded results are further validated by area and variance criteria to remove unwanted objects. Finally, a post-processing step is applied to refine the segmentation. Promising results, as validated by expert-generated ground truth data, were obtained via our proposed algorithm.
Article
Full-text available
Sun glint, the specular reflection of light from water surfaces, is a serious confounding factor for remote sensing of water column properties and benthos. This paper reviews current techniques to estimate and remove the glint radiance component from imagery. Methods for processing of ocean color images use statistical sea surface models to predict the glint from the sun and sensor positions and wind data. Methods for higher resolution imaging, used in coastal and shallow water mapping, estimate the glint radiance from the near-infrared signal. The effects of some current methods are demonstrated and possibilities for future techniques are briefly addressed.
Technical Report
NOAA generates tens of terabytes of data a day, also known as “big data” from satellites, radars, ships, weather models, optical technologies, and other sources. This unprecedented growth of data collection in recent years has resulted from enhanced sampling technologies and faster computer processing. While these data are publicly available, there is not yet sufficient access to the data by next generation processing technologies, such as machine learning (ML) algorithms that are able to improve processing efficiencies. Accessibility is the key component for utilizing analytical tools and ensuring our processing meets 21st century data needs. This report focuses on the challenges of accessibility of imagery (defined as still images and video) from the marine environment. Vast amounts of imagery are collected from optical technologies used in marine ecosystem monitoring and ocean observation programs. While technologies have dramatically increased the spatial and temporal resolution of data and increased our understanding of marine ecosystems, the drastic increase in big data, specifically imagery, presents numerous challenges. Case studies discussed in this report highlight that big data imagery are readily being collected and stored, yet the foundation for the long term storage and accessibility of big data must be based on the necessary guidance for its architecture, infrastructure, and applications to enhance the accessibility and use of these data to help fulfill NOAA’s cross-functional missions. Additionally, the report highlights key considerations and recommendations for NOAA’s data modernization efforts that align with mandates such as Public Access to Research Results, the Evidence-Based Policy Making Act, Department of Commerce Strategic Plan, the President’s Management Agenda, and White House Executive Order on Artificial Intelligence (AI). As big data and analytical tools become more commonplace for NOAA’s research and scientific operations, there is an increasing need to create end-to-end data management practices that improve data accessibility for analytical tools that utilize AI, computer vision (AI applied to the visual world), and ML. The development and application of AI and ML analytics will progress as long as there is accessibility of big data with enriched metadata; however, accessibility appears to be the primary challenge to fully utilize ML analytics. Rapid, optimal access to entire imagery and data collections is critical to create annotated imagery libraries for supervised analysis using ML algorithms. This report highlights the common need to implement accessibility solutions to facilitate efficient imagery processing using available analytical tools. Other critical requirements to enable AI include the necessary metadata for discovery, long term data archive and access, and economical multi-tier storage. As big data imagery are made more readily available to open source tools such as ML analytics, significant cost reductions in data processing will be realized by reducing the laborintensive efforts currently needed. ML tools accelerate processing of imagery with automated detection and classification resulting in more timely and precise scientific products for management decisions. Furthermore, as the broader scientific community expands its research and discovery from increased accessibility of big data imagery, the ML applications will increase the number of insightful science-based products beyond the scope of the original operational objectives, thereby increasing the value of the agency’s scientific products.
Article
We present some updates to YOLO! We made a bunch of little design changes to make it better. We also trained this new network that's pretty swell. It's a little bigger than last time but more accurate. It's still fast though, don't worry. At 320x320 YOLOv3 runs in 22 ms at 28.2 mAP, as accurate as SSD but three times faster. When we look at the old .5 IOU mAP detection metric YOLOv3 is quite good. It achieves 57.9 mAP@50 in 51 ms on a Titan X, compared to 57.5 mAP@50 in 198 ms by RetinaNet, similar performance but 3.8x faster. As always, all the code is online at https://pjreddie.com/yolo/
Article
The cost of reproduction is a key parameter determining a species' life history strategy. Despite exhibiting some of the fastest offspring growth rates among mammals, the cost of reproduction in baleen whales is largely unknown since standard field metabolic techniques cannot be applied. We quantified the cost of reproduction for southern right whales Eubalaena australis over a 3 mo breeding season. We did this by determining the relationship between calf growth rate and maternal rate of loss in energy reserves, using repeated measurements of body volume obtained from un manned aerial vehicle photogrammetry. We recorded 1118 body volume estimates from 40 female and calf pairs over 40 to 89 d. Calves grew at a rate of 3.2 cm d⁻¹ (SD = 0.45) in body length and 0.081 m³ d⁻¹ (SD = 0.011) in body volume, while females decreased in volume at a rate of 0.126 m³ d⁻¹ (SD = 0.036). The average volume conversion efficiency from female to calf was 68% (SD = 16.91). Calf growth rate was positively related to the rate of loss in maternal body volume, suggesting that maternal volume loss is proportional to the energy investment into her calf. Maternal in vestment was determined by her body size and condition, with longer and more rotund females investing more volume into their calves compared to shorter and leaner females. Lactating females lost on average 25% of their initial body volume over the 3 mo breeding season. This study demonstrates the considerable energetic cost that females face during the lactation period, and highlights the importance of sufficientmaternal energyreserves for reproduction in this capital breeding species.
Book
This book gives a start-to-finish overview of the whole Fish4Knowledge project, in 18 short chapters, each describing one aspect of the project. The Fish4Knowledge project explored the possibilities of big video data, in this case from undersea video. Recording and analyzing 90 thousand hours of video from ten camera locations, the project gives a 3 year view of fish abundance in several tropical coral reefs off the coast of Taiwan. The research system built a remote recording network, over 100 Tb of storage, supercomputer processing, video target detection and tracking, fish species recognition and analysis, a large SQL database to record the results and an efficient retrieval mechanism. Novel user interface mechanisms were developed to provide easy access for marine ecologists, who wanted to explore the dataset. The book is a useful resource for system builders, as it gives an overview of the many new methods that were created to build the Fish4Knowledge system in a manner that also allows readers to see how all the components fit together.
Conference Paper
We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 dif- ferent classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implemen- tation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called dropout that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry
Article
Underwater cameras increasingly are being used on remotely operated, autonomous, or towed vehicles to provide fishery-independent survey data in areas unsuitable for bottom trawls. To observe and quantify avoidance and attraction behaviors of fish to these vehicles, we developed an observational test bed consisting of 3 benthic stereo cameras, set in a straight line, on a coral reef in the Gulf of Mexico. During one pass of a towed camera vehicle, one of the benthic cameras viewed a school of vermilion snapper (Rhomboplites aurorubens) that exhibited a variety of avoidance behaviors. Stereo analysis was used to position some these fish, and target tracking was used to estimate their swimming performance and schooling characteristics for each second from the time the research vessel had passed the benthic cameras to the time of arrival of the towed underwater vehicle. The fish showed little reaction to the tow vessel but responded to the tow cable by swimming laterally and downward, then rapidly increased their swimming speed and avoidance behavior when the towed vehicle came into view. The use of observational test beds, stereo photography, and target tracking allows quantification of the avoidance response and provides a means to determine which stimuli produced by the sampling process elicit fish avoidance behaviors.
Article
Optical methods for surveying populations are becoming increasingly popular. These methods often produce hundreds of thousands to millions of images, making it impractical to analyze all the images manually by human annotators. Computer vision software can rapidly annotate these images, but their error rates are often substantial, vary spatially and are autocorrelated. Hence, population estimates based on the raw computer automated counts can be seriously biased. We evaluated four estimators that combine automated annotations of all the images with manual annotations from a random sample to obtain (approximately) unbiased population estimates, namely: ratio, offset, and linear regression estimators as well as the mean of the manual annotations only. Each of these estimators was applied either globally or locally (i.e., either all data were used or only those near the point in question, to take into account spatial variability and autocorrelation in error rates). We also investigated a simple stratification scheme that splits the images into two strata, based on whether the automated annotator detected no targets or at least one target. The 16 methods resulting from a combination of four estimators, global or local estimation, and one stratum or two strata, were evaluated using simulations and field data. Our results indicated that the probability of a false negative is the key factor determining the best method, regardless of the probability of false positives. Stratification was the most effective method in improving the accuracy and precision of the estimates, provided the false negative rate was not too high. If the probability of false negatives is low, stratified estimation with the local ratio estimator or local regression (essentially geographically weighted regression) is best. If the probability of false negatives is high, no stratification with a simple global linear regression or simply the manual sample mean alone is recommended.
Article
We present a method to automatically measure fish from images taken using a stereo-camera system installed in a large trawl (CamTrawl). Different visibility and fish density conditions were evaluated to establish accuracy and precision of image-based length estimates when compared with physical length measurements. The automated image-based length estimates compared well with the trawl catch values and were comparable with manual image processing in good visibility conditions. Greatest agreement with trawl catch occurred when fish were within of fully lateral presentation to the cameras, and within 150 cm of the cameras. High turbidity caused substantial over- and underestimates of length composition, and a greater number of incompletely extracted fish outlines. Multiple estimates of individual fish lengths showed a mean coefficient of variation (CV) of 3% in good visibility conditions. The agreement between manual and automated fish measurement estimates was not correlated with fish length or range from the camera ( ). Implementation of these methods can result in a large increase in survey efficiency, given the effort required to process the trawl catch.
Article
Researchers at NOAA's National Marine Fisheries Service's (NFMS) Alaska Fisheries Science Center (AFSC) have developed the Cam-trawl. The Cam-trawl is a self-contained stereo-camera system fitted to the aft end of a trawl in place of the cod-end. The absence of the cod-end allows animals to return unharmed to the environment after being imaged, and the image data provide much of the information that is typically collected from animals that are retained by traditional trawl methods. The stereo-camera system consists of two high-resolution machine vision cameras, a series of light-emitting diode (LED) strobes, a computer, a microcontroller, sensors and a battery power supply. The system uses two JAI Inc. (San Jose, California) RM-4200GE high-resolution, high-sensitivity cameras capable of capturing four-megapixel images at up to 15 frames per second. The Cam-trawl was tested over a series of experimental deployments using NOAA ships Oscar Dyson and Bell Shimada in July and August 2010.
Article
State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully-convolutional network that simultaneously predicts object bounds and objectness scores at each position. RPNs are trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. With a simple alternating optimization, RPN and Fast R-CNN can be trained to share convolutional features. For the very deep VGG-16 model, our detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007 (73.2% mAP) and 2012 (70.4% mAP) using 300 proposals per image. The code will be released.
Article
Photographic quadrat sampling is commonly used for the study of sessile benthic communities. However, photoquadrat data handling is fragmented into several processing methods, and there is a scarcity of dedicated software tools that integrate all major analysis options. photoQuad is a new software system for advanced image processing of photographic samples, dedicated to ecological applications. The software integrates a series of methods for the extraction of species area, percentage coverage, or presence/absence information, including random point counts (RP), grid cell counts (CL), freehand regions (FH), and image segmentation-based regions (SG). These are simultaneously functional in a layer-based environment, further supported by a variety of tools for image enhancement, image calibration, automatic quadrat boundary detection, and management of user-specific species libraries. The paper documents the main features of photoQuad, and demonstrates its performance through the simultaneous application of the RP, CL, FH, and SG methods on identical datasets, and the comparison of errors in species area and coverage measurements. The simulated data used for reference are disk-shaped patches, whose area and density statistics are equivalent to three benthic species characteristic of Mediterranean coralligenous communities. The analysis indicated that measurement methods differed in area and coverage bias, as well as in their sensitivity to species size. Large patches were accurately measured by all methods in terms of mean scaled error, but CL and RP provided high error variance, and their performance deteriorated with decreasing patch size; the region-based SG and FH methods provided the lowest errors and were both robust to patch size. The image and quadrat calibration process showed no statistically significant effect on the outputs, although further analysis is needed to validate this result. Overall, photoQuad constitutes a powerful software for elaborate analysis of photoquadrat images, facilitating fast and comparable evaluation of the ecological information contained therein. The photoQuad software is freely available to download and use from: http://www.mar.aegean.gr/sonarlab/photoquad.
Article
Photographic identification is an increasingly important tool in population assessment (e.g., Mizroch et al. 1990, Whitehead 1990), behavioral research (Kaufman et al. 1990, Morton 1990, Shane and McSweeney 1990, Whitehead 1997), and foraging studies (Katona and Beard 1990). With the advent of inexpensive high-resolution digital cameras and powerful image processing software, digital imagery has largely supplanted traditional emulsion film for most photo-based research. Because digital imaging is relatively new to wildlife research, many biologists are not well acquainted with the options available for managing their images. The ease with which digital images can be acquired enables huge image databases to be assembled in a fairly short time: the accumulation of thousands of images in a single field season is not uncommon. A challenge is to properly catalog and manage those images to fully exploit their many advantages. Manually reviewing large image libraries to compare records and matching new images to those stored in the image library is cumbersome and inefficient. When the features used to distinguish individuals are well defined or morphologically localized, it is possible to develop custom algorithms and dedicated software to automate the labor-intensive tasks of inspecting and matching images. These computer-assisted methods typically rely on metrical analysis of distinct features. Examples include measuring the dorsal ratios of fin notches or other irregularities on the dorsal fin of dolphins (Kreho et al. 1999, Araabi et ul. 2000, Hillman et al. 2003, Markowitz et al. 2003), analyzing the pattern of marks on the edges of sperm whale flukes (Physeter macrocephahs; Whitehead 1990), or analyzing coloration patterns on the heads of gray seals (Halichoerzls grypzls; Hilby and Love11 1990). Photo-identification may also utilize non-metrical features visible on the animal, such as scars and general pelage patterns (e.g., Yochem et al. 1990, Forcada and Aguilar 2000).
Article
Photographic and video methods are frequently used to increase the efficiency of coral reef monitoring efforts. The random point count method is commonly used on still images or frame-grabbed video to estimate the community statistics of benthos. A matrix of randomly distributed points is overlaid on an image, and the species or substrate-type lying beneath each point is visually identified. Coral Point Count with Excel extensions (CPCe) is a standalone Visual Basic program which automates, facilitates, and speeds the random point count analysis process. CPCe includes automatic frame-image sequencing, single-click species/substrate labeling, auto-advancement of data point focus, zoom in/out, zoom hold, and specification of random point number, distribution type, and frame border location. Customization options include user-specified coral/substrate codes and data point shape, size, and color. CPCe can also perform image calibration and planar area and length calculation of benthic features. The ability to automatically generate analysis spreadsheets in Microsoft Excel based upon the supplied species/substrate codes is a significant feature. Data from individual frames can be combined to produce both inter- and intra-site comparisons. Spreadsheet contents include header information, statistical parameters of each species/substrate type (relative abundance, mean, standard deviation, standard error) and the calculation of the Shannon–Weaver diversity index for each species. Additional information can be found at http://www.nova.edu/ocean/cpce/.
Conference Paper
We consider the automated recognition of human actions in surveillance videos. Most current methods build classifiers based on complex handcrafted features computed from the raw inputs. Convolutional neural networks (CNNs) are a type of deep model that can act directly on the raw inputs. However, such models are currently limited to handling 2D inputs. In this paper, we develop a novel 3D CNN model for action recognition. This model extracts features from both the spatial and the temporal dimensions by performing 3D convolutions, thereby capturing the motion information encoded in multiple adjacent frames. The developed model generates multiple channels of information from the input frames, and the final feature representation combines information from all channels. To further boost the performance, we propose regularizing the outputs with high-level features and combining the predictions of a variety of different models. We apply the developed models to recognize human actions in the real-world environment of airport surveillance videos, and they achieve superior performance in comparison to baseline methods.
Video Annotation and Reference System
  • N Barr
Barr N. 2015. Video Annotation and Reference System [WWW Document].
Integration of Technologies for Next Generation Marine Observation Systems and Fisheries Independent Surveys I -Symposium
  • R Callouet
  • M D Campbell
  • W Michaels
  • S A Murawski
  • T S Switzer
Callouet R, Campbell MD, Michaels W, Murawski SA, Switzer TS. 2017. Integration of Technologies for Next Generation Marine Observation Systems and Fisheries Independent Surveys I -Symposium. Retrieved September 28, 2018, from https://afs.confex.com/afs/2017/meetingapp.cgi
KWIVER: An open source cross-platform video exploitation framework
  • K Fieldhouse
  • M J Leotta
  • A Basharat
  • R Blue
  • D Stoup
  • C Atkins
  • L Sherrill
  • B Boeckel
  • P Tunison
  • J Becker
Fieldhouse K, Leotta MJ, Basharat A, Blue R, Stoup D, Atkins C, Sherrill L, Boeckel B, Tunison P, Becker J, et al. 2014. KWIVER: An open source cross-platform video exploitation framework. In: 2014 IEEE Applied Imagery Pattern Recognition Workshop (AIPR).
Estimation of skate abundance using a towed camera survey and automated detectors
  • D Hart
  • J-H Chang
  • M Dawkins
  • A Hoogs
  • M Bergman
Hart D, Chang J-H, Dawkins M, Hoogs A, Bergman M. 2018. Estimation of skate abundance using a towed camera survey and automated detectors. Presented at the 2018 Annual American Fisheries Society Meeting, Atlantic City, NJ.
Results of the acoustic-trawl survey of walleye pollock (Theragra chalcogramma) on the US and Russian Bering sea shelf in
  • T Honkalehto
  • A L Mccarthy
  • P H Ressler
  • K Williams
  • D Jones
Honkalehto T, McCarthy AL, Ressler PH, Williams K, and Jones D. 2012. Results of the acoustic-trawl survey of walleye pollock (Theragra chalcogramma) on the US and Russian Bering sea shelf in June-August 2010 (DY1006).
Assessment of the walleye pollock stock in the Eastern Bering Sea. Stock assessment and fishery evaluation report for the groundfish resources of the Bering Sea/Aleutian Islands regions
  • J N Ianelli
  • S Barbeaux
  • T Honkalehto
  • S Kotwicki
  • K Aydin
  • N Williamson
Ianelli JN, Barbeaux S, Honkalehto T, Kotwicki S, Aydin K, Williamson N. 2009. Assessment of the walleye pollock stock in the Eastern Bering Sea. Stock assessment and fishery evaluation report for the groundfish resources of the Bering Sea/Aleutian Islands regions. North Pac. Fish. Mgmt. Council, Anchorage, AK, section, 1, pp.49-148.
Workshop and Challenge: Automated Analysis of Marine Video for Environmental Monitoring
  • Kitware Inc
Kitware Inc. 2018. CVPR 2018 Workshop and Challenge: Automated Analysis of Marine Video for Environmental Monitoring. [accessed 2018 Sep 27].
Report of the National Marine Fisheries Service National Task Force for Improving Fish Stock Assessments
  • P M Mace
  • N W Bartoo
  • A B Hollowed
  • P Kleiber
  • R D Methot
  • S A Murawski
  • J E Powers
  • G P Scott
Mace PM, Bartoo NW, Hollowed AB, Kleiber P, Methot RD, Murawski SA, Powers, JE, Scott GP. 2001. Marine Fisheries Stock Assessment Improvement Plan. Report of the National Marine Fisheries Service National Task Force for Improving Fish Stock Assessments. NOAA Technical Memorandum, NMFS-F/SPO-56 l. 69p.
Fishery Conservation and Management Act
  • Magnuson-Stevens
Magnuson-Stevens Fishery Conservation and Management Act. 2007. 16 U.S.C. 1801.