Multi-camera real-time three-dimensional tracking of multiple flying animals

California Institute of Technology, Bioengineering, Mailcode 138-78, Pasadena, CA 91125, USA.
Journal of The Royal Society Interface (Impact Factor: 3.92). 03/2011; 8(56):395-409. DOI: 10.1098/rsif.2010.0230
Source: PubMed


Automated tracking of animal movement allows analyses that would not otherwise be possible by providing great quantities of data. The additional capability of tracking in real time--with minimal latency--opens up the experimental possibility of manipulating sensory feedback, thus allowing detailed explorations of the neural basis for control of behaviour. Here, we describe a system capable of tracking the three-dimensional position and body orientation of animals such as flies and birds. The system operates with less than 40 ms latency and can track multiple animals simultaneously. To achieve these results, a multi-target tracking algorithm was developed based on the extended Kalman filter and the nearest neighbour standard filter data association algorithm. In one implementation, an 11-camera system is capable of tracking three flies simultaneously at 60 frames per second using a gigabit network of nine standard Intel Pentium 4 and Core 2 Duo computers. This manuscript presents the rationale and details of the algorithms employed and shows three implementations of the system. An experiment was performed using the tracking system to measure the effect of visual contrast on the flight speed of Drosophila melanogaster. At low contrasts, speed is more variable and faster on average than at high contrasts. Thus, the system is already a useful tool to study the neurobiology and behaviour of freely flying animals. If combined with other techniques, such as 'virtual reality'-type computer graphics or genetic manipulation, the tracking system would offer a powerful new way to investigate the biology of flying animals.

85 Reads
  • Source
    • "These progresses allow today the automatic tracking of large groups of objects in a way that was prohibitive only a few years back. A particularly energetic boost of the research into 3D tracking has come from the study of collective behaviour in biological systems, as bird flocks (Attanasi et al., 2015), flying bats (Wu et al., 2011), insect swarms (Straw et al., 2010) (Puckett et al., 2014) (Cheng et al., 2015) and fish schools (Butail et al., 2010) (Pérez-Escudero et al., 2014). The aim in this field is to use experimental data about the actual trajectories of individual animals to infer the underlying interaction rules at the basis of collective motion (Giardina , 2008). "
    [Show abstract] [Hide abstract]
    ABSTRACT: The interest in 3D dynamical tracking is growing in fields such as robotics, biology and fluid dynamics. Recently, a major source of progress in 3D tracking has been the study of collective behaviour in biological systems, where the trajectories of individual animals moving within large and dense groups need to be reconstructed to understand the behavioural interaction rules. Experimental data in this field are generally noisy and at low spatial resolution, so that individuals appear as small featureless objects and trajectories must be retrieved by making use of epipolar information only. Moreover, optical occlusions often occur: in a multi-camera system one or more objects become indistinguishable in one view, potentially jeopardizing the conservation of identity over long-time trajectories. The most advanced 3D tracking algorithms overcome optical occlusions making use of set-cover techniques, which however have to solve NP-hard optimization problems. Moreover, current methods are not able to cope with occlusions arising from actual physical proximity of objects in 3D space. Here, we present a new method designed to work directly in 3D space and time, creating (3D+1) clouds of points representing the full spatio-temporal evolution of the moving targets. We can then use a simple connected components labeling routine, which is linear in time, to solve optical occlusions, hence lowering from NP to P the complexity of the problem. Finally, we use normalized cut spectral clustering to tackle 3D physical proximity.
  • Source
    • "Much research has assumed that adjacent camera views have overlap and utilized the spatial proximity of tracks in the overlapping area. As described in [2], tracks of objects observed in different camera views were stitched based on their spatial proximity [141], [142]. In order to track objects across disjoint camera views, appearance cues have been integrated with spatiotemporal reasoning [143], [144]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Traffic surveillance has become an important topic in intelligent transportation systems (ITSs), which is aimed at monitoring and managing traffic flow. With the progress in computer vision, video-based surveillance systems have made great advances on traffic surveillance in ITSs. However, the performance of most existing surveillance systems is susceptible to challenging complex traffic scenes (e.g., object occlusion, pose variation, and cluttered background). Moreover, existing related research is mainly on a single video sensor node, which is incapable of addressing the surveillance of traffic road networks. Accordingly, we present a review of the literature on the video-based vehicle surveillance systems in ITSs. We analyze the existing challenges in video-based surveillance systems for the vehicle and present a general architecture for video surveillance systems, i.e., the hierarchical and networked vehicle surveillance, to survey the different existing and potential techniques. Then, different methods are reviewed and discussed with respect to each module. Applications and future developments are discussed to provide future needs of ITS services.
    IEEE Transactions on Intelligent Transportation Systems 04/2015; 16(2):557-580. DOI:10.1109/TITS.2014.2340701 · 2.38 Impact Factor
  • Source
    • "The recent growth of RGB-D cameras offers new possibilities for recovering individual 3D tracks from insect colonies or swarms ([2], [3], [4]). At a global level, we consider the complete chain made of the following steps: acquisition, segmentation, tracking, and behavior analysis. "
    [Show abstract] [Hide abstract]
    ABSTRACT: In video-surveillance, non-parametric Bayesian approaches based on a Hierarchical Dirichlet Process (HDP) have recently shown their efficiency for modeling crowed scene activities. This paper follows this track by proposing a method for detecting and clustering emergent behaviors across different captures made of numerous unconstrained trajectories. Most HDP applications for crowed scenes (e.g. traffic, pedestrians) are based on flow motion features. In contrast, we propose to tackle the problem by using full individual trajectories. Furthermore, our proposed approach relies on a three-level clustering hierarchical Dirichlet process able with a minimum a priori to hierarchically retrieve behaviors at increasing semantical levels: activity atoms, activities and behaviors. We chose to validate our approach on ant trajectories simulated by a Multi-Agent System (MAS) using an ant colony foraging model. The experimentation results have shown the ability of our approach to discover different emergent behaviors at different scales, which could be associated to observable events such as 'forging' or 'deploying' for instance.
Show more


85 Reads
Available from