Deepak Khosla

Deepak Khosla
HRL Laboratories, LLC | HRL · Information & Systems Sciences

PhD, Biomedical Engineering

About

96
Publications
14,127
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
1,562
Citations

Publications

Publications (96)
Preprint
Full-text available
We propose a novel framework for value function factorization in multi-agent deep reinforcement learning (MARL) using graph neural networks (GNNs). In particular, we consider the team of agents as the set of nodes of a complete directed graph, whose edge weights are governed by an attention mechanism. Building upon this underlying graph, we introdu...
Chapter
We describe a high-accuracy, real-time, neuromorphic method and system for activity recognition in streaming or recorded videos from static and moving platforms that can detect even small objects and activities with high-accuracy. Our system modifies and integrates multiple independent algorithms into an end-to-end system consisting of five primary...
Chapter
Images of outdoor scenes are usually degraded by atmospheric particles, such as haze, fog and smoke, which fade the color and reduce the contrast of objects in the scene. This reduces image quality for manual or automated analysis in a variety of outdoor video surveillance applications, for example threat or anomaly detection. Current dehazing tech...
Conference Paper
Described is a system for object detection from dynamic visual imagery. Video imagery is received as input, and the system processes each frame to detect a motion region exhibiting unexpected motion representing a moving object. Object-based feature extraction is applied to each frame containing a detection motion region. Each frame is then divided...
Conference Paper
Full-text available
We propose an online 3D sensor-based algorithm for autonomous robot exploration in an indoor setting. Our algorithm consists of two modules, a proactive open space detection module, and a reactive obstacle avoidance module. The former, which is the primary contribution of the paper, is responsible for guiding the robot towards meaningful open space...
Conference Paper
Full-text available
This work describes an efficient perception-control coupled system and its underlying algorithms that enable autonomous exploration of indoor environments by a Micro Aerial Vehicle (MAV) equipped with a monocular camera and sonar sensors. The perception subsystem uses inputs from the camera to detect the vanishing point and doors in corridors. It d...
Article
Full-text available
Deep-learning neural networks such as convolutional neural network (CNN) have shown great potential as a solution for difficult vision problems, such as object recognition. Spiking neural networks (SNN)-based architectures have shown great potential as a solution for realizing ultra-low power consumption using spike-based neuromorphic hardware. Thi...
Patent
Described is method for object cueing in motion imagery. Key points and features are extracted from motion imagery, and features between consecutive image frames of the motion imagery are compared to identify similar image frames. A candidate set of matching keypoints is generated by matching keypoints between the similar image frames. A ground pla...
Patent
Full-text available
Described is a system for object detection using classification-based learning. A fusion method is selected, then a video sequence is processed to generate detections for each frame, wherein a detection is a representation of an object candidate. The detections are fused to generate a set of fused detections for each frame. The classification modul...
Article
Full-text available
Automated video object recognition is a topic of emerging importance in both defense and civilian applications. This work describes an accurate and low-power neuromorphic architecture and system for real-time automated video object recognition. Our system, Neuormorphic Visual Understanding of Scenes (NEOVUS), is inspired by recent findings in compu...
Patent
Full-text available
Described is a system for multiple-object recognition in visual images. The system is configured to receive an input test image comprising at least one object. Keypoints representing the object are extracted using a local feature algorithm. The keypoints from the input test image are matched with keypoints from at least one training image stored in...
Patent
Full-text available
The present invention relates to a system for identifying regions of interest in visual imagery. The system is configured to receive a series of consecutive frames representing a scene as captured from N sensors. The frames include at least a current frame and a previous frame. A surprise map can be generated based on features found in the current...
Patent
Full-text available
Described is a system for multi-class classifier threshold-offset estimation for visual object recognition. The system receives an input image with input features for classifying. A pair-wise classifier is trained for each pair of a plurality of object classes. A set of classification responses is generated, and a multi-class receiver-operating-cha...
Patent
Full-text available
Described is a system for optimizing rapid serial visual presentation (RSVP). A similarity metric is computed for RSVP images, and the images are sequenced according to the similarity metrics. The sequenced images are presented to a user, and neural signals are received to detect a P300 signal. A neural score for each image is computed, and the sys...
Patent
Full-text available
The present invention creates and stores target representations in several coordinate representations based on biologically inspired models of the human vision system. By using biologically inspired target representations a computer can be programmed for robot control without using kinematics to relate a target position in camera eyes to a target p...
Article
Autonomous object recognition in images and videos is a topic of emerging importance in commercial and defense applications. Recent advances in the field of visual neuroscience are increasingly being leveraged to develop biologically-plausible models and algorithms for visual cognition with the ultimate goal of improving object recognition accuracy...
Patent
To improve the scheduling and tasking of sensors, the present disclosure describes an improved planning system and method for the allocation and management of sensors. In one embodiment, the planning system uses a branch and bound approach of tasking sensors using a heuristic to expedite arrival at a deterministic solution. In another embodiment, a...
Patent
A bio-inspired actionable intelligence method and system is disclosed. The actionable intelligence method comprises recognizing entities in an imagery signal, detecting and classifying anomalous entities, and learning new hierarchal relationships between different classes of entities. A knowledge database is updated after each new learning experien...
Conference Paper
In this paper, we introduce a user interface called the “Threat Chip Display” (TCD) for rapid human-in-the-loop analysis and detection of “threats” in high-bandwidth imagery and video from a list of “Items of Interest” (IOI), which includes objects, targets and events that the human is interested in detecting and identifying. Typically some front-e...
Patent
Full-text available
Described is a Distributed Resource Allocation System (DRAS) for sensor control and planning. The DRAS comprises an information framework module that is configured to specify performance goals, assess current performance state, and includes sensor models to achieve the performance goals. The DRAS is configured to further allocate the sensors to ach...
Conference Paper
Full-text available
Unmanned surveillance platforms have a ubiquitous presence in surveillance and reconnaissance operations. As the resolution and fidelity of the video sensors on these platforms increases, so does the bandwidth required to provide the data to the analyst and the subsequent analyst workload to interpret it. This leads to an increasing need to perform...
Conference Paper
Full-text available
Unattended object detection, recognition and tracking on unmanned reconnaissance platforms in battlefields and urban spaces are topics of emerging importance. In this paper, we present an unattended object recognition system that automatically detects objects of interest in videos and classifies them into various categories (e.g., person, car, truc...
Conference Paper
Real-time detection of objects in video sequences captured from an aerial platforms is a key task for surveillance applications. It is common to perform expensive frame to frame registration as preprocessing to moving object detection in this type of application, and there is no principled approach to the detection of stationary targets.We explore...
Conference Paper
A key to any robust automated surveillance system is continuous, wide field-of-view sensor coverage and high accuracy target detection algorithms. Newer systems typically employ an array of multiple fixed cameras that provide individual data streams, each of which is managed by its own processor. This array can continuously capture the entire field...
Patent
Full-text available
A method and apparatus for controlling robots based on prioritized targets extracted from fused visual and auditory saliency maps. The fused visual and auditory saliency map may extend beyond the immediate visual range of the robot yet the methods herein allow the robot to maintain an awareness of targets outside the immediate visual range. The fus...
Patent
Full-text available
The present disclosure describes a fused saliency map from visual and auditory saliency maps. The saliency maps are in azimuth and elevation coordinates. The auditory saliency map is based on intensity, frequency and temporal conspicuity maps. Once the auditory saliency map is determined, the map is converted into azimuth and elevation coordinates...
Patent
Full-text available
Described is a system for finding salient regions in imagery. The system improves upon the prior art by receiving an input image of a scene and dividing the image into a plurality of image sub-regions. Each sub-region is assigned a coordinate position within the image such that the sub-regions collectively form the input image. A plurality of local...
Conference Paper
In this paper, we describe a hybrid human-machine system for searching and detecting Objects of Interest (OI) in imagery. Automated methods for OI detection based on models of human visual attention have received much interest, but are inherently bottom-up and driven by features. Humans fixate on regions of imagery based on a much stronger top-down...
Conference Paper
Though Electroencephalography (EEG)-based brain-computer interfaces (BCI) have come to outperform pure computer vision algorithms on difficult image triage tasks, none of these BCIs have leveraged the effects of motion on the human visual attention system. Here we consider the advantages of leveraging the effects of motion by testing a new method f...
Conference Paper
Full-text available
A significant challenge in creating machines with artificial vision is designing systems which can process visual information as efficiently as the brain. To address this challenge, we identify key algorithms which model the process of attention and recognition in the visual cortex of mammals. This paper presents Cover - an FPGA framework for gener...
Article
Full-text available
Advances in neuroscience have enabled researchers to develop computational models of auditory, visual and learning perceptions in the human brain. HMAX, which is a biologically inspired model of the visual cortex, has been shown to outperform standard computer vision approaches for multi-class object recognition. HMAX, while computationally demandi...
Conference Paper
Full-text available
We present a novel method to maximize multiclass classifier performance by tuning the thresholds of the constituent pairwise binary classifiers using Particle Swarm Optimization. This post-processing step improves the classification performance in multiclass visual object detection by maximizing the area under the ROC curve or various operating poi...
Conference Paper
Full-text available
Research has shown that the application of an attention algorithm to the front-end of an object recognition system can provide a boost in performance over extracting regions from an image in an unguided manner. However, when video imagery is taken from a moving platform, attention algorithms such as saliency can lose their potency. In this paper, w...
Article
This paper describes a system for multiple-object recognition and segmentation that (1) correctly identifies objects in a natural scene and provides a boundary for each object, (2) can identify multiple occurrences of the same object (e.g., two identical objects, side-by-side) in the scene from different training views. The algorithm is novel in th...
Article
This paper describes a fast and robust bio-inspired method for change detection in high-resolution visual imagery. It is based on the computation of surprise, a dynamic analogue to visual saliency or attention, that uses very little processing beyond that of the initial computation of saliency. This is different from prior surprise algorithms, whic...
Article
The Rapid Serial Visual Presentation (RSVP) protocol for EEG has recently been discovered as a useful tool for highthroughput filtering of images into simple target and nontarget categories [1]. This concept can be extended to the detection of objects and anomalies in images and videos that are of interest to the user (observer) in an applicationsp...
Article
This paper describes a method and system of human-like attention and object segmentation in visual scenes that (1) attends to regions in a scene in their rank of saliency in the image, (2) extracts the boundary of an attended proto-object based on feature contours, and (3) can be biased to boost the attention paid to specific features in a scene, s...
Article
This paper describes an algorithm and system for rapidly generating a saliency map and finding interesting regions and in large-sized (i.e., extremely high-resolution) imagery and video. Previous methods of finding salient or interesting regions have a fundamental shortcoming: they need to process the entire image before the saliency map can be out...
Article
In this paper, we describe COGNIVA, a closed-loop Cognitive-Neural method and system for image and video analysis that combines recent technological breakthroughs in bio-vision cognitive algorithms and neural signatures of human visual processing. COGNIVA is an "operational neuroscience" framework for intelligent and rapid search and categorization...
Article
Volitional search systems that assist the analyst by searching for specific targets or objects such as vehicles, factories, airports, etc in wide area overhead imagery need to overcome multiple problems present in current manual and automatic approaches. These problems include finding targets hidden in terabytes of information, relatively few pixel...
Article
Full-text available
In many real-world situations and applications that involve humans or machines (e.g., situation awareness, scene understanding, driver distraction, workload reduction, assembly, robotics, etc.) multiple sensory modalities (e.g., vision, auditory, touch, etc.) are used. The incoming sensory information can overwhelm processing capabilities of both h...
Article
Full-text available
This paper describes a bio-inspired VISion based actionable INTelligence system (VISINT) that provides automated capabilities to (1) understand objects, patterns, events and behaviors in vision data; (2) translate this understanding into timely recognition of novel and anomalous entities; and (3) discover underlying hierarchies and relationships be...
Article
Full-text available
In this paper, we describe a method and system for robust and efficient goal-oriented active control of a machine (e.g., robot) based on processing, hierarchical spatial understanding, representation and memory of multimodal sensory inputs. This work assumes that a high-level plan or goal is known a priori or is provided by an operator interface, w...
Article
Full-text available
This paper describes an efficient method and system for representing, processing and understanding multi-modal sensory data. More specifically, it describes a computational method and system for how to process and remember multiple locations in multimodal sensory space (e.g., visual, auditory, somatosensory, etc.). The multimodal representation and...
Article
Full-text available
While the task of sensing and perceiving the visual environment as we go about our daily lives is trivial for most humans, attempts to emulate the principles underlying human vision in machine vision systems have only been marginally successful. Attention, mediated by eye movements, acts as the critical gateway to visual cognition by searching for...
Article
This paper presents a bio-inspired method for spatio-temporal recognition in static and video imagery. It builds upon and extends our previous work on a bio-inspired Visual Attention and object Recognition System (VARS). The VARS approach locates and recognizes objects in a single frame. This work presents two extensions of VARS. The first extensio...
Article
Full-text available
This paper describes a bio-inspired Visual Attention and Object Recognition System (VARS) that can (1) learn representations of objects that are invariant to scale, position and orientation; and (2) recognize and locate these objects in static and video imagery. The system uses modularized bio-inspired algorithms/techniques that can be applied towa...
Article
This paper presents a bio-inspired method for spatio-temporal recognition in static and video imagery. It builds upon and extends our previous work on a bio-inspired Visual Attention and object Recognition System (VARS). The VARS approach locates and recognizes objects in a single frame. This work presents two extensions of VARS. The first extensio...
Article
Full-text available
The goal of sensor resource management (SRM) is to allocate resources appropriately in order to gain as much information as possible about a system. In our previous paper, we introduced a centralized non-myopic planning algorithm, C-SPLAN, that uses sparse sampling to estimate the value of resource assignments. Sparse sampling is related to Monte C...
Article
Full-text available
The goal of sensor resource management (SRM) is to allocate resources appropriately in order to gain as much information as possible about a system. We introduce a centralized non-myopic planning algorithm, C-SPLAN, that uses sparse sampling to estimate the value of resource assignments. Sparse sampling is related to Monte Carlo simulation. In the...
Article
Network-centric force optimization is the problem of threat engagement and dynamic Weapon-Target Allocation (WTA) across the force. The goal is to allocate and schedule defensive weapon resources over a given period of time so as to achieve certain battle management objectives subject to resource and temporal constraints. The problem addresses in t...
Article
Sensitivity analysis in an uncertainty reasoning system helps establish the relationship between the system output and the system parameters under a given input condition. Much work has been done in Bayesian reasoning and, in particular, Bayesian networks in the last decade. However, little work has been done in other uncertainty reasoning framewor...
Conference Paper
This paper describes an adaptive Kalman filter based method for accurate estimation of forward path geometry of an automobile. The forward geometry is modeled as two contiguous clothoid segments with different geometries and continuous curvature across the transition between them. This results in a closed-form parametric expression of the same poly...
Article
Full-text available
This study investigates the effects of profound acquired unilateral deafness on the adult human central auditory system by analyzing long-latency auditory evoked potentials (AEPs) with dipole source modeling methods. AEPs, elicited by clicks presented to the intact ear in 19 adult subjects with profound unilateral deafness and monaurally to each ea...
Conference Paper
The paper proposes a new set of fuzzy features based on symmetry of edges for improving the accuracy of detecting intruders. We show that the proposed fuzzy edge-symmetry feature-based classifier is comparable to the detection accuracy of a multi-scale wavelet feature system for intruder detection. We also present two approaches to fusing the resul...
Conference Paper
This paper describes a new model and method for accurate estimation of forward path geometry of an automobile. In this work the forward geometry is modelled by two contiguous clothoid segments with different geometries, but continuous curvature across the transition between them. This results in a closed-form parametric expression of the same polyn...
Article
Full-text available
Objectives: Previous studies have shown that observed patterns of auditory evoked potential (AEP) maturation depend on the scalp location of the recording electrodes. Dipole source modeling incorporates the AEP information recorded at all electrode locations. This should provide a more robust description of auditory system maturation based on age-...
Conference Paper
This paper describes a new model and method for accurate estimation of forward path geometry of an automobile. In this work, the forward geometry is modeled by two contiguous clothoid segments with different geometries, but continuous curvature across the transition between them. This results in a closed-form parametric expression of the same polyn...
Article
This paper addresses the problem of threat engagement and dynamic weapon-target allocation (WTA) across the force or network-centric force optimization. The objective is to allocate and schedule defensive weapon resources over a given period of time so as to minimize surviving target value subject to resource availability and temporal constraints....
Article
Experience-related changes in central nervous system (CNS) activity have been observed in the adult brain of many mammalian species, including humans. In humans, late-onset profound unilateral deafness creates an opportunity to study plasticity in the adult CNS consequent to monaural auditory deprivation. CNS activity was assessed by measuring long...
Article
We present a method based on the distributed dipole source model to localize sources of spontaneous human brain activity, such as the alpha rhythm. The proposed method relies on the generalized maximum entropy principle and is implemented in frequency-domain. Several computer simulation studies of synchronous and asynchronous distributed dipole sou...
Article
The estimation of cortical current activity from scalp-recorded potentials is a complicated mathematical problem that requires fairly precise knowledge of the location of the scalp electrodes. It is expected that spatial mislocalization of electrodes will introduce errors in this estimation. The present study uses simulated and real data to quantif...
Article
The estimation of multiple dipole parameters in spatio-temporal source modeling (STSM) of electroencephalographic (EEG) data is a difficult nonlinear optimization problem due to multiple local minima in the cost function. A straightforward iterative optimization approach to such a problem is very susceptible to being trapped in a local minimum, the...
Article
Magnetoencephalographic imaging is the estimation of three-dimensional neuronal current sources on the cortical surface from the measured magnetoencephalogram (MEG). It is a highly under-determined inverse problem as there are many “feasible” images which are consistent with the MEG. Previous approaches to this problem have concentrated on the use...
Article
Electroencephalographic imaging is the estimation of 3D neuronal current sources on the cortical surface from the measured electroencephalogram (EEG). It is a highly under- determined inverse problem as there are many 'feasible' images which are consistent with the scalp potentials. Previous approaches to this problem have primarily concentrated on...
Article
Generators of spontaneous human brain activity such as alpha rhythm may be easier and more accurate to localize in frequency-domain than in time-domain since these generators are characterized by a specific frequency range. We carried out a frequency-domain analysis of synchronous alpha sources by generating equivalent potential maps using the Four...
Article
The detection of sites of brain activation in functional MRI has been a topic of immense research interest and many techniques have been proposed to this end. Recently,principal component analysis (PCA) has been applied to extract the activated regions and their time course of activation. This method is based on the assumption that the activation i...
Article
Electroencephalographic imaging is the estimation of 3D neuronal current sources on the cortical surface from the measured electroencephalogram (EEG). It is a highly under- determined inverse problem as there are many 'feasible' images which are consistent with the scalp potentials. Previous approaches to this problem have primarily concentrated on...
Article
A preliminary study was conducted to segment 1.5 T fMRIs into the microvasculature and relatively large blood vessels using the intensity, phase and temporal delay of activated pixels as three correlated parameters in gradient echo images. Images acquired during visual stimulation using a checkerboard flashing at 8 Hz were investigated. Activated p...
Article
Changes in T2* of activated pixels were estimated using single voxel and 1D chemical shift imaging (CSI) proton spectroscopy during visual stimulation. A single voxel was angulated and positioned to enclose activated pixels as determined by a corresponding functional MRI study. Eddy currents in angulated voxel studies produced a bump in the water p...
Article
Though excellent spatial resolution (on the order of 1 mm) is obtainable in functional MRI (fMRI), its temporal resolution is limited to about 1 second by hemodynamics. On the other hand, magnetoencephalography (MEG) and electroencephalography (EEG) provide millisecond temporal resolution but a relatively crude (on the order of 1 cm) spatial resolu...