Shree K. Nayar's research while affiliated with Snap inc. and other places

Publications (352)

Article
We present Vidgets, a family of mechanical widgets, specifically push buttons and rotary knobs that augment mobile devices with tangible user interfaces. When these widgets are attached to a mobile device and a user interacts with them, the widgets' nonlinear mechanical response shifts the device slightly and quickly, and this subtle motion can be...
Conference Paper
We introduce the racing auditory display (RAD), an audio-based user interface that allows players who are blind to play the same types of racing games that sighted players can play with an efficiency and sense of control that are similar to what sighted players have. The RAD works with a standard pair of headphones and comprises two novel sonificat...
Article
Stretchcam is a thin camera with a lens capable of zooming with small actuations. In our design, an elastic lens array is placed on top of a sparse, rigid array of pixels. This lens array is then stretched using a small mechanical motion in order to change the field of view of the system. We present in this paper the characterization of such a syst...
Article
The depth resolution achieved by a continuous wave time-of-flight (C-ToF) imaging system is determined by the coding (modulation and demodulation) functions that it uses. Almost all current C-ToF systems use sinusoid or square coding functions, resulting in a limited depth resolution. In this article, we present a mathematical framework for explori...
Article
THE CAMERAS IN our phones and tablets have turned us all into avid photographers, regularly using them to capture special moments and document our lives. One notable feature of camera phones is they are compact and fully automatic, enabling us to point and shoot without having to adjust any settings. However, when we need to capture photos of high...
Conference Paper
We present AirCode, a technique that allows the user to tag physically fabricated objects with given information. An AirCode tag consists of a group of carefully designed air pockets placed beneath the object surface. These air pockets are easily produced during the fabrication process of the object, without any additional material or postprocessin...
Article
We present AirCode, a technique that allows the user to tag physically fabricated objects with given information. An AirCode tag consists of a group of carefully designed air pockets placed beneath the object surface. These air pockets are easily produced during the fabrication process of the object, without any additional material or postprocessin...
Conference Paper
Today's game analytics systems are powered by event logs, which reveal information about what players are doing but offer little insight about the types of gameplay that games foster. Moreover, the concept of gameplay itself is difficult to define and quantify. In this paper, we show that analyzing players' controller inputs using probabilistic top...
Article
We present DisCo, a novel display-camera communication system. DisCo enables displays and cameras to communicate with each other while also displaying and capturing images for human consumption. Messages are transmitted by temporally modulating the display brightness at high frequencies so that they are imperceptible to humans. Messages are receive...
Conference Paper
We propose a framework for developing a new class of imaging systems that are thin and flexible. Such an imaging sheet can be flexed at will and wrapped around everyday objects to capture unconventional fields of view. Our approach is to use a lens array attached to a sheet with a 2D grid of pixels. A major challenge with this type of a system is t...
Article
Images of scenes acquired in bad weather have poor con-trasts and colors. It is known that the degradation of image quality due to bad weather is exponential in the depths of the scene points. Therefore, restoring scene colors and contrasts from a single image of the scene is inherently under-constrained. Recently, it has been shown that multiple i...
Article
Camera optics design has become increasingly complex to keep pace with the demand for ever increasing imaging resolution. This complexity results in several disadvantages, including significant increase in size, cost, and weight of the camera. In this paper, we propose methods to simplify the lens of an imaging system without sacrificing resolution...
Article
In correlation-based time-of-flight (C-ToF) imaging systems, light sources with temporally varying intensities illuminate the scene. Due to global illumination, the temporally varying radiance received at the sensor is a combination of light received along multiple paths. Recovering scene properties (e.g., scene depths) from the received radiance r...
Conference Paper
Full-text available
We propose a simple pixel design, where the pixel's pho-todiode can be used to not only measure the incident light level, but also to convert the incident light into electrical energy. A sensor architecture is proposed where, during each image capture cycle, the pixels are used first to record and read out the image and then used to harvest energy...
Patent
Full-text available
Methods, systems, and media for swapping faces in images are provided. In some embodiments, a detected face and face data corresponding to an input image is received. A pose bin associated with the detected face is then identified based on the face data. Next, the detected face is aligned to a generic face associated with the pose bin. At least a p...
Article
Cameras face a fundamental trade-off between spatial and temporal resolution. Digital still cameras can capture images with high spatial resolution, but most high-speed video cameras have relatively low spatial resolution. It is hard to overcome this trade-off without incurring a significant increase in hardware costs. In this paper, we propose tec...
Conference Paper
Exposure bracketing for high dynamic range (HDR) imaging involves capturing several images of the scene at different exposures. If either the camera or the scene moves during capture, the captured images must be registered. Large exposure differences between bracketed images lead to inaccurate registration, resulting in artifacts such as ghosting (...
Conference Paper
Strong ambient illumination severely degrades the performance of structured light based techniques. This is especially true in outdoor scenarios, where the structured light sources have to compete with sunlight, whose power is often 2-5 orders of magnitude larger than the projected light. In this paper, we propose the concept of light concentration...
Patent
Full-text available
Methods, systems, and media for automatically classifying face images are provided. In some embodiments, features of the face image to be classified for an attribute are selected, wherein each of the features corresponds to a different region of the face image and specifies one or more of a type of pixel data to be evaluated for the region, a norma...
Conference Paper
Eye contact plays a crucial role in our everyday social interactions. The ability of a device to reliably detect when a person is looking at it can lead to powerful human-object interfaces. Today, most gaze-based interactive systems rely on gaze tracking technology. Unfortunately, current gaze tracking techniques require active infrared illuminatio...
Data
Images taken with wide-angle cameras tend to have severe distortions which pull points towards the optical center. This paper proposes a simple method for recovering the distortion parameters without the use of any calibration objects. Since distortions cause straight lines in the scene to appear as curves in the image, our algorithm seeks to find...
Conference Paper
A number of cameras have been introduced that sweep the focal plane using mechanical motion. However, mechanical motion makes video capture impractical and is unsuitable for long focal length cameras. In this paper, we present a focal sweep telephoto camera that uses a variable focus lens to sweep the focal plane. Our camera requires no mechanical...
Article
Full-text available
Many vision tasks such as scene segmentation, or the recognition of materials within a scene, become considerably easier when it is possible to measure the spectral re-flectance of scene surfaces. In this paper, we present an efficient and robust approach for recovering spectral re-flectance in a scene that combines the advantages of using multiple...
Article
A number of computational imaging techniques are introduced to improve image quality by increasing light throughput. These techniques use optical coding to measure a stronger signal level. However, the performance of these techniques is limited by the decoding step, which amplifies noise. Although it is well understood that optical coding can incre...
Conference Paper
We consider the problem of shape recovery for real world scenes, where a variety of global illumination (inter-reflections, subsurface scattering, etc.) and illumination defocus effects are present. These effects introduce systematic and often significant errors in the recovered shape. We introduce a structured light technique called Micro Phase Sh...
Article
Full-text available
We propose a new method named compressive structured light for recovering inhomogeneous participating media. Whereas conventional structured light methods emit coded light patterns onto the surface of an opaque object to establish correspondence for triangulation, compressive structured light projects patterns into a volume of participating medium...
Conference Paper
Today, structured light systems are widely used in applications such as robotic assembly, visual inspection, surgery, entertainment, games and digitization of cultural heritage. Current structured light methods are faced with two serious limitations. First, they are unable to cope with scene regions that produce strong highlights due to specular re...
Article
Full-text available
A computational camera uses a combination of optics and processing to produce images that cannot be captured with traditional cameras. In the last decade, computational imaging has emerged as a vibrant field of research. A wide variety of computational cameras has been demonstrated to encode more useful visual information in the captured images, as...
Article
Full-text available
The resolution of a camera system determines the fidelity of visual features in captured images. Higher resolution implies greater fidelity and, thus, greater accuracy when performing automated vision tasks, such as object detection, recognition, and tracking. However, the resolution of any camera is fundamentally limited by geometric aberrations....
Conference Paper
Full-text available
Cameras face a fundamental tradeoff between the spatial and temporal resolution - digital still cameras can capture images with high spatial resolution, but most high-speed video cameras suffer from low spatial resolution. It is hard to overcome this tradeoff without incurring a significant increase in hardware costs. In this paper, we propose tech...
Article
Full-text available
We introduce the use of describable visual attributes for face verification and image search. Describable visual attributes are labels that can be given to an image to describe its appearance. This paper focuses on images of faces and the attributes used to describe them, although the concepts also apply to other domains. Examples of face attribute...
Conference Paper
Global illumination effects such as inter-reflections and subsurface scattering result in systematic, and often significant errors in scene recovery using active illumination. Recently, it was shown that the direct and global components could be separated efficiently for a scene illuminated with a single light source. In this paper, we study the pr...
Conference Paper
Today, consumer cameras produce photographs with tens of millions of pixels. The recent trend in image sensor resolution seems to suggest that we will soon have cameras with billions of pixels. However, the resolution of any camera is fundamentally limited by geometric aberrations. We derive a scaling law that shows that, by using computations to c...
Article
The classical approach to depth from defocus (DFD) uses lenses with circular apertures for image capturing. We show in this paper that the use of a circular aperture severely restricts the accuracy of DFD. We derive a criterion for evaluating a pair of apertures with respect to the precision of depth recovery. This criterion is optimized using a ge...
Article
Full-text available
The range of scene depths that appear focused in an image is known as the depth of field (DOF). Conventional cameras are limited by a fundamental trade-off between depth of field and signal-to-noise ratio (SNR). For a dark scene, the aperture of the lens must be opened up to maintain SNR, which causes the DOF to reduce. Also, today's cameras have D...
Article
Full-text available
A major limitation of existing projection display sys-tems is that they rely on a high quality screen for pro-jecting images. We believe that relaxing this restriction will make projectors more useful and widely applicable. The fundamental problem with using an arbitrary sur-face for a screen is that the surface is bound to have its own colors and...
Conference Paper
Full-text available
Since 1960s, aperture patterns have been studied extensively and a variety of coded apertures have been proposed for various applications, including extended depth of field, defocus deblurring, depth from defocus, light field acquisition, etc. Researches have shown that optimal aperture patterns can be quite different due to different applications,...
Article
In recent years, several cameras have been introduced which extend depth of field (DOF) by producing a depth-invariant point spread function (PSF). These cameras extend DOF by deblurring a captured image with a single spatially-invariant PSF. For these cameras, the quality of recovered images depends both on the magnitude of the PSF spectrum (MTF)...
Conference Paper
Full-text available
An optical diffuser is an element that scatters light and is commonly used to soften or shape illumination. In this pa- per, we propose a novel depth estimation method that places a diffuser in the scene prior to image capture. We call this approach depth-from-diffusion (DFDiff). We show that DFDiff is analogous to conventional depth- from-defocus...
Conference Paper
In recent years, many new camera designs have been proposed which preserve image detail over a larger depth range than conventional cameras. These methods rely on either mechanical motion or a custom optical element placed in the pupil plane of a camera lens to create the desired point spread function (PSF). This work introduces a new Spectral Foca...
Conference Paper
Full-text available
We propose a novel readout architecture called coded rolling shutter for complementary metal-oxide semiconductor (CMOS) image sensors. Rolling shutter has traditionally been considered as a disadvantage to image quality since it often introduces skew artifact. In this paper, we show that by controlling the readout timing and the exposure length for...
Article
Full-text available
We propose the concept of a generalized assorted pixel (GAP) camera, which enables the user to capture a single image of a scene and, after the fact, control the tradeoff between spatial resolution, dynamic range and spectral detail. The GAP camera uses a complex array (or mosaic) of color filters. A major problem with using such an array is that t...
Article
Dirt on camera lenses, and occlusions from thin objects such as fences, are two important types of artifacts in digital imaging systems. These artifacts are not only an annoyance for photographers, but also a hindrance to computer vision and digital forensics. In this paper, we show that both effects can be described by a single image formation mod...
Article
Full-text available
Dirt on camera lenses, and occlusions from thin objects such as fences, are two important types of artifacts in digital imaging systems. These artifacts are not only an annoyance for photographers, but also a hindrance to computer vision and digital forensics. In this paper, we show that both effects can be described by a single image formation mod...
Conference Paper
Full-text available
We present two novel methods for face verification. Our first method - “attribute” classifiers - uses binary classifiers trained to recognize the presence or absence of describable aspects of visual appearance (e.g., gender, race, and age). Our second method - “simile” classifiers - removes the manual labeling required for attribute classification...
Conference Paper
Full-text available
The classical approach to depth from defocus uses two images taken with circular apertures of different sizes. We show in this paper that the use of a circular aperture severely restricts the accuracy of depth from defocus. We derive a criterion for evaluating a pair of apertures with re- spect to the precision of depth recovery. This criterion is...
Article
We present a new model of the homogeneous BSSRDF based on large-scale simulations. Our model captures the appearance of materials that are not accurately represented using existing single scattering models or multiple isotropic scattering models (e.g. the diffusion approximation). We use an analytic function to model the 2D hemispherical distributi...
Conference Paper
Full-text available
We present a novel multi-view denoising algorithm. Our algorithm takes noisy images taken from different viewpoints as input and groups similar patches in the input images using depth estimation. We model intensity-dependent noise in low- light conditions and use the principal component analysis and tensor analysis to remove such noise. The dimensi...
Conference Paper
Full-text available
In recent years, with camera pixels shrinking in size, images are more likely to include defocused regions. In order to recover scene details from defocused regions, deblurring techniques must be applied. It is well known that the quality of a deblurred image is closely related to the defocus kernel, which is determined by the pattern of the apertu...
Conference Paper
We present the design of a spherical imaging system with the following properties: (i) A 4π field of view that enables it to "see" in all directions; (ii) a single center of projection to avoid parallax within the field of view; and (iii) a uniform spatial and angular resolution in order to achieve a uniform sampling of the field of view. Our desig...
Article
Full-text available
Many computer vision algorithms rely on precise es-timates of scene radiances obtained from an image. A simple way to acquire a larger dynamic range of scene radiances is by combining several exposures of the scene. The number of exposures and their values have a dramatic impact on the quality of the combined image. At this point, there exists no p...
Conference Paper
The visual effects of rain are complex. Rain consists of spatially distributed drops falling at high velocities. Each drop refracts and reflects the environment, producing sharp intensity changes in an image. A group of such falling drops creates a complex time varying signal in images and videos. In addition, due to the finite exposure time of the...
Article
Image mosaicing creates a wide field of view image of a scene by fusing data from narrow field images. As a camera moves, each scene point is typically sensed multiple times during frame acquisition. Here we describe generalised mosaicing, which is an approach that enhances this process. An optical component with spatially varying properties is rig...
Conference Paper
Images of outdoor scenes captured in bad weather suffer from poor contrast. Under bad weather conditions, the light reaching a camera is severely scattered by the atmosphere. The resulting decay in contrast varies across the scene and is exponential in the depths of scene points. Therefore, traditional space invariant image processing techniques ar...
Conference Paper
The range of scene depths that appear focused in an image is known as the depth of field (DOF). Conventional cameras are limited by a fundamental trade-off between depth of field and signal-to-noise ratio (SNR). For a dark scene, the aperture of the lens must be opened up to maintain SNR, which causes the DOF to reduce. Also, today’s cameras have D...
Conference Paper
Many computer vision algorithms require searching a set of images for similar patches, which is a very expensive operation. In this work, we compare and evaluate a number of nearest neighbors algorithms for speeding up this task. Since image patches follow very different distri- butions from the uniform and Gaussian distributions that are typically...
Conference Paper
Full-text available
A large photo collection downloaded from the internet spans a wide range of scenes, cameras, and photographers. In this paper we introduce several novel priors for statistics of such large photo collections that are independent of these factors. We then propose that properties of these factors can be recovered by examining the deviation between the...
Conference Paper
Full-text available
We have created the first image search engine based entirely on faces. Using simple text queries such as "smiling men with blond hair and mustaches," users can search through over 3.1 million faces which have been automatically labeled on the basis of several facial attributes. Faces in our database have been extracted and aligned from images downl...
Conference Paper
We present a novel image-based method for compositing real and synthetic objects in the same scene with a high degree of visual realism. Ours is the first technique to allow global illumination and near-field lighting effects between both real and synthetic objects at interactive rates, without needing a geometric and material model of the real sce...
Article
Full-text available
We present a novel image-based method for compositing real and synthetic objects in the same scene with a high degree of visual re- alism. Ours is the first technique to allow global illumination and near-field lighting effects between both real and synthetic objects at interactive rates, without needing a geometric and material model of the real s...
Article
Full-text available
In this paper, we present a complete system for automatic face replacement in images. Our system uses a large library of face images created automatically by downloading images from the internet, extracting faces using face detection software, and aligning each extracted face to a common coordinate system. This library is constructed off-line, once...
Conference Paper
In this paper, we propose a novel imaging system that enables the capture of photos and videos with sparse informational pixels. Our system is based on the projection and detection of 3D optical tags. We use an infrared (IR) projector to project temporally-coded (blinking) dots onto selected points in a scene. These tags are invisible to the human...
Conference Paper
This paper presents a complete framework for creating a speech-enabled avatar from a single image of a person. Our approach uses a generic facial motion model which represents deformations of a prototype face during speech. We have developed an HMM-based facial animation algorithm which takes into account both lexical stress and coarticulation. Thi...
Conference Paper
We present a novel panoramic imaging system which uses a curved mirror as a simple optical attachment to a fish- eye lens. When compared to existing panoramic cameras, our "'cata-fisheye" camera has a simple, compact and inexpensive design, and yet yields high optical performance. It captures the desired panoramic field of view in two parts. The up...
Article
Our eyes are crucial to providing us with an enormous amount of information about our physical world. What is less often explored is the fact that the eye also conveys equally rich information to an external observer. Figure 7.1 shows two images of eyes. One can see a building in the example on the left and a person’s face in the example on the rig...
Article
Full-text available
Imaging of objects under variable lighting directions is an important and frequent practice in computer vision, machine vision, and image-based rendering. Methods for such imaging have traditionally used only a single light source per acquired image. They may result in images that are too dark and noisy, e.g., due to the need to avoid saturation of...
Conference Paper
Multi-sampled imaging is a general framework for using pixels on an image detector to simultaneously sample multiple dimensions of imaging (space, time, spectrum, brightness, polarization, etc.). The mosaic of red, green and blue spectral filters found in most solid-state color cameras is one example of multi-sampled imaging. We briefly describe ho...