October 2023
·
12 Reads
·
9 Citations
This page lists works of an author who doesn't have a ResearchGate profile or hasn't added the works to their profile yet. It is automatically generated from public (personal) data to further our legitimate goal of comprehensive and accurate scientific recordkeeping. If you are this author and want this page removed, please let us know.
October 2023
·
12 Reads
·
9 Citations
April 2023
·
26 Reads
Because of the diversity in lighting environments, existing illumination estimation techniques have been designed explicitly on indoor or outdoor environments. Methods have focused specifically on capturing accurate energy (e.g., through parametric lighting models), which emphasizes shading and strong cast shadows; or producing plausible texture (e.g., with GANs), which prioritizes plausible reflections. Approaches which provide editable lighting capabilities have been proposed, but these tend to be with simplified lighting models, offering limited realism. In this work, we propose to bridge the gap between these recent trends in the literature, and propose a method which combines a parametric light model with 360{\deg} panoramas, ready to use as HDRI in rendering engines. We leverage recent advances in GAN-based LDR panorama extrapolation from a regular image, which we extend to HDR using parametric spherical gaussians. To achieve this, we introduce a novel lighting co-modulation method that injects lighting-related features throughout the generator, tightly coupling the original or edited scene illumination within the panorama generation process. In our representation, users can easily edit light direction, intensity, number, etc. to impact shading while providing rich, complex reflections while seamlessly blending with the edits. Furthermore, our method encompasses indoor and outdoor environments, demonstrating state-of-the-art results even when compared to domain-specific methods.
September 2022
·
8 Reads
·
21 Citations
June 2022
·
9 Reads
·
7 Citations
April 2022
·
26 Reads
We propose a method to extrapolate a 360{\deg} field of view from a single image that allows for user-controlled synthesis of the out-painted content. To do so, we propose improvements to an existing GAN-based in-painting architecture for out-painting panoramic image representation. Our method obtains state-of-the-art results and outperforms previous methods on standard image quality metrics. To allow controlled synthesis of out-painting, we introduce a novel guided co-modulation framework, which drives the image generation process with a common pretrained discriminative model. Doing so maintains the high visual quality of generated panoramas while enabling user-controlled semantic content in the extrapolated field of view. We demonstrate the state-of-the-art results of our method on field of view extrapolation both qualitatively and quantitatively, providing thorough analysis of our novel editing capabilities. Finally, we demonstrate that our approach benefits the photorealistic virtual insertion of highly glossy objects in photographs.
August 2021
·
21 Reads
·
34 Citations
ACM Transactions on Graphics
High-quality denoising of Monte Carlo low-sample renderings remains a critical challenge for practical interactive ray tracing. We present a new learning-based denoiser that achieves state-of-the-art quality and runs at interactive rates. Our model processes individual path-traced samples with a lightweight neural network to extract per-pixel feature vectors. The rest of our pipeline operates in pixel space. We define a novel pairwise affinity over the features in a pixel neighborhood, from which we assemble dilated spatial kernels to filter the noisy radiance. Our denoiser is temporally stable thanks to two mechanisms. First, we keep a running average of the noisy radiance and intermediate features, using a per-pixel recursive filter with learned weights. Second, we use a small temporal kernel based on the pairwise affinity between features of consecutive frames. Our experiments show our new affinities lead to higher quality outputs than techniques with comparable computational costs, and better high-frequency details than kernel-predicting approaches. Our model matches or outperfoms state-of-the-art offline denoisers in the low-sample count regime (2--8 samples per pixel), and runs at interactive frame rates at 1080p resolution.
August 2021
·
31 Reads
·
9 Citations
ACM Transactions on Graphics
November 2020
·
45 Reads
·
28 Citations
Most 3D reconstruction methods may only recover scene properties up to a global scale ambiguity. We present a novel approach to single view metrology that can recover the absolute scale of a scene represented by 3D heights of objects or camera height above the ground as well as camera parameters of orientation and field of view, using just a monocular image acquired in unconstrained condition. Our method relies on data-driven priors learned by a deep network specifically designed to imbibe weakly supervised constraints from the interplay of the unknown camera with 3D entities such as object heights, through estimation of bounding box projections. We leverage categorical priors for objects such as humans or cars that commonly occur in natural images, as references for scale estimation. We demonstrate state-of-the-art qualitative and quantitative results on several datasets as well as applications including virtual object insertion. Furthermore, the perceptual quality of our outputs is validated by a user study.
July 2020
·
287 Reads
·
15 Citations
Most 3D reconstruction methods may only recover scene properties up to a global scale ambiguity. We present a novel approach to single view metrology that can recover the \emph{absolute} scale of a scene represented by 3D heights of objects or camera height above the ground as well as camera parameters of orientation and field of view, using just a monocular image acquired in unconstrained condition. Our method relies on data-driven priors learned by a deep network specifically designed to imbibe weakly supervised constraints from the interplay of the unknown camera with 3D entities such as object heights, through estimation of bounding box projections. We leverage categorical priors for objects such as humans or cars that commonly occur in natural images, as references for scale estimation. We demonstrate state-of-the-art qualitative and quantitative results on several datasets as well as applications including virtual object insertion. Furthermore, the perceptual quality of our outputs is validated by a user study.
July 2020
·
275 Reads
Most 3D reconstruction methods may only recover scene properties up to a global scale ambiguity. We present a novel approach to single view metrology that can recover the absolute scale of a scene represented by 3D heights of objects or camera height above the ground as well as camera parameters of orientation and field of view, using just a monocular image acquired in unconstrained condition. Our method relies on data-driven priors learned by a deep network specifically designed to imbibe weakly supervised constraints from the interplay of the unknown camera with 3D entities such as object heights, through estimation of bounding box projections. We leverage categorical priors for objects such as humans or cars that commonly occur in natural images, as references for scale estimation. We demonstrate state-of-the-art qualitative and quantitative results on several datasets as well as applications including virtual object insertion. Furthermore, the perceptual quality of our outputs is validated by a user study.
... Several methods have been utilised to represent these conditions such as regression-based methods, like Spherical Harmonics (SH) [22] and Spherical Gaussians (SG) [36], and Image Based Lighting (IBL) [11] methods that render lighting from an Equirectangular Panorama (ERP) High Dynamic Range image (HDR(I)). IBL has become the leading way to represent lighting conditions [10,26,30,33,39,40] due to its ability to capture high-frequency textures, as well as global illumination, meaning it can be used to light a range of surfaces from rough diffuse to mirror reflective. The current trends in IBL and illumination estimation are to increase the resolution, details and accuracy of the generated ERP images [10,26,39,40]. ...
October 2023
... Not adapting a neural network architecture to work with the ERP format can lead to artefacts such as obvious seams at the side borders of the image and badly generated objects at the top and bottom (poles) of the ERP where it is most warped. In the field of illumination estimation, only a few models have made adjustments [2,9,10,33]. These few approaches are typically limited to either changes to the loss functions or changes to the image at network inference. ...
September 2022
... Textures and physics-based rendering (PBR) materials are crucial for representing the visual appearance of materials in computer-generated images (CGI) for animation, games, and other arts [2][3][4][5][6][7][8][9][10][11][12][13][14] . Textures and PBR materials also become increasingly important for generating virtual worlds and synthetic data for training artificial intelligence systems [15][16][17][18][19][20][21][22][23][24] . Materials are usually represented either as 2D textures for 2D scenes or using physics-based rendering (PBR) materials (also called SVBRDF materials) which describe the distribution of material properties as a set of maps. ...
June 2022
... Chaitanaya et al. [9] propose a recurrent U-Net denoiser which directly outputs the denoised frame and can be executed at interactive frame rates on an Nvidia Titan X. Hasselgren et al. [13] use a recurrent U-Net to generate a feature map that is used for spatiotemporal adaptive sampling and to construct per-pixel linear denoising kernels. To reduce the cost associated with constructing per-pixel kernels, Işik et al. [16] generate per-pixel features that are used to construct adaptive dilated kernels. Balint et al. [4] use a series of lightweight U-Nets to construct a low-pass pyramid filter, where the kernel constructor is also trained to separate the input radiance between pyramidal layers as an alternative to classical downsampling and upsampling approaches, which are often prone to aliasing. ...
August 2021
ACM Transactions on Graphics
... Meng et al. [19] projected the noisy input image onto the bilateral grid based on the guide image learned by the neural network and then, sliced the grid to obtain the denoised images. Isik et al. [11] proposed a filtering algorithm by computing a pairwise affinity to quantify the relationship between per-pixel deep features. Fan et al. [6] learned lightweight importance maps and constructed multi-scale filtering kernels to reduce the time cost of the kernel prediction method. ...
August 2021
ACM Transactions on Graphics
... Whether estimating the scale of an input image, controlling the scale during the generation process, or calculating the scale of the results, how to conduct indoor measurement has become an urgent problem to be solved. Existing methods, like (1.) photogrammetry (Kutlu and Soyluk, 2024), (2.) estimation from given objects (Criminisi, Reid and Zisserman, 2000), and (3.) geometric camera calibration (Zhu et al., 2020), either need capturing images from different angles by expensive devices or prior geometric knowledge or accurate camera calibration. Such approaches are incompatible with our AI-aided interior design workflow, which requires real-time measurement from any given single interior image. ...
November 2020
... Despite there are automated camera calibration algorithms based on roadside monitoring cameras [9], [10], such methods mainly rely on vanishing point detection of vehicles and assume that the vehicle trajectory approximates a straight line or the road is straight, making the camera calibration accuracy susceptible to the influence of vehicle trajectories. In addition to the traditional auto camera calibration methods, researchers have proposed numerous deep learning-based camera calibration methods in recent years [11]- [14]. These methods are trained on largescale public datasets and offer advantages such as being unaffected by traffic flow and not requiring any manual input or prior scene information. ...
July 2020
... These methods facilitate 3D reconstruction and multi-view analysis but remain challenged by the demands of real-time processing and dynamic scenes. Recent developments have also explored joint estimation of intrinsic and extrinsic parameters, as demonstrated by [13], [3], [16], [23]. These approaches promise more integrated calibration processes through deep learning, highlighting the potential for efficient real-time calibration. ...
June 2018
... Classic methods leverage reference image components, including calibration grids [61] or vanishing points [11], to estimate camera parameters. Recently, data-driven approaches have been proposed to use deep neural networks to infer the focal length [15,50] and camera poses [19,28,54] directly from in-the-wild images, or to use dense representation [16] to encode camera parameters for a more robust estimation. In contrast, our method ORG jointly estimates intrinsic and extrinsic camera parameters together with object geometry and ground positions, achieving a self-contained pipeline for 3D-aware image editing and reconstruction. ...
December 2017