Jiabao Li’s research while affiliated with University of California System and other places

Neural rendering techniques, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3D‐GS), have led to significant advancements in scene reconstruction and novel view synthesis (NVS). These methods assume the use of an ideal pinhole model, which is free from lens distortion and optical aberrations. However, fisheye lenses introduce unavoidable aberrations due to their wide‐angle design and complex manufacturing, leading to multi‐view inconsistencies that compromise scene reconstruction quality. In this paper, we propose an end‐to‐end framework that integrates a standard 3D reconstruction pipeline with our lens aberration model to simultaneously calibrate lens aberrations and reconstruct 3D scenes. By modelling the real imaging process and jointly optimising both tasks, our framework eliminates the impact of aberration‐induced inconsistencies on reconstruction. Additionally, we propose a curriculum learning approach that ensures stable optimisation and high‐quality reconstruction results, even in the presence of multiple aberrations. To address the limitations of existing benchmarks, we introduce AbeRec, a dataset composed of scenes captured with lenses exhibiting severe aberrations. Extensive experiments on both existing public datasets and our proposed dataset demonstrate that our method not only significantly outperforms previous state‐of‐the‐art methods on fisheye lenses with severe aberrations but also generalises well to scenes captured by non‐fisheye lenses. Code and datasets are available at https://github.com/CPREgroup/Calibrating‐Fisheye‐Lens‐Aberration‐for‐NVS.

Neural Radiance Fields (NeRF) have shown excellent quality in three-dimensional (3D) reconstruction by synthesizing novel views from multi-view images. However, previous NeRF-based methods do not allow users to perform user-controlled camera setting editing in the scene. While existing works have proposed methods to modify the radiance field, these modifications are limited to camera settings within the training set. Hence, we present Camera Settings Editing of Neural Radiance Fields (CaSE-NeRF) to recover a radiance field from a set of views with different camera settings. In our approach, we allow users to perform controlled camera settings editing on the scene and synthesize the novel view images of the edited scene without re-training the network. The key to our method lies in modeling each camera parameter separately and rendering various 3D defocus effects based on thin lens imaging principles. By following the image processing of real cameras, we implicitly model it and learn gains that are continuous in the latent space and independent of the image. The control of color temperature and exposure is plug-and-play, and can be easily integrated into NeRF-based frameworks. As a result, our method allows for manual and free post-capture control of the viewpoint and camera settings of 3D scenes. Through our extensive experiments on two real-scene datasets, we have demonstrated the success of our approach in reconstructing a normal NeRF with consistent 3D geometry and appearance. Our related code and data is available at https://github.com/CPREgroup/CaSE-NeRF.

Passive Non-Line-Of-Sight (NLOS) imaging techniques have drawn increasing attention for their ability to recover hidden objects that are not visible in observed images. These methods only rely on indirect reflections from the scene surface without the need for expensive measurement equipment or complex measurement processes, in contrast to active NLOS imaging techniques. In this paper, we propose a novel unrolling network based on matrix factorization, which can jointly recover hidden video and light transport matrix. The network has multiple stages, each stage is equivalent to an optimized iteration of matrix factorization. We adopt spatial-variant convolution and neural representation of the light transport matrix in the network, resulting in higher efficiency. To evaluate the proposed method, we compare it with the state-of-the-art hidden video recovery methods on three public datasets. Both qualitative and quantitative comparisons show that our network outperforms the compared method in terms of robustness and accuracy.

p>Spec-NeRF jointly optimizes the degradation parameters and achieves high-quality multi-spectral image reconstruction results at novel views, which only requires a low-cost camera (like a phone camera but in RAW mode) and several off-the-shelf color filters. We also provide real scenarios and synthetic datasets for related studies. Code is available at https://github.com/CPREgroup/SpecNeRF-v2 </p

p>Spec-NeRF jointly optimizes the degradation parameters and achieves high-quality multi-spectral image reconstruction results at novel views, which only requires a low-cost camera (like a phone camera but in RAW mode) and several off-the-shelf color filters. We also provide real scenarios and synthetic datasets for related studies. Code is available at https://github.com/CPREgroup/SpecNeRF-v2 </p

Achieving practical full projector compensation requires the projection display to adapt quickly to textured projection surfaces and unexpected movements without interrupting the display procedure. A possible solution to achieve this involves using a projector and an RGB camera and correcting both color and geometry by directly capturing and analyzing the projected natural image content, without the need for additional patterns. In this study, we approach full projector compensation as a numerical optimization problem and present a physics-based framework that can handle both geometric calibration and radiometric compensation for a Projector-camera system (Procams), using only a few sampling natural images. Within the framework, we decouple and estimate the Procams' factors, such as the response function of the projector, the correspondence between the projector and camera, and the reflectance of projection surfaces. This approach provides an interpretable and flexible solution to adapt to the changes in geometry and reflectance caused by movements. Benefitting from the physics-based scheme, our method guarantees both accurate color calculation and efficient movement and reflectance estimation. Our experimental results demonstrate that our method surpasses other state-of-the-art end-to-end full projector compensation methods, with superior image quality, reduced computational time, lower memory consumption, greater geometric accuracy, and a more compact network architecture. The data and source code are accessible at https://github.com/kylin-leo/FullProjectorCompensation .

Hyperspectral images (HSIs) provide rich spectral information that has been widely used in numerous computer vision tasks. However, their low spatial resolution often prevents their use in applications such as image segmentation and recognition. Fusing low-resolution HSIs with high-resolution RGB images to reconstruct high-resolution HSIs has attracted great research attention recently. In this paper, we propose an unsupervised blind fusion network that operates on a single HSI and RGB image pair and requires neither known degradation models nor any training data. Our method takes full advantage of an unrolling network and coordinate encoding to provide a state-of-the-art HSI reconstruction. It can also estimate the degradation parameters relatively accurately through the neural representation and implicit regularization of the degradation model. The experimental results demonstrate the effectiveness of our method both in simulations and in our real experiments. The proposed method outperforms other state-of-the-art nonblind and blind fusion methods on two popular HSI datasets. Our related code and data is available at https://github.com/CPREgroup/Real-Spec-RGB-Fusion .

p>We propose an unsupervised blind fusion network that operates on a single HSI and RGB image pair and requires neither known degradation models nor any training data. Our method takes full advantage of an unrolling network and coordinate encoding to provide a state-of-the-art HSI reconstruction. It can also estimate the degradation parameters relatively accurately through the neural representation and implicit regularization of the degradation model.</p