No file available
To read the file of this research,
you can request a copy directly from the authors.
GenPC: Zero-shot Point Cloud Completion via 3D Generative Priors
Preprints and early-stage research may not have been peer reviewed yet.
Abstract
Existing point cloud completion methods, which typically depend on predefined synthetic training datasets, encounter significant challenges when applied to out-of-distribution, real-world scans. To overcome this limitation, we introduce a zero-shot completion framework, termed GenPC, designed to reconstruct high-quality real-world scans by leveraging explicit 3D generative priors. Our key insight is that recent feed-forward 3D generative models, trained on extensive internet-scale data, have demonstrated the ability to perform 3D generation from single-view images in a zero-shot setting. To harness this for completion, we first develop a Depth Prompting module that links partial point clouds with image-to-3D generative models by leveraging depth images as a stepping stone. To retain the original partial structure in the final results, we design the Geometric Preserving Fusion module that aligns the generated shape with input by adaptively adjusting its pose and scale. Extensive experiments on widely used benchmarks validate the superiority and generalizability of our approach, bringing us a step closer to robust real-world scan completion.
ResearchGate has not been able to resolve any citations for this publication.
Point cloud completion addresses filling in the missing parts of a partial point cloud obtained from depth sensors and generating a complete point cloud. Although there has been steep progress in the supervised methods on the synthetic point cloud completion task, it is hardly applicable in real-world scenarios due to the domain gap between the synthetic and real-world datasets or the requirement of prior information. To overcome these limitations, we propose a novel self-supervised framework ACL-SPC for point cloud completion to train and test on the same data. ACL-SPC takes a single partial input and attempts to output the complete point cloud using an adaptive closed-loop (ACL) system that enforces the output same for the variation of an input. We evaluate our ACL-SPC on various datasets to prove that it can successfully learn to complete a partial point cloud as the first self-supervised scheme. Results show that our method is comparable with unsupervised methods and achieves superior performance on the real-world dataset compared to the supervised methods trained on the synthetic dataset. Extensive experiments justify the necessity of self-supervised learning and the effectiveness of our proposed method for the real-world point cloud completion task. The code is publicly available from this link.
The captured 3D point clouds by depth cameras and 3D scanners are often corrupted by noise, so point cloud denoising is typically required for downstream applications. We observe that: (i) the scale of the local neighborhood has a significant effect on the denoising performance against different noise levels, point intensities, as well as various kinds of local details; (ii) non-iteratively evolving a noisy input to its noise-free version is non-trivial; (iii) both traditional geometric methods and learning-based methods often lose geometric features with denoising iterations, and (iv) most objects can be regarded as piece-wise smooth surfaces with a small number of features. Motivated by these observations, we propose a novel and task-specific point cloud denoising network, named RePCD-Net, which consists of four key modules: (i) a recurrent network architecture to effectively remove noise; (ii) an RNN-based multi-scale feature aggregation module to extract adaptive features in different denoising stage; (iii) a recurrent propagation layer to enhance the geometric feature perception across stages; and (iv) a feature-aware CD loss to regularize the predictions towards multi-scale geometric details. Extensive qualitative and quantitative evaluations demonstrate the effectiveness and superiority of our method over state-of-the-arts, in terms of noise removal and feature preservation.
Point clouds obtained with 3D scanners or by image‐based reconstruction techniques are often corrupted with significant amount of noise and outliers. Traditional methods for point cloud denoising largely rely on local surface fitting (e.g. jets or MLS surfaces), local or non‐local averaging or on statistical assumptions about the underlying noise model. In contrast, we develop a simple data‐driven method for removing outliers and reducing noise in unordered point clouds. We base our approach on a deep learning architecture adapted from PCPNet, which was recently proposed for estimating local 3D shape properties in point clouds. Our method first classifies and discards outlier samples, and then estimates correction vectors that project noisy points onto the original clean surfaces. The approach is efficient and robust to varying amounts of noise and outliers, while being able to handle large densely sampled point clouds. In our extensive evaluation, both on synthetic and real data, we show an increased robustness to strong noise levels compared to various state‐of‐the‐art methods, enabling accurate surface reconstruction from extremely noisy real data obtained by range scans. Finally, the simplicity and universality of our approach makes it very easy to integrate in any existing geometry processing pipeline. Both the code and pre‐trained networks can be found on the project page (https://github.com/mrakotosaon/pointcleannet). image
Learning and analyzing 3D point cloud with deep networks is challenging due to the sparseness and irregularity of the data. In this paper, we present a data-driven point cloud upsampling technique. The key idea is to learn multi-level features per point, and then expanding them via a multi-branch convolution unit, to implicitly expand the point set in feature space. The expanded feature is then split to a multitude of features, which are then reconstructed to an upsampled point set. Our network is applied at a patch-level, with a joint loss function that encourages the upsampled points to remain on the underlying surface with a uniform distribution. We conduct various experiments using synthesis and scan data to evaluate our method and demonstrate its superiority over some baseline methods and an optimization-based method. The results show that our upsampled results have better uniformity, and sampled closer to the underlying surface.
We propose a data-driven method for recovering miss-ing parts of 3D shapes. Our method is based on a new deep learning architecture consisting of two sub-networks: a global structure inference network and a local geometry refinement network. The global structure inference network incorporates a long short-term memorized context fusion module (LSTM-CF) that infers the global structure of the shape based on multi-view depth information provided as part of the input. It also includes a 3D fully convolutional (3DFCN) module that further enriches the global structure representation according to volumetric information in the input. Under the guidance of the global structure network, the local geometry refinement network takes as input lo-cal 3D patches around missing regions, and progressively produces a high-resolution, complete surface through a volumetric encoder-decoder architecture. Our method jointly trains the global structure inference and local geometry refinement networks in an end-to-end manner. We perform qualitative and quantitative evaluations on six object categories, demonstrating that our method outperforms existing state-of-the-art work on shape completion.
We have created a dataset of more than ten thousand 3D scans of real objects. To create the dataset, we recruited 70 operators, equipped them with consumer-grade mobile 3D scanning setups, and paid them to scan objects in their environments. The operators scanned objects of their choosing, outside the laboratory and without direct supervision by computer vision professionals. The result is a large and diverse collection of object scans: from shoes, mugs, and toys to grand pianos, construction vehicles, and large outdoor sculptures. We worked with an attorney to ensure that data acquisition did not violate privacy constraints. The acquired data was irrevocably placed in the public domain and is available freely at http://redwood-data.org/3dscan.
We present a novel dataset captured from a VW station wagon for use in mobile robotics and autonomous driving research. In total, we recorded 6 hours of traffic scenarios at 10–100 Hz using a variety of sensor modalities such as high-resolution color and grayscale stereo cameras, a Velodyne 3D laser scanner and a high-precision GPS/IMU inertial navigation system. The scenarios are diverse, capturing real-world traffic situations, and range from freeways over rural areas to inner-city scenes with many static and dynamic objects. Our data is calibrated, synchronized and timestamped, and we provide the rectified and raw image sequences. Our dataset also contains object labels in the form of 3D tracklets, and we provide online benchmarks for stereo, optical flow, object detection and other tasks. This paper describes our recording platform, the data format and the utilities that we provide.
We present ControlNet, a neural network architecture to add spatial conditioning controls to large, pretrained textto-image diffusion models. ControlNet locks the productionready large diffusion models, and reuses their deep and robust encoding layers pretrained with billions of images as a strong backbone to learn a diverse set of conditional controls. The neural architecture is connected with “zero convolutions” (zero-initialized convolution layers) that progressively grow the parameters from zero and ensure that no harmful noise could affect the finetuning. We test various conditioning controls, e.g., edges, depth, segmentation, human pose, etc., with Stable Diffusion, using single or multiple conditions, with or without prompts. We show that the training of ControlNets is robust with small (<50k) and large (>1m) datasets. Extensive results show that ControlNet may facilitate wider applications to control image diffusion models.
In this paper, we propose a Transformer encoder-decoder architecture, called PoinTr, which reformulates point cloud completion as a set-to-set translation problem and employs a geometry-aware block to model local geometric relationships explicitly. The migration of Transformers enables our model to better learn structural knowledge and preserve detailed information for point cloud completion. Taking a step towards more complicated and diverse situations, we further propose
AdaPoinTr
by developing an adaptive query generation mechanism and designing a novel denoising task during completing a point cloud. Coupling these two techniques enables us to train the model efficiently and effectively: we reduce training time (by 15x or more) and improve completion performance (over 20%). Additionally, we propose two more challenging benchmarks with more diverse incomplete point clouds that can better reflect real-world scenarios to promote future research. We also show our method can be extended to the scene-level point cloud completion scenario by designing a new geometry-enhanced semantic scene completion framework. Extensive experiments on the existing and newly-proposed datasets demonstrate the effectiveness of our method, which attains 6.53 CD on PCN, 0.81 CD on ShapeNet-55 and 0.392 MMD on real-world KITTI, surpassing other work by a large margin and establishing new state-of-the-arts on various benchmarks. Most notably, AdaPoinTr can achieve such promising performance with higher throughputs and fewer FLOPs compared with the previous best methods in practice.
Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates. We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (≥ 30 fps) novel-view synthesis at 1080p resolution. First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows realtime rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.
How will you repair a physical object with some missings? You may imagine its original shape from previously captured images, recover its overall (global) but coarse shape first, and then refine its local details. We are motivated to imitate the physical repair procedure to address point cloud completion. To this end, we propose a cross-modal shape-transfer dual-refinement network (termed CSDN), a coarse-to-fine paradigm with images of full-cycle participation, for quality point cloud completion. CSDN mainly consists of “shape fusion” and “dual-refinement” modules to tackle the cross-modal challenge. The first module transfers the intrinsic shape characteristics from single images to guide the geometry generation of the missing regions of point clouds, in which we propose IPAdaIN to embed the global features of both the image and the partial point cloud into completion. The second module refines the coarse output by adjusting the positions of the generated points, where the local refinement unit exploits the geometric relation between the novel and the input points by graph convolution, and the global constraint unit utilizes the input image to fine-tune the generated offset. Different from most existing approaches, CSDN not only explores the complementary information from images but also effectively exploits cross-modal data in the
whole
coarse-to-fine completion procedure. Experimental results indicate that CSDN performs favorably against twelve competitors on the cross-modal benchmark.
Most existing point cloud completion methods suffer from the discrete nature of point clouds and the unstructured prediction of points in local regions, which makes it difficult to reveal fine local geometric details. To resolve this issue, we propose SnowflakeNet with snowflake point deconvolution (SPD) to generate complete point clouds. SPD models the generation of point clouds as the snowflake-like growth of points, where child points are generated progressively by splitting their parent points after each SPD. Our insight into the detailed geometry is to introduce a skip-transformer in the SPD to learn the point splitting patterns that can best fit the local regions. The skip-transformer leverages attention mechanism to summarize the splitting patterns used in the previous SPD layer to produce the splitting in the current layer. The locally compact and structured point clouds generated by SPD precisely reveal the structural characteristics of the 3D shape in local patches, which enables us to predict highly detailed geometries. Moreover, since SPD is a general operation that is not limited to completion, we explore its applications in other generative tasks, including point cloud auto-encoding, generation, single image reconstruction, and upsampling. Our experimental results outperform state-of-the-art methods under widely used benchmarks.
This work provides an architecture to enable robotic grasp planning via shape completion. Shape completion is accomplished through the use of a 3D convolutional neural network (CNN). The network is trained on our own new open source dataset of over 440,000 3D exemplars captured from varying viewpoints. At runtime, a 2.5D pointcloud captured from a single point of view is fed into the CNN, which fills in the occluded regions of the scene, allowing grasps to be planned and executed on the completed object. Runtime shape completion is very rapid because most of the computational costs of shape completion are borne during offline training. We explore how the quality of completions vary based on several factors. These include whether or not the object being completed existed in the training data and how many object models were used to train the network. We also look at the ability of the network to generalize to novel objects allowing the system to complete previously unseen objects at runtime. Finally, experimentation is done both in simulation and on actual robotic hardware to explore the relationship between completion quality and the utility of the completed mesh model for grasping.
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.0 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.
This paper describes a general purpose, representation independent
method for the accurate and computationally efficient registration of
3-D shapes including free-form curves and surfaces. The method handles
the full six-degrees of freedom and is based on the iterative closest
point (ICP) algorithm, which requires only a procedure to find the
closest point on a geometric entity to a given point. The ICP algorithm
always converges monotonically to the nearest local minimum of a
mean-square distance metric, and experience shows that the rate of
convergence is rapid during the first few iterations. Therefore, given
an adequate set of initial rotations and translations for a particular
class of objects with a certain level of 'shape complexity', one can
globally minimize the mean-square distance metric over all six degrees
of freedom by testing each initial registration. For examples, a given
'model' shape and a sensed 'data' shape that represents a major portion
of the model shape can be registered in minutes by testing one initial
translation and a relatively small set of rotations to allow for the
given level of model complexity. One important application of this
method is to register sensed data from unfixtured rigid objects with an
ideal geometric model prior to shape inspection. The described method is
also useful for deciding fundamental issues such as the congruence
(shape equivalence) of different geometric representations as well as
for estimating the motion between point sets where the correspondences
are not known. Experimental results show the capabilities of the
registration algorithm on point sets, curves, and surfaces.
Unpaired point cloud completion on real scans using adversarial training
- Xuelin Chen
- Baoquan Chen
- Niloy J Mitra
Xuelin Chen, Baoquan Chen, and Niloy J. Mitra. Unpaired
point cloud completion on real scans using adversarial training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
OpenReview.net, 2020. 2
P2C: self-supervised point cloud completion from single partial clouds
- Ruikai Cui
- Shi Qiu
- Saeed Anwar
- Jiawei Liu
- Chaoyue Xing
- Jing Zhang
- Nick Barnes
Ruikai Cui, Shi Qiu, Saeed Anwar, Jiawei Liu, Chaoyue Xing,
Jing Zhang, and Nick Barnes. P2C: self-supervised point
cloud completion from single partial clouds. In IEEE/CVF
International Conference on Computer Vision, ICCV 2023,
Paris, France, October 1-6, 2023, pages 14305-14314. IEEE,
2023. 2
Objaverse-xl: A universe of 10m+ 3d objects
- Matt Deitke
- Ruoshi Liu
- Matthew Wallingford
- Huong Ngo
- Oscar Michel
- Aditya Kusupati
- Alan Fan
- Christian Laforte
- Vikram Voleti
- Eli Samir Yitzhak Gadre
- Aniruddha Vanderbilt
- Carl Kembhavi
- Georgia Vondrick
- Kiana Gkioxari
- Ludwig Ehsani
- Ali Schmidt
- Farhadi
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo,
Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte,
Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana
Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverse-xl:
A universe of 10m+ 3d objects. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural
Information Processing Systems 2023, NeurIPS 2023, New
Orleans, LA, USA, December 10 -16, 2023, 2023. 3
Diffusion models beat gans on image synthesis
- Prafulla Dhariwal
- Alexander Quinn
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion
models beat gans on image synthesis. In Advances in Neural
Information Processing Systems 34: Annual Conference on
Neural Information Processing Systems 2021, NeurIPS 2021,
December 6-14, 2021, virtual, pages 8780-8794, 2021. 4
Zero-shot point cloud completion via 2d priors. CoRR, abs/2404.06814
- Tianxin Huang
- Zhiwen Yan
- Yuyang Zhao
- Gim Hee Lee
Tianxin Huang, Zhiwen Yan, Yuyang Zhao, and Gim Hee
Lee. Zero-shot point cloud completion via 2d priors. CoRR,
abs/2404.06814, 2024. 1, 2, 3, 4, 8
Point cloud completion with pretrained text-to-image diffusion models
- Yoni Kasten
- Ohad Rahamim
- Gal Chechik
Yoni Kasten, Ohad Rahamim, and Gal Chechik. Point cloud
completion with pretrained text-to-image diffusion models.
In Advances in Neural Information Processing Systems 36:
Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December
10 -16, 2023, 2023. 1, 2, 3, 5, 6, 7
Direct visibility of point sets
- Sagi Katz
- Ayellet Tal
- Ronen Basri
Sagi Katz, Ayellet Tal, and Ronen Basri. Direct visibility of
point sets. ACM Trans. Graph., 26(3):24, 2007. 4
Morphing and sampling network for dense point cloud completion
- Minghua Liu
- Lu Sheng
- Sheng Yang
- Jing Shao
- Shi-Min Hu
Minghua Liu, Lu Sheng, Sheng Yang, Jing Shao, and Shi-Min
Hu. Morphing and sampling network for dense point cloud
completion. In AAAI conference on artificial intelligence,
pages 11596-11603, 2020. 2
Nerf: Representing scenes as neural radiance fields for view synthesis
- Ben Mildenhall
- P Pratul
- Matthew Srinivasan
- Jonathan T Tancik
- Ravi Barron
- Ren Ramamoorthi
- Ng
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik,
Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf:
Representing scenes as neural radiance fields for view synthesis. In Computer Vision -ECCV 2020 -16th European
Conference, Glasgow, UK, August 23-28, 2020, Proceedings,
Part I, pages 405-421. Springer, 2020. 3
Self-supervised point cloud completion via inpainting
- Himangi Mittal
- Brian Okorn
- Arpit Jangid
- David Held
Himangi Mittal, Brian Okorn, Arpit Jangid, and David Held.
Self-supervised point cloud completion via inpainting. In
32nd British Machine Vision Conference 2021, BMVC 2021,
Online, November 22-25, 2021, page 7. BMVA Press, 2021.
2
Dreamfusion: Text-to-3d using 2d diffusion
- Ben Poole
- Ajay Jain
- Jonathan T Barron
- Ben Mildenhall
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.
Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh
International Conference on Learning Representations, ICLR
2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
1, 3
Pointnet: Deep learning on point sets for 3d classification and segmentation
- Hao Charles R Qi
- Kaichun Su
- Leonidas J Mo
- Guibas
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.
Pointnet: Deep learning on point sets for 3d classification and
segmentation. In IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 652-660, 2017. 2
Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis
- Tianchang Shen
- Jun Gao
- Kangxue Yin
- Ming-Yu Liu
- Sanja Fidler
Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and
Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In Advances
in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021,
NeurIPS 2021, December 6-14, 2021, virtual, pages 6087-6101, 2021. 3
Dreamgaussian: Generative gaussian splatting for efficient 3d content creation
- Jiaxiang Tang
- Jiawei Ren
- Hang Zhou
- Ziwei Liu
- Gang Zeng
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang
Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna,
Austria, May 7-11, 2024. OpenReview.net, 2024. 1, 3