PointSea: Point Cloud Completion via Self-structure Augmentation

Abstract

Point cloud completion is a fundamental yet not well-solved problem in 3D vision. Current approaches often rely on 3D coordinate information and/or additional data (e.g., images and scanning viewpoints) to fill in missing parts. Unlike these methods, we explore self-structure augmentation and propose PointSea for global-to-local point cloud completion. In the global stage, consider how we inspect a defective region of a physical object, we may observe it from various perspectives for a better understanding. Inspired by this, PointSea augments data representation by leveraging self-projected depth images from multiple views. To reconstruct a compact global shape from the cross-modal input, we incorporate a feature fusion module to fuse features at both intra-view and inter-view levels. In the local stage, to reveal highly detailed structures, we introduce a point generator called the self-structure dual-generator. This generator integrates both learned shape priors and geometric self-similarities for shape refinement. Unlike existing efforts that apply a unified strategy for all points, our dual-path design adapts refinement strategies conditioned on the structural type of each point, addressing the specific incompleteness of each point. Comprehensive experiments on widely-used benchmarks demonstrate that PointSea effectively understands global shapes and generates local details from incomplete input, showing clear improvements over existing methods. Our code is available at https://github.com/czvvd/SVDFormer_PointSea.

Full text

International Journal of Computer Vision
https://doi.org/10.1007/s11263-025-02400-y
PointSea: Point Cloud Completion via Self-structure Augmentation
Zhe Zhu1·Honghua Chen1·Xing He1·Mingqiang Wei1
Received: 8 July 2024 / Accepted: 17 February 2025
© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2025
Abstract
Point cloud completion is a fundamental yet not well-solved problem in 3D vision. Current approaches often rely on 3D
coordinate information and/or additional data (e.g., images and scanning viewpoints) to fill in missing parts. Unlike these
methods, we explore self-structure augmentation and propose PointSea for global-to-local point cloud completion. In the
global stage, consider how we inspect a defective region of a physical object, we may observe it from various perspectives for a
better understanding. Inspired by this, PointSea augments data representation by leveraging self-projected depth images from
multiple views. To reconstruct a compact global shape from the cross-modal input, we incorporate a feature fusion module
to fuse features at both intra-view and inter-view levels. In the local stage, to reveal highly detailed structures, we introduce
a point generator called the self-structure dual-generator. This generator integrates both learned shape priors and geometric
self-similarities for shape refinement. Unlike existing efforts that apply a unified strategy for all points, our dual-path design
adapts refinement strategies conditioned on the structural type of each point, addressing the specific incompleteness of each
point. Comprehensive experiments on widely-used benchmarks demonstrate that PointSea effectively understands global
shapes and generates local details from incomplete input, showing clear improvements over existing methods. Our code is
available at https://github.com/czvvd/SVDFormer_PointSea.
Keywords PointSea ·Point cloud completion ·Self-structure augmentation ·Cross-modal fusion
1 Introduction
Raw-captured point clouds are often incomplete due to fac-
tors like occlusion, surface reflectivity, and limited scanning
range (see Fig. 2). Before they can be used in downstream
applications (e.g., digital twin), they need to be faithfully
completed, a process known as point cloud completion.
Recent years have witnessed significant progress in this
field (Yuan et al., 2018; Huang et al., 2020; Zhang et al., 2020;
Yu et al., 2021; Xiang et al., 2023; Yan et al., 2022; Zhang et
Communicated by Wanli Ouyang.
BZhe Zhu
zhuzhe0619@nuaa.edu.cn
Honghua Chen
chenhonghuacn@gmail.com
Xing He
hexing@nuaa.edu.cn
Mingqiang Wei
mqwei@nuaa.edu.cn
1Nanjing University of Aeronautics and Astronautics, Nanjing,
China
al., 2022a; Tang et al., 2022; Zhou et al., 2022; Zhang et al.,
2023d; Yu et al., 2023a; Wang et al., 2022a). However, the
sparsity and large structural incompleteness of point clouds
still limit their ability to produce satisfactory results. There
are two primary challenges in point cloud completion:
•Crucial semantic parts are often absent in the partial
observations.
•Detailed structures cannot be effectively recovered.
The first challenge leads to a vast solution space for point-
based networks (Yuan et al., 2018; Xiang et al., 2023;Yuet
al., 2021; Zhou et al., 2022) to robustly locate missing regions
and create a partial-to-complete mapping. Some alternative
methods attempt to address this issue by incorporating addi-
tional color images (Zhang et al., 2021b; Aiello et al., 2022;
Zhu et al., 2024) or viewpoints (Zhang et al., 2022a; Gong
et al., 2021;Fuetal.,2023). However, the paired images
with well-calibrated camera parameters are hard to obtain,
as well as the scanning viewpoints. To resolve the second
challenge, some recent approaches (Xiang et al., 2023;Yan
et al., 2022) utilize skip-connections between multiple refine-
123
Content courtesy of Springer Nature, terms of use apply. Rights reserved.