ArticlePublisher preview available

Automatic and rapid whole-body 3D shape measurement based on multinode 3D sensing and speckle projection

Optica Publishing Group
Applied Optics
Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract and Figures

Automatic and rapid whole-body 3D shape measurement has attracted extensive attention in recent years and been widely used in many fields. Rapid 3D reconstruction, automatic 3D registration, and dedicated system layout are critical factors to enable an excellent 3D measurement system. In this paper, we present an automatic and rapid whole- body 3D measurement system that is based on multinode 3D sensors using speckle projection. A rapid algorithm for searching homologous point pairs is suggested, which takes advantage of the optimized projective rectification and simplified subpixel matching techniques, leading to an improved time efficiency of 3D reconstruction. Meanwhile, a low-cost automatic system with a flexible setup and an improved calibration strategy are proposed, where system parameters of each node sensor can be simultaneously estimated with the assistance of a cubic and a planar target. Furthermore, an automatic range data registration strategy by employing two aided cameras is investigated. Experiment results show that the presented approach can realize automatic whole-body 3D shape measurement with high efficiency and accuracy.
This content is subject to copyright. Terms and conditions apply.
Automatic and rapid whole-body 3D shape
measurement based on multinode 3D sensing
and speckle projection
JIPING GUO,1,2 XIANG PENG,1,*AMENG LI,2XIAOLI LIU,1AND JIPING YU2
1College of Optoelectronic Engineering, Key Laboratory of Optoelectronics Devices and Systems, Education Ministry of China,
Shenzhen University, Shenzhen 518060, China
2Geometry Metrology and Test Institute, Shenzhen Academy of Metrology & Quality Inspection, Shenzhen 518060, China
*Corresponding author: xpeng@szu.edu.cn
Received 7 August 2017; revised 17 September 2017; accepted 19 September 2017; posted 19 September 2017 (Doc. ID 304142);
published 31 October 2017
Automatic and rapid whole-body 3D shape measurement has attracted extensive attention in recent years and
been widely used in many fields. Rapid 3D reconstruction, automatic 3D registration, and dedicated system
layout are critical factors to enable an excellent 3D measurement system. In this paper, we present an automatic
and rapid whole- body 3D measurement system that is based on multinode 3D sensors using speckle projection. A
rapid algorithm for searching homologous point pairs is suggested, which takes advantage of the optimized pro-
jective rectification and simplified subpixel matching techniques, leading to an improved time efficiency of 3D
reconstruction. Meanwhile, a low-cost automatic system with a flexible setup and an improved calibration strat-
egy are proposed, where system parameters of each node sensor can be simultaneously estimated with the assis-
tance of a cubic and a planar target. Furthermore, an automatic range data registration strategy by employing two
aided cameras is investigated. Experiment results show that the presented approach can realize automatic whole-
body 3D shape measurement with high efficiency and accuracy. © 2017 Optical Society of America
OCIS codes: (150.6910) Three-dimensional sensing; (120.0120) Instrumentation, measurement, and metrology; (150.1488)
Calibration; (120.6150) Speckle imaging.
https://doi.org/10.1364/AO.56.008759
1. INTRODUCTION
Automatic, rapid, and complete 3D data acquisition of whole
object surfaces with high accuracy in real time has attracted
more and more attention from both academic and industrial
circles in recent years. The 3D imaging and measurement tech-
niques based on fringe projection, together with stereovision
have become more popular because of the advantages of non-
contact and fast speed of measurement. Such a technique has
been widely used in many fields, such as industry inspection,
heritage conservation, and body scanning, etc. [14].
Generally, there are three critical issues that should be taken
into consideration when using 3D reconstruction methods
based on fringe projection to realize automatic and rapid mea-
surement of 3D data: (1) quick range data acquisition with high
accuracy, which is necessary for range image reconstruction
of object surface in a single view; (2) design and calibration
of automatic 3D sensing layout; (3) strategy for automatic
registration for range data obtained from multiple views and
a rapid algorithm for 3D modeling.
Owing to the obvious advantages of high data density and
high scanning efficiency, 3D reconstruction based on structural
illumination has been developed rapidly, and plenty of ap-
proaches have been reported recently [57]. According to
the number of fringe patterns utilized, those approaches can
be classified as the multiple-shots method (MSM) and the
single-shot method (SSM). One of the typical MSMs uses
fringe projection and phase-shifting technique, which allow
for high resolution and accuracy of the measurement [810].
But it needs several (more than one) frame patterns to recon-
struct single-range data, making for a low data acquisition rate,
and thus measurement errors can be easily induced by system
vibration. The SSM can reconstruct range data by projecting
only one frame pattern. It thus has an ability to avoid the per-
turbation caused by system vibration, the dynamic displace-
ments or shape changes of the object surface, and therefore,
the SSM has great potential for real-time acquisition of 3D
moving objects [1117]. It is important to note that speckle
pattern projection has good ability to resist ambient light
Research Article Vol. 56, No. 31 / November 1 2017 / Applied Optics 8759
1559-128X/17/318759-10 Journal © 2017 Optical Society of America
... However, it extracts few small-sized speckles. Guo et al. [19] employed the optimized projective rectification and simplified subpixel matching techniques to search for corresponding point pairs rapidly. Yeh et al. [20] utilized Gabor filters, SIFT (Scale-Invariant Feature Transform) and projection to extract features of laser speckle images. ...
Article
Full-text available
Speckle structured light has become a research hotspot due to its ability to acquire target three-dimensional information with single image projection in recent years. To address the challenges of a low number of extracted speckle feature points, high mismatch rate and poor real-time performance in traditional algorithms, as well as the obstacle of requiring expensive annotation data in deep learning-based methods, a lightweight and efficient self-supervised convolutional neural network (CNN) is proposed to achieve high-precision and rapid matching of speckle images. First, to efficiently utilize the speckle projection information, a feature extraction backbone based on the depthwise separable cross convolution blocks is proposed. Second, in the feature detection module, a softargmax detection head is designed to refine the coordinates of speckle feature points to sub-pixel accuracy. In the feature description module, a coarse-to-fine module is presented to further refine matching accuracy. Third, we adopt strategies of transfer learning and self-supervised learning to improve the generalization and feature representation capabilities of the model. Data augmentation and real-time training techniques are used to improve the robustness of the model. The experimental results show that the proposed method achieves a mean matching accuracy of 91.62% for speckle feature points on the pilot’s helmet, with mere 0.95% mismatch rate. The full model runs at 42ms for a speckle image pair on an RTX 3060.
... Although this approach can partially reduce device complexity, it still requires a highly precise and large-range circular track. Another time-multiplexing solution is to rotate the object so that the measuring device can collect three-dimensional data from different regions of the sample surface [3,[16][17][18][19]. Qian et al. achieved full-surface digitization of a sample fixed on a rotating stage using fringe projection profilometry (FPP) [19]. ...
Article
Full-text available
The digitization of objects’ full surfaces finds widespread applications in fields such as virtual reality, art and design, and medical and biological sciences. For the realization of three-dimensional full-surface digitization of objects within complex sceneries, we propose a straightforward, efficient, and robust panoramic three-dimensional optical digitization system. This system contains a laser-based optical three-dimensional measurement system and a bi-mirror. By integrating mirrors into the system, we enable the illumination of the object from all angles using the projected laser beam in a single scanning process. Moreover, the main camera employed in the system can acquire three-dimensional information of the object from several different viewpoints. The rotational scanning method enhances the efficiency and applicability of the three-dimensional scanning process, enabling the acquisition of surface information of large-scale objects. After obtaining the three-dimensional data of the sample from different viewpoints using laser triangulation, mirror reflection transformation was employed to obtain the full-surface three-dimensional data of the object in the global coordinate system. The proposed method has been subjected to precision and validity experiments using samples with different surface characteristics and sizes, resulting in the demonstration of its capability for achieving correct three-dimensional digitization of the entire surface in diverse complex sceneries.
... aerospace design, manufacturing, verification, validation, and services). Guo et al [98] proposed an automated data-driven measurement pipeline for rapid 3D shape reconstruction including characterisation of the selected optical measuring instrument, multiple measurement acquisitions and automated full registration. Their optimised algorithm could find corresponding homologous point pairs in range data from different view angles, using knowledge extracted from the rectified images. ...
Article
Full-text available
Manufacturing has recently experienced increased adoption of optimised and fast solutions for checking product quality during fabrication, allowing for manufacturing times and costs to be significantly reduced. Due to the integration of machine learning algorithms, advanced sensors and faster processing systems, smart instruments can autonomously plan measurement pipelines, perform decisional tasks and trigger correctional actions as required. In this paper, we summarise the state of the art in smart optical metrology, covering the latest advances in integrated intelligent solutions in optical coordinate and surface metrology, respectively for the measurement of part geometry and surface texture. Within this field, we include the use of a priori knowledge and implementation of machine learning algorithms for measurement planning optimisation. We also cover the development of multi-sensor and multi-view instrument configurations to speed up the measurement process, as well as the design of novel feedback tools for measurement quality evaluation.
... The projector is not for texture stereo; thus, the method combines with the high-cost matching of PatchMatch [14], which is a randomized algorithm and is not suitable for ASIC and field-programmable gate array (FPGA). Various applications [15][16][17][18] used random speckle pattern [4][5][6], which consists of blobs with a fixed pixel size more than 1 pixel. The randomization is also used to support coded projections [19,20]. ...
Article
Full-text available
A stereo camera is fundamental for 3D sensing. We can only use simple stereo algorithms for real-time applications; however, it has low accuracy. Adding texture by projection is a solution to the problem. The system is called projected texture stereo. Projecting light patterns is essential and deeply related to the matching algorithm in the system. This article proposes a projected pattern and a new texture stereo camera system with a suitable stereo matching algorithm. Experimental results show that our system is superior to Intel RealSense, the commercially successful projected texture stereo camera.
... Using the single-shot speckle structure [17][18][19], the acquisition time of the stereo image pair was short, but the real-time reconstruction effect was usually not achieved. Although the literature [27] achieved a real-time reconstruction frequency of 30 fps, the accuracy was only 0.55 mm, which is far from the accuracy of the proposed method. Aiming at 3D face acquisition equipment using a fixed speckle projector, the method in this study implements real-time image acquisition and real-time 3D face reconstruction. ...
Article
Full-text available
The reconstruction of 3D face data is widely used in the fields of biometric recognition and virtual reality. However, the rapid acquisition of 3D data is plagued by reconstruction accuracy, slow speed, excessive scenes and contemporary reconstruction-technology. To solve this problem, an accurate 3D face-imaging implementation framework based on coarse-to-fine spatiotemporal correlation is designed, improving the spatiotemporal correlation stereo matching process and accelerating the processing using a spatiotemporal box filter. The reliability of the reconstruction parameters is further verified in order to resolve the contention between the measurement accuracy and time cost. A binocular 3D data acquisition device with a rotary speckle projector is used to continuously and synchronously acquire an infrared speckle stereo image sequence for reconstructing an accurate 3D face model. Based on the face mask data obtained by the high-precision industrial 3D scanner, the relationship between the number of projected speckle patterns, the matching window size, the reconstruction accuracy and the time cost is quantitatively analysed. An optimal combination of parameters is used to achieve a balance between reconstruction speed and accuracy. Thus, to overcome the problem of a long acquisition time caused by the switching of the rotary speckle pattern, a compact 3D face acquisition device using a fixed three-speckle projector is designed. Using the optimal combination parameters of the three speckles, the parallel pipeline strategy is adopted in each core processing unit to maximise system resource utilisation and data throughput. The most time-consuming spatiotemporal correlation stereo matching activity was accelerated by the graphical processing unit. The results show that the system achieves real-time image acquisition, as well as 3D face reconstruction, while maintaining acceptable systematic precision.
... To deal with this issue, numerous encoding strategies of structured patterns have emerged [1] to enrich or add surface texture and thus simplify the correspondence problem. Among the structured patterns employed, the random speckle pattern is widely investigated and applied [5][6][7][8][9] due to its unique capabilities to determine the correspondences. Especially, speckle-patternbased 3D imaging does well without any precise, costly, and complex phase-shifting projection devices [10] in comparison with the sinusoidal fringe pattern projection (SFP) and phaseshifting (PS)-based 3D imaging technology, which greatly facilitates the development of cost-effective and innovative projection devices of 3D imaging technology. ...
Article
Full-text available
In this paper, a compact, cost-effective, and fast rotary speckle projector (RSP) is designed and manufactured for high-precision three-dimensional (3D) face data acquisition. Compared with the common speckle projectors, RSP uses a simple speckle pattern design method and has a good performance in high-speed projection and compact structure, which allows a flexible balance between measurement accuracy and time cost in a real acquisition task. Using a carefully designed rotation angle of the speckle mask, temporally and spatially non-correlative speckle patterns in the measurement volume can be generated. The rotation angle of the speckle mask is carefully checked and optimally selected via detailed theoretical analysis, simulation, and experiments to ensure 3D reconstruction accuracy across the reconstruction area. Subsequently, a binocular 3D face imaging system composed of the RSP and two cameras is constructed. With captured stereo speckle image pairs, we adopted our previously well-established spatial-temporal correlation method to determine the disparity. The accuracy of the 3D face imaging system was verified by using a real face mask, which is standardized by a certified, high-precision industrial 3D scanner. The real face data collection under various expressions has demonstrated that the proposed system also has a good performance for 3D face imaging in dynamic scenes.
Article
A structured-light projection system was designed for microscale objects with surface heights that ranged from tens to hundreds of microns. The system was composed of a universal projector and microscope system that supported editing the attributes of structured-light patterns in real-time and was capable of projecting microscale patterns. On this basis, reconstructing the metal surfaces of microscale objects based on grid patterns of structured light was investigated, the internal and external parameters of microscope vision and projection systems were calibrated, and an image algorithm for grid-node detection was designed. The results indicated that the proposed method successfully reconstructed the three-dimensional (3D) surface of microscale objects, and the reconstruction results were consistent with the original surfaces. With 95% confidence, the reconstruction precision in the X- and Y -directions was approximately ±4.0 μ m and in the Z -direction was approximately ±7.5 μ m. The designed system and the proposed method were suitable for 3D-shape measurement of microstructures in microscopic fields and can be adapted to meet a broader range of applications, as compared to current methods.
Article
Rectification is a critical process for speckle project-ion systems to ensure reliable image matching in producing a depth map. However, few studies have discussed it up to the present. In this paper, we present a flexible but robust approach to address this issue. It mainly consists of two steps: calibration and rectification. The calibration step takes one or several pairs of images of the projected speckle pattern on a plane to estimate a unit vector indicating the direction of the line where emitter center lies on. The rectification step virtually rotate the camera based on this vector to obtain rectified images. Given camera intrinsic parameters, this approach only needs a minimum configuration of two freely captured speckle images of a planar surface to complete system rectification, which makes it particularly suited for the case with no access to the internal projected speckle pattern. Elaborate experiments on real speckle images as well as synthetic images from different camera poses are conducted, demonstrating the effectiveness and robustness of the proposed approach. Furthermore, we also demonstrate that when the internal pattern is recovered from the rectified images, it reaches an accuracy comparable with the ones from the best state-of-the-art methods in performing a stereo matching, but with a much simpler operation to obtain.
Article
Using a metrology system simulation approach, an algorithm is presented to determine the best position for a robot mounted 3D vision system. Point cloud data is simulated, taking into account sensor performance, to create a ranked list of the best camera positions. These can be used by a robot to autonomously determine the most advantageous camera position for locating a target object. The algorithm is applied to an Ensenso active stereo 3D camera. Results show that when used in combination with a RANSAC object recognition algorithm, it increased positional precision by two orders of magnitude, from worst to best case.
Article
A recently proposed phase unwrapping method using geometric constraints performs well without requiring additional camera, more patterns or global search. The major limitation of this technique is the confined measurement depth range (MDR) within 2π in phase domain. To enlarge the MDR, this paper proposes using color fringes for three-dimensional (3D) shape measurement. Each six fringe periods encoded with six different colors are treated as one group. The local order within one group can be identified with reference to the color distribution. Then the phase wrapped period-by-period is converted into the phase wrapped group-by-group. The geometric constraints of the fringe projection system are used to determine the group order. Such that the MDR is extended from 2π to 12π by six times. Experiment results demonstrate the success of the proposed method to measure two isolated objects with large MDR.
Conference Paper
This paper presents a new Phase Measuring Deflectometry (PMD) method to measure specular object having discontinuous surfaces. A mathematical model is established to directly relate absolute phase and depth, instead of phase and gradient. Based on the model, a hardware measuring system has been set up, which consists of a beam splitter to change the optical path, and two LCD screens to display the same sinusoidal fringe patterns. By using model-based and machine vision method, system calibration is accomplished to provide the required parameters and conditions. The verification tests are given to evaluate the effectiveness of the developed system. The 3D shape of an artificial step having multiple specular surfaces and a concave mirror has been measured. Initial experimental results show that the proposed measurement method can obtain 3D shape of specular objects with discontinuous surface effectively.
Article
This paper presents a method that recovers high-quality 3D absolute coordinates point by point with only five binary patterns. Specifically, three dense binary dithered patterns are used to compute the wrapped phase; and the average intensity is combined with two additional binary patterns to determine fringe order pixel by pixel in phase domain. The wrapped phase is temporarily unwrapped point by point by referring to the fringe order. We further developed a computational framework to reduce random noise impact due to dithering, defocusing and random noise. Since only five binary fringe patterns are required to recover one 3D frame, extremely high speed 3D shape measurement can be achieved. For example, we developed a system that captures 2D images at 3333 Hz, and thus performs 3D shape measurement at 667 Hz.
Article
It is a challenging issue to get satisfied results in terms of 3D imaging for shiny surface with fringe projection profilometry (FPP), as the wide variation of surface reflectance for shiny surface will lead to bad exposure, which requires the high dynamic range imaging (HDRI) technique. HDRI with monochromatic illumination and single-shot raw data of the color camera is presented in this paper. From the single-shot raw data, 4 monochrome sub-images corresponding to R, G, G and B channels can be separated respectively. After the attenuation ratios between R&G, G&B channels are calibrated, an image with higher dynamic range can be synthesized with the 4 sub-images, which can help to avoid the impact of bad exposure and improve the accuracy of phase calculation. Experiments demonstrate the validity of proposed method for shiny surface.
Article
This paper presents a system for positioning markers and tracking the pose of a rigid object with 6 degrees of freedom in real-time using 3D digital image correlation, with two examples for medical imaging applications. Traditional DIC method was improved to meet the requirements of the real-time by simplifying the computations of integral pixel search. Experiments were carried out and the results indicated that the new method improved the computational efficiency by about 4–10 times in comparison with the traditional DIC method. The system was aimed for orthognathic surgery navigation in order to track the maxilla segment after LeFort I osteotomy. Experiments showed noise for the static point was at the level of 10⁻³ mm and the measurement accuracy was 0.009 mm. The system was demonstrated on skin surface shape evaluation of a hand for finger stretching exercises, which indicated a great potential on tracking muscle and skin movements.
Article
This paper proposes a method to potentially conquer one of the challenges in the optical metrology community: optically measuring three-dimensional (3D) objects with high surface contrast. We discover that for digitally equally phase-shifted fringe patterns, if the fringe period P is an even number, the , (k=1, 2, 3, …) step algorithm can accurately recover phase even if the fringe patterns are saturated; and if P is an odd number, step algorithm can also accurately recover phase even if the fringe patterns are saturated. This finding leads to a novel method to optically measure shiny surfaces, where the saturation due to surface shininess could be substantially alleviated. Both simulations and experiments successfully verified the proposed method.