Low-cost data-driven 3D reconstruction and its applications

Abstract

This paper describes two approaches for estimating human 3D shapes (i.e. full bodies or feet) using a regular smartphone or just entering a set of parameters (e.g. age, gender and self-taken measurements). The proposed approaches are based on data-driven 3D reconstructions, using parameterised shape spaces created from large 3D body or feet databases. The reconstruction algorithm finds the combination of shape parameters that best matches either the silhouettes extracted from the images or the body measurements entered. Despite not being actual body scanners, these solutions are easy-to-use and can provide enough accuracy for applications such as virtual try-on, made-to-measure or size allocation of certain types of wearables. Moreover, they can be distributed to the final consumer or to the points of sale at a really reduced cost (or even for free), thus overcoming the main barriers to the massive spreading of body scanners’ use to e-commerce, retail shops, new production pipelines or new business models. In order to illustrate these technologies, some examples of application to different contexts are provided, namely virtual worlds, e-commerce and personalisation.

Full text

Low-Cost Data-Driven 3D Reconstruction and its Applications
Alfredo BALLESTER*, Eduardo PARRILLA, Julio A. VIVAS, Ana PIÉROLA, Jordi URIEL,
Sergio A. PUIGCERVER, Paola PIQUERAS, Clara SOLVES-CAMALLONGA,
Marisol RODRÍGUEZ, Juan C. GONZÁLEZ, Sandra ALEMANY
Instituto de Biomecánica de Valencia, Universidad Politécnica de Valencia, Valencia, Spain
DOI: 10.15221/15.184 http://dx.doi.org/10.15221/15.184
Abstract
This paper describes two approaches for estimating human 3D shapes (i.e. full bodies or feet) using a
regular smartphone or just entering a set of parameters (e.g. age, gender and self-taken
measurements). The proposed approaches are based on data-driven 3D reconstructions, using
parameterised shape spaces created from large 3D body or feet databases. The reconstruction
algorithm finds the combination of shape parameters that best matches either the silhouettes extracted
from the images or the body measurements entered. Despite not being actual body scanners, these
solutions are easy-to-use and can provide enough accuracy for applications such as virtual try-on,
made-to-measure or size allocation of certain types of wearables. Moreover, they can be distributed to
the final consumer or to the points of sale at a really reduced cost (or even for free), thus overcoming
the main barriers to the massive spreading of body scanners’ use to e-commerce, retail shops, new
production pipelines or new business models. In order to illustrate these technologies, some examples
of application to different contexts are provided, namely virtual worlds, e-commerce and
personalisation.
Keywords: 3D, foot, body, scanning, shape, measurement, data-driven, PCA, e-commerce,
personalization, size allocation, configuration, virtual try-on, retail, clothing, footwear, orthotics, insoles,
transports, ergonomics, low-cost, smartphone, app, booth
1. Introduction
The access to the 3D representation of people’s body shape has multiple applications to consumer
goods which performance is related to body dimensions or shape. This is the case of wearables such
as clothing, footwear, headgear, orthotics; or equipment/environments such as furniture, transports or
workstations. Some of the existing and potential applications of 3D human representations include
personalisation, virtual try-on or size allocation for wearables or product configuration/adjustment for
equipment/environments.
However, the cost of 3D scanners is high; the devices are too bulky for homes and retail stores; and its
proper use requires expertise to get the relevant parameters from the 3D object (e.g. body
measurements). These three barriers are currently hindering the massive spreading of 3D scanners as
consumer good or as typical in-store appliance.
This paper focuses on two approaches for estimating accurately full bodies and feet. The first
approach uses 2-3 calibrated images as input while the second one uses a small set of measurements
of the user as input. Both approaches are underpinned by data-driven 3D reconstructions, using
parameterised shape spaces created from large 3D human body or feet databases.
In particular, this paper presents four 3D data-driven reconstruction developments:
(a) 3D feet reconstruction obtained from three images per foot gathered using a mobile app [1]
(b) 3D feet reconstruction obtained from one image per foot gathered using a special booth
(c) 3D body reconstruction obtained from two images gathered using a mobile app
(d) 3D body reconstruction obtained from age, gender and 4 body measurements
*alfredo.ballester@ibv.upv.es; +34 610 562 532; http://anthropometry.ibv.org
6th International Conference on 3D Body Scanning Technologies, Lugano, Switzerland, 27-28 October 2015

2. The parameterised shape spaces used
All methods followed in this paper are based on the “space of human body shapes” concept and on the
methods introduced by Allen et al. [2]. For our research work we created different shape spaces using
range scan data gathered by IBV in different projects and research works.
2.1. Foot shape space
To create the feet shape space we used a database of over 700 male and female feet scans gathered
using Infoot scanner from I-Ware. They were registered using Allen et al. [2] method with a template
made up of 5.000 vertices with five foot landmarks (i.e. 1st and 2nd toe tips; 1st and 5th metatarsal heads;
and pternion; fig. 1). This provided a homologous database with one-to-one correspondence across all
feet. Database parameterisation was achieved by conducting Principal Component Analysis (PCA) to
the homologous database, from which the first 40 components were retained. This provided a
data-driven way to generate a vast array of different feet based on 40 parameters.
Fig. 1. Foot landmarks used in the registration of the database and foot template topology
2.2. Full body shape space
To create the full body shape spaces, we used three databases containing more than 800 children,
1.800 male and 8.900 female scans gathered using Vitus XXL scanner from Human Solutions. These
scans were registered and harmonised using an adaptation of different template-based methods
[2,3,4,5]. In this case, we used a high resolution template body mesh of 150K vertices, 99K triangles, a
17-bone skeleton and a set of 35 landmarks as described by Ballester et al. [6], fig. 2. Database
parameterisation was also achieved by conducting Procrustes Alignment and Principal Component
Analysis (PCA) to the homologous data.
Fig. 2. Template, body
landmarks and skeleton
Fig. 3. Posture variability
reduction
Fig. 4. Homologous and harmonized sample of
females
Different shape spaces were created depending on two factors:
(i) Type of input to be used in the application (i.e. images or parameters). For the 3D recreation of
full bodies from measurements, the databases used were the posture-harmonised ones. In the
case of image-based recreation, posture deviations from the standard harmonised posture were
randomly included in the database for arms, elbows and legs (ranging from 0 to 30º) in order to
cope with posture deviations of the users being photographed.
(ii) Population segment addressed by the applications. This factor was relevant in order to reduce
the degrees of freedom of the shape space and improve the results. The population segments
and the number of components used in the body shape space are indicated in table 1.
6th International Conference on 3D Body Scanning Technologies, Lugano, Switzerland, 27-28 October 2015

Table 1. Shape space configuration for different applications.
Application Population segment No. of comps.
Children 3D body reconstruction from images 800 children aged 3-12 100
Female 3D body reconstruction from images 6000 females aged 18-65 100
Female 3D body reconstruction from measurements 6000 females aged 18-65 20
3. Data-driven reconstructions of 3D bodies or body parts
Data-driven 3D reconstruction proposed consists of two steps: user data input (i.e. either images or
measurements); and 3D reconstruction. The reconstruction approaches presented are based on
iteratively resolving a minimisation problem that finds the combination of shape parameters (i.e.
principal component scores) that best matches either the silhouettes extracted from the images or the
body measurements entered.
3.1. Image-based 3D reconstruction of Feet
We have developed two different user data input systems for the feet reconstruction: a mobile app [7]
and a booth specifically designed for this purpose. The booth is equipped with a Raspberry Pi, one
camera on top, a green/blue matte bottom surface, and two mirrors placed in such an angle that can
obtain three views of the feet with a single photograph.
In the case of the mobile app (fig. 5), the input data consists of three images of the foot on top of a
DINA4 sheet. Each photograph should be taken from a different point of view; namely zenithal, medial
and lateral views. The app sends the images to a remote server where all the processing takes place.
Fig. 5. Screenshots of the Instructions of the foot scanning app showing the three views and resulting 3D feet
In the case of the booth (fig. 6), the input data consists of one photograph containing the three views
thanks to the mirrors. The booth is managed through a local application running on a Windows 8 tablet
that is connected via Bluetooth to the booth. The tablet application also sends the images to a remote
server where all the processing takes place.
Fig. 6. Booth setting, one-shot scanning process, single image and resulting feet
Regarding 3D reconstruction, in both cases, the images are processed to extract the foot silhouettes.
In the case of the app, the A4 sheet corners [8], are used to calibrate each view [9]. In the case of the
booth, the booth space and the camera are already calibrated. The 3D reconstruction departs from the
average foot model created from the PCA (zero at all scores) and consists of minimising the distance
from the silhouettes of the PCA model projected into the image planes to the silhouettes extracted from
the foot images. Minimization is approached by iteratively modifying the PCA scores. At each iteration,
6th International Conference on 3D Body Scanning Technologies, Lugano, Switzerland, 27-28 October 2015

the vertices defining the projected silhouettes of the PCA model are computed, and then distances are
minimised using L-BFGS-B optimisation method based on explicit gradients [10 ]. After each
optimization, the new set of vertices defining the silhouettes is used in the next iteration. This process
is repeated until it converges, which usually takes less than ten iterations (fig.7).
Fig. 7. Iterative process for the fitting of the 3D foot into the silhouette
For validation purposes, four synthetic feet models representing the main foot shape variations (i.e.
high/low instep height and high/low foot width) were manufactured using a SLS printer (EOS Formiga
P100) and then scanned using a high precision 3D scanner (ATOS GOM, accuracy of 0.1mm). These
feet were reconstructed using our mobile app in a Samsung S3 and our special booth. For each of the
systems, each foot was scanned five times. A set of measurements was extracted from the feet
digitised with the high precision scanner, with our special booth, and with our mobile app. The
accuracy of the location of 12 anatomical landmarks was also estimated for our reconstruction
approaches. Moreover, the distance field error (average surface-to-surface distance per vertex) was
calculated between the digitised feet and each of our reconstructions. Prior to the calculation, feet were
aligned using Iterative Closest Point method (ICP).
The distance field error per vertex of the 3D reconstructions are presented respectively in table 2 (for
the app and the booth). Figure 8 shows the average distance field error per vertex for the booth. The
accuracy of the measurements extracted from the app and the booth reconstructions are presented in
table 3.
Fig. 7. Average distance field error per vertex of the Booth for the five
repetitions of the four synthetic feet (scale from 0 to 7mm)
Table 2. Overall distance field error
(average surface-to-surface distance
per vertex) for the four synthetic feet
in mm
App Booth
Mean 1.120 1.064
50
th
percentile 0.938 0.823
95
th
percentile 2.774 2.963
Table 3. Average error of the measurements extracted from image-based 3D reconstructions
App Booth
Measurement Mean
(mm)
SD
(mm)
Mean
(%)
SD
(%)
Mean
(mm)
SD
(mm)
Mean
(%)
SD
(%)
Foot Length 0.8 0.5 0.3 0.2 1.0 0.6 0.4 0.2
Toes Girth 2.1 2.2 1.0 1.1 2.2 1.3 1.0 0.6
Toes Width 1.2 0.9 1.3 1.0 0.4 0.3 0.4 0.3
Ball Girth 3.4 0.9 1.5 0.4 2.1 0.9 0.9 0.4
Ball Width 1.7 0.9 1.8 0.9 1.2 0.6 1.3 0.6
Instep Girth 3.1 1.3 1.3 0.5 2.7 1.8 1.2 0.8
Instep Height 2.7 0.9 4.3 1.5 2.3 1.4 3.8 2.3
3.2. Image-based 3D reconstruction of bodies
A mobile app was developed in order to gather user data input for the full body reconstruction (fig. 9).
In the full body case, the input consists of two images of the person wearing tight clothing or underwear
and three sociodemographic parameters: age, stature and body mass. The first image is a front
6th International Conference on 3D Body Scanning Technologies, Lugano, Switzerland, 27-28 October 2015

photograph of the person standing with the arms and legs slightly separated from the body and the
second one is a lateral photograph with the arms stuck to the body. In the case of the body
reconstruction, the calibration element selected was the user stature, and thus results are sensible to
this parameter. The app requires that a second person takes the pictures and includes a silhouette for
each of the photographs to help to position the user. The silhouettes are automatically adapted to
match the average person corresponding to the age, stature and body mass entered.
3D reconstruction is conducted following a similar process as the one of the foot with the exception of
the image calibration process which in this case uses stature of the photographed person and
parameters from the telephone such as the camera, gyroscope and accelerometer. However, the
segmentation of the image is more challenging than in feet due to the variability of the backgrounds,
clothing colours, and user posture, so the adaptive silhouette is also used to support the segmentation
algorithms (fig. 10).
Fig. 9. Data gathering using the mobile app
Fig. 10. Screenshots from the app showing the adaptive guiding
silhouette and the image segmentation
Two tests were conducted using the image-reconstruction system. The preliminary test was conducted
as a proof of concept of the full body reconstruction using a shape space of females aged 13 to 75.
Three adult female volunteers were photographed in a controlled environment using the app installed
in a Motorola Moto G and they were also scanned with the Vitus XXL scanner. Then, body
measurements were extracted using the same algorithms for both 3D representations [6]. Results of
this preliminary testing are included in table 4. The comparison of the pictures, reconstructed shapes
and actual scans are presented in figure 11.
Fig. 11. Qualitative comparison between picture, actual scan and 3D reconstruction using the app
A second test was conducted for validation purposes using refined algorithms for segmentation and
reconstruction and using a shape space of children aged 3 to 12. 10 children and their parents
participated in the study. The results of the second test for the comparison of the measurements
extracted from body scans and 3D reconstructions are also included in table 4
6th International Conference on 3D Body Scanning Technologies, Lugano, Switzerland, 27-28 October 2015

Table 4. Average absolute and relative error of 10 body measurements from image-based 3D reconstructions
A
dult Females (proof) Children (prototype)
Measurement
AAE (mm) ARE (%) AAE (mm) ARE (%)
Body Height 16 1.0 22 1.6
Bust/chest girth 6 0.7 18 2.8
Underbust girth 21 2.7 - -
Waist girth 33 4.1 28 4.5
Hip girth 17 1.7 11 1.5
Thigh girth 30 4.7 10 2.3
Upper arm girth 9 3.8 9 4.1
Wrist girth 5 3.2 5 3.8
Outside leg length 18 1.7 12 1.4
Arm length 16 2.8 13 2.7
3.3. Measurement-based 3D reconstruction of bodies
This 3D reconstruction algorithm was based on Ballester et al. methodology [6], which consists of
estimating a combination of principal components by a Partial Least Square Regression model from a
set of parameters (e.g. measurements, age, body mass, etc.), and then iteratively measure and modify
the 3D shape within the principal components shape space until it converges (i.e. set of
measurements). A shape space of 20 components from adult women was used for this study. The
initial solution obtained by the PLS regression model used 6 parameters, namely age, height, body
mass, bust girth, waist girth and hip girth. The iterative optimisation was conducted for the height, bust
girth, waist girth and hip girth.
For validation purposes, more than 100 adult women volunteers participated in a self-image
recognition experiment where they were asked to choose between two 3D representations of her
selves, one created from an adaptation of a commercial parametric avatar (i.e. Victoria Character from
DAZ Studio), and the other one created from the proposed measurement-based reconstruction.
In order to make the he generic 3D body shapes created more realistic, they were enriched with skin
and underwear texture, and with generic details like hair, hands, shoes etc. that scaled along with the
shape space coordinates. The same texture mapping was applied to the adaptation of the parametric
DAZ model. The reconstruction algorithm was also implemented into a web service and a website was
created to support the data input and the visualisation of results. For the creation of the reconstruction
the participants entered their self-reported age, height, weight, bust, waist and hips.
Results showed that 84% of participants chose the data-driven representation as more representative
of themselves than the DAZ-based avatar. An example with two volunteers is presented in Figure 12.
The experiment showed that data-driven representations became the more representative as the user
shape was more different to the base DAZ model.
Fig. 12. Qualitative comparison between picture, DAZ-based avatar and Data-driven avatar
4. Real world applications of data-driven low-cost 3D reconstruction
The different technologies and solutions presented are being integrated into real world applications for
the orthotics, footwear and clothing sectors at different levels of maturity. Five of these cases are
briefly presented in the following paragraphs:
6th International Conference on 3D Body Scanning Technologies, Lugano, Switzerland, 27-28 October 2015

• SunFeet: personalised 3D printed comfort insoles (http://www.sunfeet.es/en/). The first
application of the foot reconstruction app and the booth was being used as input for comfort
insole personalisation. This service enables the user to create custom-made insoles in terms
of arch shape, colour combinations and mechanical properties, depending on the use they will
have (i.e. casual, sports or formal/heeled). Insoles are made up of a hard shell individually
manufactured using 3D printing and cushioning materials. This service was validated with
nearly 300 users, and its commercialisation in Europe started in 2015.
• Feetz: personalised 3D printed shoes (http://www.feetz.com/). The second application of the
booth and app is to gather the user’ foot data input to create personalised shoes using
innovative 3D printing technology. The feetz shoes are expected to be launched in 2016.
• ShopInstantShoe: footwear reshaping at the point of sale. This is a service developed under a
research project [11] conducted in cooperation with Calzamedi S.L., Nimesis SAS and MDB
Texinov. The service consists of the customization (reshaping) of the footwear forefoot at the
point of sale based on the clients’ foot shape. the Booth is used in order to gather the foot
shape of the client. The shoe vamps are equipped with shape memory materials that enable,
not only the permanent reshaping but also the possibility to go back to the original shape if the
user does not want to buy the shoes after trying them on.
• KidSize: size allocation for childrenswear. The first application of the body reconstruction app
is a service that is being developed for the Spanish and French associations of manufacturers
and retailers of childrenswear (ASEPRI and NovaChild) and its European umbrella association
(Childrens Fashion Europe) under a research project [12]. KidSize uses the captured 3D body
shapes in order to make an accurate size allocation for childrenswear. The service consists of
a button next to the clothing size selection at the ecommerce and an iFrame showing the
recommended size and the predicted fitting by body areas. It is currently entering the
validation phase and it is expected to be included into Bóboli (Star Textil S.A.) and Sucre
d’Orge Kids (Groupe Salmon Arc-en-ciel) online shops in 2016.
• VisuaLook: virtual try-on of fashion. The first application of the measurement-based 3D
reconstruction is a service that is being developed in cooperation with Tecnologías DIM S.A.
under a research project [13]. VisuaLook service will provide a scalable solution for the
visualisation of virtual looks at online shops. It is expected that the service includes the
image-based mobile app 3D reconstruction along with the measurement-based one.
4. Conclusions and future work
Despite not being actual body scanners, the presented solutions are easy-to-use and can provide
enough accuracy (i.e. 1mm average distance field error) for certain applications such as virtual try-on,
made-to-measure or size allocation of footwear, clothing or orthotics, among other wearables [14].
Moreover, they can be distributed to the final consumer or to the points-of-sale at a really reduced cost
(or even for free), thus overcoming the main barriers to the massive spreading of its use in
e-commerce, new retail experiences, new production pipelines or new business models.
The key advantages of the image-based solutions compared to the traditional body measurement
methods are: firstly, the ability to gather individual shape features derived from posture/skeletal, such
as the dorsal and lumbar curves or shoulders shape, or slight dissymmetries, which might be relevant
for a successful made-to-measure or bespoke product design; secondly, an app or booth does not
require the same level of expertise and time to get quality measurements as the measuring tape needs,
so this makes it a more robust tool.
Nevertheless, additional R&D work is required for each of the proposed 3D body reconstruction
solutions. For instance, it is important that these solutions are validated with non-Caucasian population
targets. The results of this validation could eventually conclude that the shape spaces should be
enriched in order to include the body or feet shape variability associated to other population groups. In
the case of the image-based full body reconstruction, new developments are expected to be
completed for adult males and females in 2016, as well as analogue developments for body parts like
hands or legs.
Regarding the 3D reconstruction of full bodies, firstly, a distance field error (surface-to-surface distance
per vertex) metric should be established and calculated, using for instance a set of rigid full body
mannequins as in the validation of the feet reconstruction. Improvements in the segmentation of
6th International Conference on 3D Body Scanning Technologies, Lugano, Switzerland, 27-28 October 2015

images can also be achieved with the use of shape priors [15]. Moreover, in our present approach for
the whole body segmentation additional posture deviations have been included into the shape space
(to cope with the different postures of users in the photographs taken), an alternative approach could
be removing from the shape space all posture/skeletal information and incorporating it as a separate
parameter set within the optimisation process. This could be achieved, for instance, using adaptations
of alternative registration concepts such as the one introduced by Anguelov et at. [16].
Acknowledgements
The authors thank Feetz, Inc. and Tecnologías DIM, as well as all the European Commission and
Spanish Ministry of Science and all companies and organisations participating in KidSize [13],
Easy-IMP [17], DemoSIS, ShopInstantShoe [11] and TALLA-ME [18] projects, in particular, ASEPRI,
NovaChild, Star Textil S.A, Groupe Salmon Arc-en-ciel, ISRI, Ozongo and Calzamedi S.L. for their
contribution to the R&D work described in this paper.
References
[1] E. Parrilla et al. (2015). Low-cost 3D foot scanner using a mobile app. Footwear Science, 7(sup1),
S26-S28.
[2] B. Allen et al. (2003). The space of human body shapes: reconstruction and parameterization
from range scans. In ACM Transactions on Graphics (TOG) (Vol. 22, No. 3, pp. 587-594). ACM.
[3] B. Amberg et al (2007). “Optimal step nonrigid icp algorithms for surface registration”. In
Computer Vision and Pattern Recognition, 2007. CVPR'07. IEEE Conference.
[4] Sumner, R. and Popovic, J., “Deformation Transfer for Triangle Meshes.” SIGGRAPH 2004.
[5] R. Trieb et al. (2013) EUROFIT: Integration, Homogenisation and Extension of the Scope of Large
3D Anthropometric Data Pools for Product Development. In 4th Int Conf on 3D Body Scanning
Technologies, Long Beach, CA, USA.
[6] A. Ballester et al. (2014). 3D-based resources fostering the analysis, use, and exploitation of
available body anthropometric data. In 5th Int Conf on 3D Body Scanning Technologies,
Hometrica Consulting, Lugano, Switzerland.
[7] SunFeet App: Android (GooglePlay) https://goo.gl/Q96Y5l and iOS (iTunes) https://goo.gl/x3ntOX
[8] C. Rother et al (2004). Grabcut: Interactive foreground extraction using iterated graph cuts. ACM
Transactions on Graphics (TOG), 23(3), 309-314.
[9] M. Bujnak et al. (2008, June). A general solution to the P4P problem for camera with unknown
focal length. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE (pp. 1-8).
[10] C. Zhu et al. (1997). Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale
bound-constrained optimization. ACM Trans on Mathematical Software (TOMS), 23(4), 550-560.
[11] FP7-SME-2008-1-232109. “Development of a cost-effective footwear based on shape memory
materials to provide an instant fitting personalization service at the retail shop for enhancing user's
comfort” (ShopInstantShoe), FP7, European Commission.
[12] H2020-SMEINST-1-2014 -662847. “Novel clothing e-commerce application for reliable size
assignment and realistic fitting visualization” (VisuaLook), H2020, European Comission.
[13] FP7-SME-2013-606091. "Development of a new extended product-service to overcome size
assignment and fitting barriers for children fashion on-line market addressing customer needs"
(KidSize), FP7, European Commission
[14] H. Lee et al. (2005). Development of a low cost foot-scanner for a custom shoe tailoring system.
In 7th Symposium on Footwear Biomechanics.
[15] D. Cremers et al. (2001). Diffusion-snakes: combining statistical shape knowledge and image
information in a variational framework. En IEEE Workshop on Variational and Level Set Methods
in Computer Vision, 2001. Proceedings (pp. 137-144). http://doi.org/10.1109/VLSM.2001.938892
[16] Anguelov et al. (2005). SCAPE: shape completion and animation of people. In ACM Transactions
on Graphics (TOG) (Vol. 24, No. 3, pp. 408-416). ACM.
[17] FP7-FoF-NMP-2013-5-609078. “Collaborative development of intelligent wearable meta-products
in the cloud” (EASY-IMP), FP7, European Commission.
[18] DPI2013-47279-C2-2-R. “Herramientas para la predicción de la talla y el ajuste de ropa infantil a
partir de la reconstrucción 3D del cuerpo y de técnicas 'big data'” (TALLA-ME) Programa Estatal
de Investigación, Desarrollo e Innovación Orientada a los Retos de la Sociedad.
6th International Conference on 3D Body Scanning Technologies, Lugano, Switzerland, 27-28 October 2015