A practical framework for virtual viewing and
Qi Duan, Jianjun Yu, Xubo Yang, Shuangjiu Xiao
Digital Art Laboratory,
School of Software
Shanghai Jiao Tong University
Abstract. Recently many practical applications have concerned with
observing objects that often have specula reflection properties. They in-
tend to know how the specula reflections and details on the surface of
objects vary according to different light conditions and view positions
without calculating any 3D information of the objects. In this paper,
we propose an innovative framework combining the novel view synthesis
algorithm and the relighting algorithm to fulfill these practical require-
ments. This framework can observe objects under different light condi-
tions and in arbitrary view positions, with specula reflection properties.
Another important feature of this framework is that all the algorithms
are based on image without acquiring any 3D model since any kind of
model may contain some high confidential secrets. Meanwhile an image
measurement criterion is proposed to verify the feasibility and effective-
ness of the framework.
Key words: Novel View Synthesis, Image-based Relighting, Image Mea-
Recently, the technique that can render images of product under arbitrary illu-
minations or view points became increasingly important in Factory Automation
application. It is also of great value in movie, museums, and electronic commerce
applications. For example, in machine vision applications, rendering method is
used to simulate and decide the optimal lightning or viewpoint of the camera. In
order to enhance the exhibition effect of object in some special circumstances,
various special lights will be placed in certain positions to augment the effects
of details on product surface; meanwhile the observation position and angle also
influence the display result. The position of viewpoints and the condition of lu-
minance are two key factors that influence the effects. To determine the best
luminance condition and view point position, it is very helpful to design a tool
that can precisely observe the surface reflection effect of target under various
light conditions, such as different colors and types of LED light sources, and
2 A practical framework for virtual viewing and relighting
also can observe objects in some arbitrary view positions. Considering that the
product may contain some high confidential secrets, those requirements can be
summarized as follows:
– A method to simulate the result that the object is placed under various light
conditions, for example, different numbers, sizes, types, colors of light re-
sources and different distances between objects and light. Considering practi-
cal application requirements, we need to get the best result with the minimal
– A technique to synthesize novel images of the objects. Considering the ob-
jects may be some industrial products relating to commercial commercial
confidential secrets, the less 3D information of the target we calculate, the
However, currently all the related work solves only one of the two requirements.
Thus, we design this new practical framework to handle this problem.
1.1 Related work
Two main algorithms employed in this framework are the image-based relighting
and the novel view synthesis.
Image-based Relighting(IBL) attracts a lot of attention recently, because
it does not require precise physical models. Image-based relighting methods
can be classified into three categories, named reflectance-based relighting, ba-
sis function-based relighting and plenoptic function-based relighting. Reflectance
Function-based Relighting techniques explicitly estimate the reflectance function
at each visible point of the object or scene, which is known as the Anisotropic
Reflection Model or the Bidirectional Surface Scattering Reflectance Distribu-
tion Function (BSSRDF). Lin et al. researched the dual of the Lumigraph,
in which the camera is fixed and a point light source is moved. Basis Function-
based Relighting techniques take advantage of the linearity of the rendering op-
erator with respect to illumination. Relighting process is accomplished via linear
combination of a set of pre-rendered ”basis” images. Nimeroff et al.  used a
technique of combining images to relight a scene. Debevec et al. describes a
Light Stage in which an object can be placed. Plenoptic Function-based Relight-
ing techniques are based on the computational model, the Plenoptic Function.
So the plenoptic function-based relighting techniques extract out the illumina-
tion component from the aggregate time parameter, and facilitate relighting of
Novel image synthesis has been studied in the last decade. The work in this
area can be divided into two main classes roughly: image-based modeling and
rendering(IBMR) and image-based morphing and rendering. The first class is
designed to reconstruct 3D models from photographs. Debevec et al. have
A practical framework for virtual viewing and relighting3
shown a method of modeling and rendering architecture from a small number
of photos. Criminisi et al. create 3D texture-mapped models relying solely
on one single image for its input. Byong Mok Oh et al. take a single photo
as input and model the scene. After modeling the scene, we can change from
different viewpoints; modify the shape, color and illumination of the scene. The
second class does not calculate any 3D information of the scene. It use various
methods to synthesis the novel view image such as image warping, image in-
terpolation, image extrapolation and so on. For example, Seitz and Dyer
interpolate along the base line of image pairs to obtain correct images. Levoy and
Hanrahan and Gortler et al. interpolate between a dense set of several
thousand-example images to obtain novel view images. Peleg and Herman
relax the fixed camera constraint to synthesis results. Avidan and Shashua
introduce the concept of trilinear tensor space and use trilinear tensor to syn-
thesize the novel view images.
However, the image-base relighting approach can only help us to view the tar-
get under various light conditions and the novel view synthesis method can only
generate the novel image under arbitrary viewpoints. Our framework integrates
the image based relighting approach and the novel view synthesis method, which
can render images of product under arbitrary illuminations and view points.
The following sections describe the framework, the experiment result and the
conclusion. In section 2 we introduce the method to simulate the object under
various light conditions and the process to synthesize a novel view image of object
in our framework. Image measurement criterion and framework performance are
described in section 3. Finally, we conclude and present the idea for future work
in section 4.
2.1 Framework architecture
The most significant feature of the framework is the ability to observe the tar-
get under arbitrary illumination and novel view positions, and all this results
are totally based on images without acquiring any 3D information. Therefore,
the framework integrates two important algorithms above as two modules. The
structure is shown in Figure 1. Firstly, in image-based relighting module, the
camera position is fixed and the relighting process is performed, then the result
image is obtained under arbitrary luminance situation. This image is named
the source image. Secondly, the camera position and orientation is moved in a
little range and a picture of the target object is captured as reference image.
Using source image and reference image as the input of novel view synthesis
module, the final novel image can be synthesized. The relationship between the
source image and the reference image is simply related to the two different cam-
era positions and calculated only once in the process procedure. Once the light
environment changes, the relighting result is recalculated and the source image
is updated, leaving reference image unmodified. Then the novel image is syn-
4 A practical framework for virtual viewing and relighting
thesized again. Employing this framework, we can observe the target of various
luminance situations under arbitrary viewpoints.
The basis function-based relighting and the trilinear tensor novel view synthe-
sis are selected in this framework instead of reflectance-based relighting, plenop-
tic function-based relighting and other novel view algorithms since these two
image-based method are superior in keeping confidential secret. The two impor-
tant methods will be introduced in the following sections of this chapter.
Fig.1. Framework Architecture.The architecture contains two main algorithm
modules. The source image is the output of relighting module, and the input of the
second module, as well as the reference image. The novel image is the final result as
the output of the novel module.
2.2 Image-Based Relighting
Image-based Relighting (IBL) can synthesize realistic images without a complex
and long rendering process as in traditional geometry-based computer graphics.
The algorithm detail is divided into four parts and will be introduced in the
Basis image acquisition. A light stage is designed to capture the basis images,
which is shown in Figure 2. This stage is a spherical structure attached with
prepared light sources. A point light is used as the ”standard” light source and
the target is placed at the center of the stage. The system captures images
of the target while only illuminated by the ”standard” light. The light moves
around the surface of sphere and for each light position and a corresponding basis
image is captured. This process is repeated until the entire sphere of incident
illumination is sampled at a pre-defined resolution. Since the distance between
the light and the target is much bigger than the target’s size, the point light
source can be considered as ”directional light source”.
A practical framework for virtual viewing and relighting5
Fig.2. Light Stage and Result Images. The target on the platform is illuminated
by ”standard” light. For each light direction, a corresponding image is captured. The
right image shows the capture results under our spherical sample methods.
Luminance mapping. When doing the renderings, it will be convenient to
have a table that maps from the image number to a direction on the sphere.
Thus, the sphere map is extended to latitude-longitude map, which is shown in
Fig.3. Sphere map to Latitude-Longitude map. According to the Light Stage,
we split the sphere surface that light source covered in different regions. Then these
regions are projected into a latitude-longitude map, each grid corresponding to a region
in sphere surface.
From the Figure 3 we notice that different section on the sphere has different
area. To obtain an accurate result, the impact for each region of samples is
scaled according to the area of the section on the sphere. The scaling factors in
latitude-longitude map can be determined by the following equation 1:
6 A practical framework for virtual viewing and relighting
Relighting process. To render the target under arbitrary illumination, the
virtual light illumination is mapped onto the sphere map firstly. Then the sphere
map is converted to the Longitude-Latitude map. So the relighting problem can
be solved by linear combination of the basis images. Relighting of the sequence
of images Iican be described by the following equation 2:
Here the equation 2 is approximated by a linear combination equation 3 of
the images Iiand the mean of the lighting environment Eiover the solid angle
scaled by the relative area of the solid angle projected onto the unit sphere.
Where Ciis the difference between arbitrary light intensity and standard light
intensity.ID and IR are the intensity of arbitrary light and standard light re-
With the images set, latitude-longitude maps and the weights, the image
under novel illuminations can be generated.
Fig.4. Virtual Ring Light Relighting. A virtual ring light is simulated using the
combination method of spatial arrays of point light sources and all the point light
sources are mapped to the latitude-longitude map. The relighting image under virtual
ring light source is shown in the left.
2.3 Novel View Synthesis
The novel view synthesis algorithm is employed to synthesize the image under
arbitrary view position. This technique is based on two or three sample images
A practical framework for virtual viewing and relighting7
and the primary process is the tirlinear tensor operation, which is quite fit to
Feature point detection. During the novel process, the relationship between
the sample images is crucial to this approach. To get a more accurate result, the
matching feature points are automatically detected in the sample images. The
Harris corner points are firstly detected and then the putative matches of
the feature points in images are calculated. In the end, the RANSAC iteration
algorithm is employed to detect the feature points accurately and robustly.
Trilinear tensor operation. At the beginning of the process, a tensor is calcu-
lated to be the basis tensor. In the projection theory, given three sample images,
there are three different projection matrixes [I,0],[A,V?],[B,V??]. The trilinear
tensor is a 3×3×3 matrix described by a bilinear function of the camera matrices
vector v?,v” respectively.
In the implement, to accelerate and simplify our process, we use only two
sample images, which means aj
ages is calculated and the elements of the fundamental matrix are rearranged
into the tensor; the basis tensor can also be obtained in this way.
To acquire novel image of the object, a 3 × 3 homography R and a 3 × 1
vector t should be specified, which represent the rotation component and trans-
lation component from sample image to novel image. The new tensor Gjk
be calculated in equation 5:
i,v?j,v??krepresent the elements of the homographies A,B and
i. The fundamental matrix of these two im-
images onto the desired novel view image.
iis just the new trilinear tensor used to re-project the first two sample
Re-projection process. After obtaining the new trilinear tensor and two sam-
ple image, the final procedure is to re-project the two sample image onto the
desired new novel image. The final novel image can be generated from the source
image using the re-projection equation 6.
Where p is the matching point in the source image, su
line intersecting at the corresponding point of p in reference image, p??is the point
in the novel image. Using the new tensor, two sample images and re-projection
equation, the new novel image can be re-projected directly.
jrepresents the eipolar
8 A practical framework for virtual viewing and relighting
In this section, some experiments are designed to verify the accuracy and ef-
ficiency of this framework. Regarding to the two different algorithms, diverse
verification experiments are carried out separately according to different mod-
Fig.5. Relighting Comparison. The left image is the generated relighting image
and the right image is the captured real image. The region in red square is the specula
reflection area on the surface.
As we see from Figure 5, it is very difficult to distinguish the difference be-
tween the synthesized relighting image and the real image. To verify the accuracy
of this relighting algorithm, the comparison between the synthesized image and
the real photo is essential. Generally speaking, taking difference pixel by pixel
is the most accurate way to measure the difference of two images. Most image
measurement methods before concentrate on the effect of contrast and sharpness
of an image on human perception. They are all used to compute the perceivable
difference of one image pair, such as the CIELAB color model and CIE2000
color difference formula. Those methods are not suitable for our verification
because they focus on human perception while our goal is to measure the quality
and effectiveness of the relighting algorithm. Under such circumstances, a new
appropriate approach is designed to evaluate the result of this framework.
Taking the aim of this module into account, the luminance property of the
image is quite important. So the histograms of the R,G,B channel as well as
the luminance of the two images are compared in this approach. In addition to
the histogram, the mean value and variance of the two images are calculated as
well, which are two widely used statistical parameters. Here we show the analysis
results of the relighting result and the real photo in Table 1 and Figure 6.
As we can see from the analysis results and Figure 5, the position and in-
tensity of specula reflection and the self-occlusion shadow on the target surface
under arbitrary luminance conditions are simulated precisely, and the range of
error between image pair is limited to a very small fraction. This experiment
proves the method used for image-based relighting is extremely accurate and
A practical framework for virtual viewing and relighting9
Fig.6. Histogram Analysis. Histogram comparison of the image pair in red channel
(left above), green channel (right above), blue channel (left below) and luminance (right
Considering the interpolation and extrapolation characteristic of novel view
algorithm, it is not suitable to measure the efficiency of our framework in quan-
titative methods. Therefore, the criterion of measurement is that the result sat-
isfies people’s vision effect and the physical distortions of the target object occur
in the acceptable range. Meanwhile the intensity and specula reflection property
as well as shadow details can be preserved in the novel image. Figure 7 shows
the experiment result of this module, which is also the final novel result of this
The efficiency of this framework is also what we concern about. The spec-
ification of the computer is Dell Dimension(TM) E520,Intel(R) Pentium(R) D
915 with Dual Core Processing,1GB Dual Channel DDR2 Memory and 160 GB
SATA HDD. This process time of this framework lies on the resolution of basis
images. In this experiment, the resolution of image is fixed at 512×342. During
the relighting process, the process time also depend on the condition of arbitrary
light situation such as type, color, shape, distance of light source. In the worst
situation, all the 36×17 basis images should be handled, at this time the process
10A practical framework for virtual viewing and relighting
Channel Mean Value VarianceError
Our Method Real Photo Our Method Real Photo
Table 1. Statistical result of the image pair.
Fig.7. Novel View Results. The left above image is the reference image and the
left below image is the generated source image. The other images are the synthesized
novel view images that we specify the camera move around the Z-axis of the target
from left to right. In the right blow image, which means the camera position changes
in a large range, the distortion of the target is obvious and unacceptable.
time is more than 70 seconds. But in the practical application, only parts of the
basis images are required to simulate under the arbitrary light situation, take
the virtual ring light source shown in Figure 4 for example, the process time
is nearly 12 seconds. During the novel view process, we only need to re-project
and calculate the result image per pixel once. So the process time is less than 1
second. We can nearly synthesize the real-time novel result.
4 Conclusion and Future Work
In this paper we present a framework for observing the target to satisfy some
practical requirements. We choose the image-based methods to avoid calculating
3D information or model of objects and still can synthesize the result under ar-
bitrary luminance conditions and view positions. Under various light condition
A practical framework for virtual viewing and relighting11
and viewpoint, we can observe different details on the object’s surface clearly
especially the specula and shadow details of the target. The accuracy and effi-
ciency of this framework is acceptable. In future, we will still do some researches
in these aspects: accelerating the process of relighting, improving the accuracy
of novel view methods, extending the novel view algorithm that can observe the
Acknowledgments. This research is sponsored by 863 National High Technol-
ogy R&D Program of China (No. 2006AA01Z307) and National Natural Science
Foundation of China (No. 60403044)
1. James T. Kajiya: Anisotropic reflection models. SIGGRAPH ’85: Proceedings of
the 12th annual conference on Computer graphics and interactive techniques. ACM
Press, New York, NY, USA (1985) 15–21
2. Henrik Wann Jensen and Stephen R. Marschner and Marc Levoy and Pat Hanrahan:
A practical model for subsurface light transport. SIGGRAPH ’01: Proceedings of
the 28th annual conference on Computer graphics and interactive techniques. ACM
Press, New York, NY, USA (2001) 511–518
3. Zhoulin Lin, Tien-Tsin Wong, and Heung-Yeung Shum: Relighting with the Re-
flected Irradiance Field: Representation, Sampling and Reconstruction. IEEE Com-
puter Vision and Pattern Recognition, Vol. 49. Springer Netherlands (2002) 229–246
4. J. Nimeroff, E. Simoncelli, and J. Dorsey: Efficient rerendering of naturally illumi-
nated environments. In Eurographics Rendering Workshop 1994, Darmstadt, Ger-
many, June 1994. Springer-Verlag (1994)
5. Paul Debevec and Tim Hawkins and Chris Tchou and Haarm-Pieter Duiker and
Westley Sarokin and Mark Sagar: Acquiring the reflectance field of a human face.
SIGGRAPH ’00: Proceedings of the 27th annual conference on Computer graphics
and interactive techniques. ACM Press/Addison-Wesley Publishing Co., New York,
NY, USA (2000) 145–156
6. Adelson, E. H. and Bergen, J. R: The plenoptic function and the elements of early
vision. Computational Models of Visual Processing (2001) 3.20
7. Debevec P. E., Taylor C. J., Malik J.: Modeling and rendering architecture from pho-
tographs: A hybrid geometry and image-based approach. In Proc. SIGGRAPH’96.
ACM Press,New Orleans, LS, USA(1996) 11–20.
8. A. Criminisi, I. Reid and A. Zisserman: Single View Metrology. International Journal
of Computer Vision, Vol. 40. Springer Netherlands (2000) 123–148
9. Byong Mok Oh and Max Chen and Julie Dorsey and F. Durand: Image-based mod-
eling and photo editing. SIGGRAPH ’01: Proceedings of the 28th annual conference
on Computer graphics and interactive techniques. ACM Press, New York, NY, USA
10. Thaddeus Beier, Shawn Neely: Feature-Based Image Metamorphosis. ACM SIG-
GRAPH Computer Graphics, Vol. 26. ACM Press, New York, NY, USA (1992)
11. Steven M. Seitz, Charles R. Dyer: Physically-Valid View Synthesis By Image In-
terpolation. IEEE Workshop on Representation of Visual Scenes. IEEE Computer
Society, Los Alamitos, CA, USA (1995) 18.
12 A practical framework for virtual viewing and relighting
12. S.M. Seitz and C.R. Dyer: View Morphing. SIGGRAPH ’96: Proceedings of the
23rd annual conference on Computer graphics and interactive techniques. ACM
Press, New York, NY, USA (1996) 21–30
13. Marc Levoy and Pat Hanrahan: Light field rendering. SIGGRAPH ’96: Proceedings
of the 23rd annual conference on Computer graphics and interactive techniques.
ACM Press, New York, NY, USA (1996) 31–42
14. Steven J. Gortler and Radek Grzeszczuk and Richard Szeliski and Michael F.
Cohen: The lumigraph. SIGGRAPH ’96: Proceedings of the 23rd annual conference
on Computer graphics and interactive techniques. ACM Press, New York, NY, USA
15. S. Peleg and J. Herman: Panoramic Mosaic by Manifold Projection. IEEE Com-
puter Society Conference on Computer Vision and Pattern Recognition. IEEE Com-
puter Society, Los Alamitos, CA, USA (1997) 338
17. S. Avidan and A. Shashua: Unifying two-view and three-view geometry. In ARPA,
Image Understanding Workshop (1997).
18. S. Avidan and A. Shashua: Novel View Synthesis by Cascading Trilinear Tensors.
IEEE Transactions on Visualization and Computer Graphics (TVCG),Vol. 04. IEEE
Computer Society, Los Alamitos, CA, USA (1998) 293–306
19. B. Hill and Th. Roger and F. W. Vorhagen: Comparative analysis of the quanti-
zation of color spaces on the basis of the CIELAB color-difference formula. ACM
Trans. Graph. . ACM Press, New York, NY, USA (1997) 109–154
20. Luo MR, Cui G, Rigg, B: The development of the CIE2000 colour-difference for-
mula: CIEDE2000. Color Research and Application,(2001) 26:340–350