Conference PaperPDF Available

HDR multiview image sequence generation: toward 3D HDR video

Authors:
  • CYENS Research Center on Interactive Media Smart Systems and Emerging Technologies

Abstract and Figures

Creating High Dynamic Range (HDR) images of static scenes combining several Low Dynamic Range (LDR) images is a common procedure nowadays. However, 3D HDR video content creation and management is an active, unsolved research topic. This work analyze the latest advances in 3D HDR imaging and proposes a method to build Stereo HDR images from LDR input image sequences acquired with a multi-view camera. Our method is based on the Patch Match algorithm which has been adapted to take advantage of epipolar geometry constraints of multiview cameras. Our method does not require the traditional stereo matching for disparity map calculation to obtain accurate matching between the stereo images. Geometric calibration is not required either. We use an 8-view LDR camera from which we generate an 8-view HDR output. The eight tonnemapped HDR images are used to feed an auto-stereoscopic display. Experimental results show accurate registration and HDR reconstruction for each LDR view.
Content may be subject to copyright.
Journées de l’Association Française d’Informatique Graphique, Reims Image 2014
Génération de séquences d’images multivues HDR:
vers la vidéo HDR
HDR multiview image sequence generation:
toward 3D HDR video
R. Ramirez Orozco 1,2, C. Loscos 2, I. Martin 1and A. Artusi 3
1Group of Geometry and Graphics(ViRVIG). Universitat de Girona (UdG), Spain
2CReSTIC SIC, Universite du Reims Champagne-Ardenes (URCA), France
3Graphics & Imaging Laboratory (GILab). Universitat de Girona (UdG), Spain
Résumé
La génération d’images " high-dynamic range " (HDR) des scènes statiques en combinant plusieurs images " low
dynamic range " (LDR) est une procédure maintenant acceptée. Pourtant, la recherche sur la création et la mani-
pulation de contenu vidéo HDR 3D est très active car c’est encore un problème non résolu. Ce travail analyse les
avancées récentes en imagerie 3D-HDR et propose une méthode pour construire des séquences 3D HDR à partir
de données LDR acquises par une caméra multivue. Notre méthode est basée sur un algorithme basé "patch"
(zones) adapté pour s’appuyer sur les contraintes épipolaires des caméras multivues pour obtenir l’alignement
des pixels nécessaire à la génération d’images HDR. Elle ne s’appuie pas sur l’alignement stéréo traditionnel
basé disparité et n’a pas non plus besoin d’une calibration géométrique exacte des objectifs des caméras. Pour
des données d’entrée de 8 images LDR acquises par un équipement dédié, nos résultats expérimentaux produisent
un alignement suffisant pour la génération de 8 images HDR, sur lesquels un algorithme de tone mapping peut
être appliqué pour une sortie sur un écran autostéréoscopique.
Creating High Dynamic Range (HDR) images of static scenes combining several Low Dynamic Range (LDR)
images is a common procedure nowadays. However, 3D HDR video content creation and management is an active,
unsolved research topic. This work analyze the latest advances in 3D HDR imaging and proposes a method to build
Stereo HDR images from LDR input image sequences acquired with a multi-view camera. Our method is based on
the Patch Match algorithm which has been adapted to take advantage of epipolar geometry constraints of multi-
view cameras. Our method does not require the traditional stereo matching for disparity map calculation to obtain
accurate matching between the stereo images. Geometric calibration is not required either. We use an 8-view LDR
camera from which we generate an 8-view HDR output. The eight tonnemapped HDR images are used to feed an
auto-stereoscopic display. Experimental results show accurate registration and HDR reconstruction for each LDR
view.
Mots clé : Computational Photography, 3D High Dyna-
mic Range, Multi-Exposure Stereo Matching
1. Introduction
Most of modern digital cameras provide some kind of auto
exposure control to determine the correct parameters of aper-
ture, ISO value and shutter speed to cover the widest light
intensity range possible for a given scene. But still the ma-
jority of cameras available at consumer level have limited
sensors, capable of recording only a very small part of light
intensities. Camera sensors produce saturated pixels where
the amount of light exceeds the range of values allowed and
under-exposed pixels when not enough light reach the ca-
mera sensor. One of the goals in HDR imaging is to enlarge
the range of light recorded of a scene.
The most extended way to acquire HDR images is com-
bining images with different exposures of the same scene.
There are specific HDR sensors already in the market but
too expensive for their expansion at user level. There are dif-
ferent approaches to combine information from a sequence
of Low Dynamic Range (LDR) exposures [MP95, DM97,
c
AFIG 2014, Association Française d’Informatique Graphique (http://www.asso-afig.fr)
R. Ramirez Orozco & C. Loscos & I. Martin & A. Artusi / Génération de séquences d’images multivues HDR:vers la vidéo HDR
IEE99, MN99, RBS03]. Ghosting effects may appear in the
output HDR image when the pixels of source images are not
perfectly aligned. There are two main causes for ghosting :
camera movement during acquisition and the objects moving
in the scene. There are several solutions for image alignment,
many of them explained in a detailed survey Zitová [ZF03].
However, these methods were created for images captured
under the same lighting conditions and the transition to dif-
ferently exposed images is not straightforward.
Some methods to deal with dynamic scenes have been
presented but there is not a standard solution for this, spe-
cially when both camera and objects in the scene move. One
of the main drawbacks for HDR video acquisition is the
lack of robust methods for deghosting HDR images resul-
ting from multi-exposed sequences of dynamic scenes. Both
Hadziabdic et al. [HTM13] and Srikantha et al. [SS12] pro-
vide good reviews and comparisons between up to date me-
thods for HDR deghosting.
The creation of high dynamic range (HDR) content has
been an intense field of research lately, not only for digital
photography but for a wide range of imaging applications.
Stereoscopic 3D images and video are also important goals
for the HDR research community. Most work on 3D HDR
are based on combining two stereo exposures.
Bonnard et al. [BLV12] propose a methodology to create
content that combines depth (3D) and high dynamic range
video for auto-stereoscopic displays. They use an Octo-
cam [PcPD10] which is a camera prototype with eight ob-
jectives synchronized in time that produces pairs of four dif-
ferent exposures of the same scene (Figure 1). Bonnard et al.
used reconstructed depth information from epipolar geome-
try to lead the pixel match procedure. This method lack of
robustness especially on under and over exposed areas.
We propose a solution to combine stereoscopic LDR
images into HDR using image correspondences based on the
Patch Match algorithm [BSFG09]. This algorithm has been
used recently by Sen et al. [SKY12] to build HDR images.
The results were promising for multi-exposure sequences
where the reference image is moderately under or over ex-
posed, but it fails otherwise. We propose to take advantage
of geometric constraints in the set of images to help the
matching process. We iterate over the set of multi-exposed
images from the Octo-cam selecting a reference image each
time, then all the remaining images are registered using the
modified patch match and finally they are merged into one
HDR per view. No geometric calibration or disparity map
calculation is needed while using Patch Match.
The rest of the paper is organized as follow. Section 2 fo-
cuses on giving a background about the state of the art on 3D
HDR video, section 3 describes the solution we are develo-
ping and finally section 4 shows some experimental results
and comparisons.
2. Background
2.1. Stereo HDR Acquisition
Several prototypes have been proposed to acquire HDR
videos from multi-exposure sequence of images. Most of ap-
proaches [TKS06,LC09,SMW10,Ruf11,THM13, BRG14,
TCJHM14] are based on a rig of two cameras placed like a
conventional stereo configuration. They are focused on fin-
ding accurate correspondences to generate HDR images per
each view. Troccoli et al. [TKS06] proposed to use cross cor-
relation stereo matching to get a primary disparity match.
The correspondences are used to calculate the camera res-
ponse function (CRF) to convert pixels values to radiance
space. Stereo matching is executed again, this time in ra-
diance space to extract one depth maps per view.
Lin and Chang [LC09] use SIFT descriptors to find
correspondences. Afterward they select the best correspon-
dences using epipolar constrains and use them calculate the
CRF. On radiance space they perform a stereo matching al-
gorithm based on belief propagation to derive the disparity
map. A ghost removal technique is used to avoid artifacts
due to noise or stereo mismatches. Rüfenacht [Ruf11] pre-
sents two different approaches to obtain stereoscopic HDR
video content. The first is the temporal approach, where dif-
ferent exposures are captured by temporally changing the ex-
posure times of both cameras at a time, recording two frames
of the same exposure in each shot alternating the exposure
time between consecutive pairs of frames. The second is cal-
led ’spatial approach’, here each camera have a different ex-
posure time for all the shots, in this case each frame in the
same shot have different exposures.
Akhavan et al. [THM13,TCJHM14] study different ways
to obtain disparity maps from HDR, LDR and tonemapped
images and compare them. Selmanovic et al. [SDBRC14]
propose to generate Stereo HDR video from a pair HDR-
LDR, using an HDR camera and a traditional digital camera.
In this case, an HDR view needs to be reconstructed. Three
methods were proposed : (1) generate the HDR frame by
warping the existing one using a disparity map,(2) increase
the range of the LDR view using an expansion operator and
(3) an hybrid of the two which, according to their compari-
son, provides the best results. Bätz et al. [BRG14] present
a framework with two LDR cameras. The CRF calculation is
performed offline. The input images are rectified before the
disparity estimation. The stereo matching exposure invariant
used Zero-Mean Normalized Cross Correlation (ZNCC) like
matching cost. The matching is performed on the gray-scale
radiance space image followed by a local optimization and
disparities refinement.
Stereo HDR acquisition has been addressed with many
different approaches in the last decade. However, multi-view
HDR video (more than two views) has not yet been addres-
sed except by Bonnard et al. [BLV12]. All existing me-
thods rely on accurate stereo matching to obtain the disparity
maps.They require accurate geometric calibration to ensure
that the epipolar geometry conditions are satisfied.
2.2. Image Alignment
In the HDR context most of methods for image alignment
focus on movement between images caused by hand-held
capture, small movement of tripods or moving pixels from
dynamic objects in the scene. Most of them assume that all
images are taken from the same viewpoint so they are not
suitable for the multi-view images since the kind of transfor-
c
AFIG 2014.
R. Ramirez Orozco & C. Loscos & I. Martin & A. Artusi / Génération de séquences d’images multivues HDR:vers la vidéo HDR
Figure 1: Set of LDR multi-view images taken with an eight view camera using natural color filters to control exposures.
mations that takes place are slightly different. Sand and Tel-
ler [ST04] proposed a combination of feature matching and
optical for spatiotemporal alignment of different exposed vi-
deos. They align a pair of videos by searching for frames that
best match according to an image registration process. This
process uses locally weighted regression to interpolate and
extrapolate image correspondences. This method is robust
to changes in exposure and lighting, but if there are objects
moving at high speed, artifacts may still appear.
Mangiat adn Gibson [MG10] propose to use a method of
block-based motion estimation and refine the motion vec-
tors in saturated regions using color similarity in the adja-
cent frames of an alternating multiexposed sequence. Ni-
quin et al. [NPR10] use the octocam to reconstruct a 3D
scene using a pixel matching method based on graph cuts.
This method is suitable for images with the same exposure
but the precision of the matching is not good for multiex-
posed images. Sun et al. [SMW10] proposed an algorithm
based on the assumption that the disparity map between two
rectified images can be modeled as a Markov random field.
The matching problem is then posed as a Bayesian labeling
problem in which the optimal values are obtained minimi-
zing an energy function. The energy function is composed
of a pixel dissimilarity term (using NCC as similarity mea-
sure) and a smoothness term which correspond to the MRF
likelihood and the MRF prior, respectively. Most of stereo
matching algorithms perform energy minimization schemes,
which imply high computational cost.
Sen et al. [SKY12] recently presented a method based
on a patch-based energy-minimization formulation that inte-
grates alignment and reconstruction in a joint optimization.
This allows to produce an HDR result that is aligned to one
of the exposures and contains information from all of them,
but artifacts may appears when there are large under or over
exposed areas in the reference image.
3. Patch-based Stereo HDR Generation
3.1. The Patch Match algorithm
Stereo matching is a mature research field ; very accurate
algorithms are available for images taken under the same
lighting conditions and same exposure. However, most of
algorithms are not fully accurate for images with important
radiometric variations. We propose a framework based on a
variation of the work from Barneset al. [BSFG09, BSGF10]
and Sen’s work [SKY12]. We propose to adapt the mat-
ching process to the multi-view context resulting in a more
robust and faster solution. Our method does not requires ste-
reo matching for disparity maps calculation. Calibration can
be also avoided since the matching process does not requires
a perfect epipolar geometry. To understand the basis of our
approach we need first to introduce briefly the original patch-
based algorithm [BSFG09,BSGF10].
The Patch Match solves the matching problem between
two images A and B at patch level using Nearest Neighbor
Fields (NNF) minimization. NNF is defined over all possible
patch coordinates (locations of patch centers) in image A ;
for some distance function D between two patches of images
A and B. Given a patch coordinate ain image A and its cor-
responding nearest neighbor bin image B, NNF(a) is simply
b. The values of NNF for all coordinates are stored in an ar-
ray with the same dimensions of A.
It is a randomized algorithm that works iteratively impro-
ving the NNF until convergence. Initially it can be filled ei-
ther with random values sampled across image B or with
previous hint information. An iterative process is performed
to improve the NNF for a fixed number of iterations or un-
til a convergence criteria. The algorithm tries to improve
NNF using two sets of candidates : propagation and random
search. Propagation uses the known adjacent nearest neigh-
bors patches to improve NNF, and converges very quickly
but it may end in a local minimum. A second set of candi-
dates is used to avoid local minimum by introducing random
samples. This step is called random search, the candidates
are sampled from a distribution of pixels located at an expo-
nentially decreasing distance from each patch. For each pixel
vi, the candidates uiare sampled at an exponentially decrea-
sing distance ui=v0+wαiRiwhere Riis a uniform random
in [-1,1]x[-1,1], wis the maximum search radius and αis a
fixed ratio between window sizes (1/2 in the original paper).
After NNFs are calculated, a distance metric proposed by
Simakov [SCSI08] is used to guarantee both the coherence
and completeness of the output image.
3.2. Stereo HDR Generation
Most of current auto-stereoscopic displays accept from
five up to nine different stereo views [LLR13]. The ca-
c
AFIG 2014.
R. Ramirez Orozco & C. Loscos & I. Martin & A. Artusi / Génération de séquences d’images multivues HDR:vers la vidéo HDR
mera used to capture the input sequence is the Octo-
cam [PcPD10], a multiview camera prototype composed by
eight objectives disposed horizontally (Figure 2). The eight
sensors are synchronized in time and they produce four pairs
of different exposures of the same scene like shown in the Fi-
gure 1. The different exposures are achieved by mean of neu-
tral density filters in each objective [BLV12]. Filters fix the
percentage of light arriving to each sensor dividing by two
the amount of light received by each sensor. This is equi-
valent to keep constant the aperture and divide by two the
exposure time (1 F-stop) in a conventional digital camera.
The fact that the eight sensors are synchronized prevents of
capturing movement in the scene. The ghosting problem due
to dynamic objects does not exist because all the objectives
capture the scene at same time and same shutter speed.
There are some geometrical features that could help to re-
duce the number of potential corresponding patches. Since
all the objectives of the camera are aligned horizontally, the
resulting images have a different perspective but pixels in
different images of the sequence share the same epipolar
line. This fact reduces the random search to 1D (only on
the epipolar line). The use of a random search function ins-
tead of a full energy minimization scheme contributes also
to speed up the solution. We propose to modify the search
function to look only in the same horizontal line adding a
value in the vertical axis to prevent errors in case of geome-
trical miscalibration of images.
Figure 2: The Octo-cam multiview Camera prototype.
4. Experimental Results
This section shows some experimental results of our
proposal. We have tested our approach using different se-
quences of input images acquired both with the camera des-
cribed by Prevoteau et al. [PcPD10] and images from the
Middlebury Stereo Dataset [vis06]. Images from Middle-
bury are 8-bits .png files while from the octocam we get li-
near raw 10 bit per pixel images. From Middlebury sets we
have only 7 different views and from the Octo-cam, 8 views.
Our tests execute matching at two different stages of the
procedure : in LDR or in radiance domain. Figure 3 and Fi-
gure 4 show the result of matching directly the images with
different exposures. For tests of matching the radiance do-
main we calculated the camera response curve using RAN-
SAC’s method [MN99] to transform 8 bits images from
Middlebury. Despite that the matching is successful in both
cases, matching radiance domain converges much faster. Fi-
gure 3 shows the convergence of the PatchMatch used on
two different exposed images from the sequence of Figure 1
(a) Reference (b) Target
(c) Difference 1st Iteration (d) Reconstructed 1st Iteration
(e) Difference 5th Iteration (f) Reconstructed 5th Iteration
Figure 3: Images d) and f) show the result of registering b)
to a) for the first and the fifth iterations of our Patch Match
implementation. Images c) and e) shows the normalized SSD
difference between the result and the reference image
The two input images in Figure 4 are from the Middlebury
stereo dataset. In this test the matching was made on the ori-
ginal 8-bits LDR image with the same patch size than descri-
bed above. This results were obtained with a number of ite-
rations equal to 5, the normalized sum of squared difference
(SSD) error between the result and the reference images is
shown in Figure 4 (c). The biggest errors are located around
the color boundaries, it is important to compare results using
different similarity measures to avoid artifacts related with
the use of SSD for comparing colors. The influence of the
patch size and the computation time on the results need to
be addressed.
Figure 5 shows an example of the matching process
executed after transforming images to the same domain.
The convergence in this case is faster than matching 8-bits
images. The full test showed in Figure 6 was generated using
only two iterations per image. The images shown as in Fi-
gure 5 d) and e) are mapped back to 8-bits using the camera
response curve.
Figure 6 shows a full test over a set of seven images
from the Middlebury datased mentioned before. The resul-
ting HDR images have no visible ghosting effects. As we
can see in Figure 7, matching is not correct in very satura-
ted areas. This is a problem also for Sen et al. [SKY12].
This happens because there is no information in the refe-
rence image to compare with. For instance, letters are not
possible to recover for the lasts images in the sequence since
the whole sheet of paper is totally saturated. In future ap-
proaches it is needed to address such problem.
c
AFIG 2014.
R. Ramirez Orozco & C. Loscos & I. Martin & A. Artusi / Génération de séquences d’images multivues HDR:vers la vidéo HDR
(a) Reference (b) Target
(c) Difference 1st Iteration (d) Reconstructed 1st Iteration
(e) Difference 5th Iteration (f) Reconstructed 5th Iteration
Figure 4: Images d) and f) show the result of registering b)
to a) for the first and the fifth iterations of our Patch Match
implementation. Images c) and e) shows the normalized SSD
difference between the result and the reference image
5. Conclusions
This paper presents work in progress for image regis-
tration of image sequences for auto-stereoscopic 3D HDR
content creation. We propose to modify the PatchMatch al-
gorithm to make it more robust and suitable for stereo mat-
ching. The random search function has been modified to take
advantage of the epipolar geometry. Our method for stereo
HDR generation does not require disparity map calculation
or geometric calibration. We discussed about such modifica-
tions and presented some promising experimental results.
6. Acknowledgments
This work was partially financed by the Spanish Minis-
try of Science of Spain through the Innovation Sub-Program
Ramón y Cajal RYC-2011-09372. It is also supported by the
EU COST HDRi IC1005. Thanks a lot to Jennifer Bonnard
for her continuous help and to those who have helped us in
this work.
(a) Reference (b) Target
(c) Difference 1st Iteration (d) Reconstructed 1st Iteration
(e) Difference 2nd Iteration (f) Reconstructed 2dn Itera-
tion
Figure 5: Images d) and f) show the result of registering b)
to a) for two first iterations of our Patch Match implementa-
tion. Images c) and e) shows the normalized SSD difference
between the result and the reference image
Références
[BLV12] BONNA RD J., LOSCOS C., VALET TE G.,
NOURRIT J.-M., LUCAS L. : High-dynamic range video
acquisition with a multiview camera. Optics, Photonics,
and Digital Technologies for Multimedia Applications II
(2012), 84360A–84360A–11.
[BRG14] BÄTZ M., RICH TE R T., GAR BA S J.-U.,
PAPS T A., SEI LE R J., KAUP A. : High dynamic range
video reconstruction from a stereo camera setup. Si-
gnal Processing : Image Communication. Vol. 29, Num.
2 (2014), 191 – 202. Special Issue on Advances in High
Dynamic Range Video Research.
[BSFG09] BARNES C., SHECH TM AN E., FI NKELS TE IN
A., GOLDMAN D. B. : Patchmatch : A randomized cor-
respondence algorithm for structural image editing. ACM
Transactions on Graphics (Proc. SIGGRAPH). Vol. 28,
Num. 3 (aug 2009).
[BSGF10] BARNES C., SHECH TM AN E., GOLDMAN
D. B., FIN KEL ST EIN A. : The generalized PatchMatch
correspondence algorithm. In European Conference on
Computer Vision (sep 2010).
[DM97] DEBE VE C P., MA LIK J. : Recovering high dyna-
mic range radiance maps from photographs. In In procee-
dings of ACM SIGGRAPH (Computer Graphics) (1997),
vol. 31, pp. 369–378.
c
AFIG 2014.
R. Ramirez Orozco & C. Loscos & I. Martin & A. Artusi / Génération de séquences d’images multivues HDR:vers la vidéo HDR
Figure 6: On the left hand side, a set of LDR multi-view
images, on the right hand side the resulting tone mapped
HDR taking each LDR as reference respectively.
Figure 7: On the left hand side, a set of LDR 10 bits multi-
view images obtained from the Octo-cam, on the right hand
side the resulting tone mapped HDR.
c
AFIG 2014.
R. Ramirez Orozco & C. Loscos & I. Martin & A. Artusi / Génération de séquences d’images multivues HDR:vers la vidéo HDR
[HTM13] HADZIABDIC K. K., TE LALOVIC J. H., MAN-
TIUK R . : Comparison of deghosting algorithms for multi-
exposure high dynamic range imaging. In Proceedings of
the 29th Spring Conference on Computer Graphics (New
York, NY, USA, 2013), SCCG ’13, ACM, pp. 021 :21–
021 :28.
[IEE99] IEEE :. Dynamic range improvement through
multiple exposures (1999), IEEE.
[LC09] LIN H.-Y., CH AN G W.-Z. : High dynamic range
imaging for stereoscopic scene representation. In Image
Processing (ICIP), 2009 16th IEEE International Confe-
rence on (2009), pp. 4305–4308.
[LLR13] LUCA S L., LOSCOS C., REMION Y. : 3D Video
from Capture to Diffusion. Wiley-ISTE, octobre 2013.
[MG10] MANG IAT S., GIBSON J. : High dynamic range
video with ghost removal. Proc. SPIE. Vol. 7798 (2010),
779812–779812–8.
[MN99] MITS UNAGA T., NAYAR S. : Radiometric self
calibration. IEEE International Conf. Computer Vision
and Pattern Recognition. Vol. 1 (1999), 374–380.
[MP95] MANN S., PICARD R. W. : On Being undigi-
tal With Digital Cameras : Extending Dynamic Range
By Combining Differently Exposed Pictures. Percep-
tual Computing Section, Media Laboratory, Massachu-
setts Institute of Technology, 1995.
[NPR10] NIQUIN C., PÈVOST S., REMIO N Y. : An occlu-
sion approach with consistency constraint for multiscopic
depth extraction. International Journal of Digital Multi-
media Broadcasting (2010).
[PcPD10] PRÉVOTE AU J., CON PIOTI N S. C., DEBONS
D., LU CA S L., REM ION Y. : Multi-view shooting geome-
try for multiscopic rendering with controlled distortion.
International Journal of Digital Multimedia Broadcasting
(IJDMB), special issue Advances in 3DTV : Theory and
Practice. Vol. 2010 (mars 2010), 1–11.
[RBS03] ROB ERT SON M. A., BORMAN S., ST EVENS ON
R. L. : Estimation-theoretic approach to dynamic range
enhancement using multiple exposures. Journal of Elec-
tronic Imaging. Vol. 12, Num. 2 (2003), 219–228.
[Ruf11] RUF ENACH T D. : Stereoscopic High Dynamic
Range Video. PhD thesis, Ecole Polytechnique Fédérale
de Lausanne (EPFL), Switzerland, Agost 2011.
[SCSI08] SIMA KOV D., CASPI Y., SHECHT MAN E.,
IRANI M. : Summarizing visual data using bidirectio-
nal similarity. IEEE Conference on Computer Vision and
Pattern Recognition 2008 (CVPR’08) (2008).
[SDBRC14] SELM AN OVIC E., DEBATTISTA K.,
BASHFORD-ROGERS T., CHALMERS A. : Enabling ste-
reoscopic high dynamic range video. Signal Processing :
Image Communication. Vol. 29, Num. 2 (2014), 216 –
228. Special Issue on Advances in High Dynamic Range
Video Research.
[SKY12] SEN P., KALA NTARI N. K., YAESOUB I M.,
DARABI S., GOLDMAN D. B., SHE CH TMAN E. : Ro-
bust patch-based HDR reconstruction of dynamic scenes.
ACM Transactions on Graphics (Proceedings of SIG-
GRAPH Asia 2012). Vol. 31, Num. 6 (novembre 2012),
203 :1–203 :11.
[SMW10] SUN N., MA NSOUR H., WARD R. : Hdr image
construction from multi-exposed stereo ldr images. In
Proceedings of the IEEE International Conference on
Image Processing (ICIP) (Hong Kong, 2010).
[SS12] SRIK AN THA A., SIDIBÉ DÉSIRÉ D. : Ghost De-
tection and Removal for High Dynamic Range Images :
Recent Advances. Signal Processing : Image Communi-
cation (février 2012), 10.1016/j.image.2012.02.001. 23
pages.
[ST04] SAND P., TELLE R S. : Video matching. ACM
Transactions on Graphics. Vol. 23, Num. 3 (août 2004),
592–599.
[TCJHM14] TARA A., CHR IST IA N K., JI-HOC., MAR -
GRIT G. : Stereo hdr disparity map computation using
structured light. In HDRi2014 Second International
Conference and SME Workshop on HDR imaging (2014).
[THM13] TARA A., HYUNJIN Y., MARGRIT G. : A
framework for hdr stereo matching using multi-exposed
images. In Proceedings of HDRi2013 First International
Conference and SME Workshop on HDR imaging (Paper
no. 8, Oxford/Malden, 2013), The Eurographics Associa-
tion and Blackwell Publishing Ltd.
[TKS06] TRO CC OLI A., KANG S. B., SEITZ S. : Multi-
view multi-exposure stereo. In 3D Data Processing, Vi-
sualization, and Transmission, Third International Sym-
posium on (June 2006), pp. 861–868.
[vis06] VISION.MIDD LE BURY.EDU : Middlebury stereo
datasets. http ://vision.middlebury.edu/stereo/data/, 2006.
[ZF03] ZITOVA B., FL US SER J. : Image registration me-
thods : a survey. Image and Vision Computing. Vol. 21
(2003), 977–1000.
c
AFIG 2014.
ResearchGate has not been able to resolve any citations for this publication.
Conference Paper
Full-text available
We propose a principled approach to summarization of visual data (images or video) based on optimization of a well-defined similarity measure. The problem we consider is re-targeting (or summarization) of image/video data into smaller sizes. A good ldquovisual summaryrdquo should satisfy two properties: (1) it should contain as much as possible visual information from the input data; (2) it should introduce as few as possible new visual artifacts that were not in the input data (i.e., preserve visual coherence). We propose a bi-directional similarity measure which quantitatively captures these two requirements: Two signals S and T are considered visually similar if all patches of S (at multiple scales) are contained in T, and vice versa. The problem of summarization/re-targeting is posed as an optimization problem of this bi-directional similarity measure. We show summarization results for image and video data. We further show that the same approach can be used to address a variety of other problems, including automatic cropping, completion and synthesis of visual data, image collage, object removal, photo reshuffling and more.
Article
Full-text available
This paper aims to present a review of recent as well as classic image registration methods. Image registration is the process of overlaying images (two or more) of the same scene taken at different times, from different viewpoints, and/or by different sensors. The registration geometrically align two images (the reference and sensed images). The reviewed approaches are classified according to their nature (area-based and feature-based) and according to four basic steps of image registration procedure: feature detection, feature matching, mapping function design, and image transformation and resampling. Main contributions, advantages, and drawbacks of the methods are mentioned in the paper. Problematic issues of image registration and outlook for the future research are discussed too. The major goal of the paper is to provide a comprehensive reference source for the researchers involved in image registration, regardless of particular application areas.
Conference Paper
Full-text available
Multi-view stereo algorithms typically rely on same-exposure images as inputs due to the brightness constancy assumption. While state-of-the-art depth results are excellent, they do not produce high-dynamic range textures required for high-quality view reconstruction. In this paper, we propose a technique that adapts multi-view stereo for different exposure inputs to simultaneously recover reliable dense depth and high dynamic range textures. In our technique, we use an exposure-invariant similarity statistic to establish correspondences, through which we robustly extract the camera radiometric response function and the image exposures. This enables us to then convert all images to radiance space and selectively use the radiance data for dense depth and high dynamic range texture recovery. We show results for synthetic and real scenes.
Conference Paper
Full-text available
In this paper, we present an algorithm that generates high dynamic range (HDR) images from multi-exposed low dynamic range (LDR) stereo images. The vast majority of cameras in the market only capture a limited dynamic range of a scene. Our algorithm first computes the disparity map between the stereo images. The disparity map is used to compute the camera response function which in turn results in the scene radiance maps. A refinement step for the disparity map is then applied to eliminate edge artifacts in the final HDR image. Existing methods generate HDR images of good quality for still or slow motion scenes, but give defects when the motion is fast. Our algorithm can deal with images taken during fast motion scenes and tolerate saturation and radiometric changes better than other stereo matching algorithms.
Article
We present a new approach for improving the effective dynamic range of cameras by using multiple photographs of the same scene taken with different exposure times. Using this method enables the photographer to accurately capture scenes that contain high dynamic range by using a device with low dynamic range, which allows the capture of scenes that have both very bright and very dark regions. We approach the problem from a probabilistic standpoint, distinguishing it from the other methods reported in the literature on photographic dynamic range improvement. A new method is proposed for determining the camera's response function, which is an iterative procedure that need be done only once for a particular camera. With the response function known, high dynamic range images can be easily constructed by a weighted average of the input images. The particular form of weighting is controlled by the probabilistic formulation of the problem, and results in higher weight being assigned to pixels taken at longer exposure times. The advantages of this new weighting scheme are explained by com- parison with other methods in the literature. Experimental results are presented to demonstrate the utility of the algorithm. © 2003 SPIE
Article
High dynamic range (HDR) imaging from a set of sequential exposures is an easy way to capture high-quality images of static scenes, but suffers from artifacts for scenes with significant motion. In this paper, we propose a new approach to HDR reconstruction that draws information from all the exposures but is more robust to camera/scene motion than previous techniques. Our algorithm is based on a novel patch-based energy-minimization formulation that integrates alignment and reconstruction in a joint optimization through an equation we call the HDR image synthesis equation. This allows us to produce an HDR result that is aligned to one of the exposures yet contains information from all of them. We present results that show considerable improvement over previous approaches.
Article
Stereoscopic and high dynamic range (HDR) imaging are two methods that enhance video content by respectively improving depth perception and light representation. A large body of research has looked into each of these technologies independently, but very little work has attempted to combine them due to limitations in capture and display; HDR video capture (for a wide range of exposure values over 20 f-stops) is not yet commercially available and few prototype HDR video cameras exist. In this work we propose techniques which facilitate stereoscopic high dynamic range (SHDR) video capture by using an HDR and LDR camera pair. Three methods are proposed: one based on generating the missing HDR frame by warping the existing one using a disparity map; increasing the range of LDR video using a novel expansion operator; and a hybrid of the two where expansion is used for pixels within the LDR range and warping for the rest. Generated videos were compared to the ground truth SHDR video captured using two HDR video cameras. Results show little overall error and demonstrate that the hybrid method produces the least error of the presented methods.
Article
We propose a new method for ghost-free high dynamic range (HDR) video taken with a camera that captures alternating short and long exposures. These exposures may be combined using traditional HDR techniques, however motion in a dynamic scene will lead to ghosting artifacts. Due to occlusions and fast moving objects, a gradient-based optical flow motion compensation method will fail to eliminate all ghosting. As such, we perform simpler block-based motion estimation and refine the motion vectors in saturated regions using color similarity in the adjacent frames. The block-based search allows motion to be calculated directly between adjacent frames over a larger search range, yet at the cost of decreased motion fidelity. To address this, we investigate a new method to fix registration errors and block artifacts using a cross-bilateral filter to preserve the edges and structure of the original frame while retaining the HDR color information. Results show promising dynamic range expansion for videos with fast local motion.
Article
High dynamic range (HDR) image generation and display technologies are becoming increasingly popular in various applications. A standard and commonly used approach to obtain an HDR image is the multiple exposures' fusion technique which consists of combining multiple images of the same scene with varying exposure times. However, if the scene is not static during the sequence acquisition, moving objects manifest themselves as ghosting artefacts in the final HDR image. Detecting and removing ghosting artefacts is an important issue for automatically generating HDR images of dynamic scenes. The aim of this paper is to provide an up-to-date review of the recently proposed methods for ghost-free HDR image generation. Moreover, a classification and comparison of the reviewed methods is reported to serve as a useful guide for future research on this topic.