Adaptive Local Illumination Change Compensation Method for H.264/AVC-Based Multiview Video Coding

Abstract

In multiview video, illumination changes can occur due to imperfect camera calibration and variations of the camera position and direction. These characteristics can cause performance degradation in multiview video coding (MVC) that uses inter-view prediction by referring to the pictures obtained from the neighboring views. In order to overcome this problem, an adaptive local illumination change compensation method is proposed. In the proposed method, we compensate for the illumination changes in the macroblock (MB) unit between the current picture and the reference picture under the assumption that the DC component in the MB is influenced by the local illumination changes. By using the proposed method, the compression ratio in the multiview video coding was increased, and a 0.1~0.6 dB peak signal-to-noise ratio (PSNR) improvement was obtained compared with the case where the proposed method was not used.

Full text

Page 1

1496IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 17, NO. 11, NOVEMBER 2007
Adaptive Local Illumination Change Compensation
Method for H.264/AVC-Based Multiview
Video Coding
Jae-Ho Hur, Sukhee Cho, and Yung-Lyul Lee, Senior Member, IEEE
Abstract—In multiview video, illumination changes can occur
due to imperfect camera calibration and variations of the camera
position and direction. These characteristics can cause perfor-
mance degradation in multiview video coding (MVC) that uses
inter-viewpredictionbyreferringtothepicturesobtainedfromthe
neighboring views. In order to overcome this problem, an adaptive
local illumination change compensation method is proposed. In
the proposed method, we compensate for the illumination changes
in the macroblock (MB) unit between the current picture and the
reference picture under the assumption that the DC component in
the MB is influenced by the local illumination changes. By using
the proposed method, the compression ratio in the multiview video
coding was increased, and a 0.1
(PSNR) improvement was obtained compared with the case where
the proposed method was not used.
0.6 dB peak signal-to-noise ratio
Index Terms—Block matching algorithm (BMA), H.264/AVC,
illumination change compensation, inter-view prediction, mean-
removed sum of absolute differences (MRSAD), multiview video
coding (MVC).
I. INTRODUCTION
M
video coding experts group (VCEG) and ISO/IEC moving pic-
ture experts group (MPEG), is expected to become a new video
coding standard for the realization of future video applications
such as 3D-TV and free viewpoint video [1]. The MVC group
in the JVT has chosen the H.264/AVC [2]-based MVC method
that was proposed by [3] as the MVC reference model, since
this method showed better coding efficiency than H.264/AVC
simulcast coding and the other methods that were submitted in
response to the call for proposals made by the MPEG [4].
However, when illumination changes occur between pictures
in the view-temporal direction, the displacement vector estima-
tion (DVE), which is the motion vector estimation or disparity
vector estimation in MVC, cannot be accurately performed, so
that the error of the displacement vector (DV) and the amounts
ULTIVIEW video coding (MVC), which is being stan-
dardized in the joint video team (JVT) of the ITU-T
Manuscript received December 16, 2006. This work was supported in part by
Seoul R&DB program (11098). This paper was recommended by Guest Editor
Y. He.
J.-H. Hur and Y.-L. Lee are with the Computer Engineering Department,
Sejong University, DMS Laboratory, Seoul 143-747, Korea (e-mail: jhhur@
dms.sejong.ac.kr; yllee@sejong.ac.kr).
S. Cho is with the Electronics and Telecommunication Research Institute,
Broadcasting Systems Research Group, Taegeon 305-707, Korea (e-mail:
shee@etri.re.kr).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TCSVT.2007.903774
of the residual signals may be increased and, consequently, the
coding efficiency may be decreased. The reason why this phe-
nomenonistakensoseriouslyisthattheperformanceofMVCis
affected not only by local illumination changes due to the varia-
tionsintheviewingpositionanddirection,butalsobytheglobal
signal changes due to imperfect camera calibration.
In order to compensate for these illumination changes, sev-
eral methods [5]–[11], [15] have been proposed. Fundamen-
tally, the algorithms in [5]–[7] use two factors, viz. the scale and
offset, to compensate for the discrepancy induced by the illu-
mination changes. In order to calculate these factors, the least
square method is used in [5] and [6], and the variance and mean
valuesareusedin[7].Also,the2-DLloydmaxalgorithmisused
to efficiently represent the scale and offset factors in [7]. How-
ever, these algorithms require high computational complexity
while achieving relatively low coding efficiency, making it dif-
ficult to use them in MVC.
In this paper, we propose a macroblock (MB)-based adaptive
illumination change compensation (MBAIC) method for the lu-
minance components in MVC. The illumination change is com-
pensated for in the MB unit between the current picture and
the reference picture under the assumption that the DC compo-
nent in the MB is influenced by the local illumination change.
The proposed method is applied to displacement vector estima-
tion/compensation by using a modified block matching mea-
sure, that is, the so-called mean-removed sum of absolute dif-
ferences (MRSAD).
Therestofthepaperisorganizedasfollows.InSectionII,the
H.264/AVC-based MVC reference model is introduced briefly.
In Section III, the detailed algorithm of the proposed method is
described and the integration of the proposed method into the
MVC reference model is described. In Section IV, the perfor-
mance of the proposed method is compared to that of the MVC
reference model not using the proposed method by analyzing
the various experimental results.
II. H.264/AVC-BASED MVC
As already mentioned in the previous section, the
H.264/AVC-based MVC method that is chosen as a ref-
erence model for the standardization of MVC has shown
significant coding efficiency. Fig. 1 depicts an example of the
H.264/AVC-based MVC structure, in which there are eight
parallel views. As shown in Fig. 1, this structure utilizes the
hierarchical B pictures, which not only improves the coding
efficiency, but also provides temporal scalability. This structure
can be divided into three kinds of picture sets, i.e., the picture
1051-8215/$25.00 © 2007 IEEE

Page 2

HUR et al.: ILLUMINATION CHANGE COMPENSATION METHOD FOR H.264/AVC-BASED MULTIVIEW VIDEO CODING1497
Fig. 1. H.264/AVC-based MVC structure on 1-D parallel arrangement with eight cameras.
set predicted by the inter-view pictures on the view axis, the
picture set predicted by the temporal pictures on the tem-
poral axis, and the picture set predicted by the view-temporal
(spatio-temporal) pictures on the view and temporal axes. In
this structure, the
and pictures are only used at random
access points. The
pictures are used between the
pictures that are random access points and are used in all
other positions except at these random access points, where the
subscript
means the temporal decomposition level. The
pictures on the temporal axis T0 are predicted spatially, the
pictures on the view axes V0, V2, V4, and V6 are predicted
temporally, and the
pictures on the view axes V1, V3, V5,
and V7 are predicted temporally and spatially. Our proposed
method is integrated into the MVC structure shown in Fig. 1.
and
III. MB-BASED ADAPTIVE ILLUMINATION CHANGE
COMPENSATION (MBAIC) METHOD
Basically, the proposed illumination change compensation
method is based on a hybrid video coding structure and is
performed adaptively in the MB unit. As shown in Fig. 2(a),
the proposed encoder performs adaptive DVE using MB-based
illumination change compensation and the prediction of the
difference value of an illumination change (DVIC), which is
an offset factor used for the MB-based illumination change
compensation. By using adaptive DVE, the illumination change
can be compensated for locally. Also, DVIC is efficiently coded
through the proposed prediction method of DVIC. As shown in
Fig.2(b), theproposed decoder performs adaptivedisplacement
and illumination change compensation.
A. Adaptive DVE Using the MB-Based Illumination Change
Compensation
In the H.264/AVC-based MVC reference model, the sum of
absolute differences (SAD) is used as a measure for the DVE.
The currentframe is denoted by
, where and are defined by the index of the column and
row, respectively, and the reference frame is denoted by
The SAD calculation of the
8
16, 88, 8 4, 48, and 4
with spatial coordinates
.
blocks, viz. 16
4, is performed as follows:
16, 168,
(1)
where
the start position of each MB in the current frame.
Whentheilluminationchangeoccursbetweendifferentviews
in the multiview video, the correlation between these different
views may be decreased. In this case, the conventional SAD
measure cannot be used appropriately for inter-view prediction.
In order to overcome this problem, an alternative measure is
needed instead of SAD.
The proposed illumination change compensation method as-
sumes that the discrepancy caused by the illumination change
is equal to the discrepancy of the DC component between the
represents a candidate DV and represents

Page 3

1498IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 17, NO. 11, NOVEMBER 2007
Fig. 2. Block diagram ofthe proposedMB-based adaptiveilluminationchange
compensation method. (a) Encoded. (b) Decoder.
current block and the reference block. In order to mitigate the
discrepancy during DVE, the mean values of the current block
andthereferenceblockaresubtractedfromeachother.Byusing
this concept, the MRSAD is used as a measure for the DVE as
follows:
(2)
(3)
where
pixels in the current block and the reference block, respectively.
andare the average value for all
In the implementation of the proposed method,
both set to 16, because the use of a variable block-size MRSAD
does not give rise to any improvement of the coding efficiency
which would offset the resulting increase of the computational
complexity. In (2) and (3),
while
is calculated for every candidate reference
block pointed to by
. The DV
MRSAD is computed, and the DVIC is calculated in the MB
unit as follows:
andare
is calculated only one time,
minimizing the
(4)
The DVIC means an offset factor which is used for the com-
pensation of the illumination change in the MB unit. By using
the DVIC, the displacement and illumination change-compen-
sated residual signal in the
calculated as follows:
th MB of the current frame is
(5)
where
tion change-compensated residual signal to be transformed and
quantizedinthe
th MBofthecurrentframe,and
,.
For the adaptive local illumination compensation, the SAD-
based conventional coding procedure or the proposed MRSAD-
based coding procedure can be used for coding the current MB,
for which a 1-bit flag (mb_ic_flag) per MB is needed to signal
to the decoder whether the proposed coding procedure is used
or not. On the decoder side, if mb_ic_flag is 0, the illumination
change compensation is not performed for the current MB, i.e.,
the conventional decoding procedure is performed. Otherwise,
the displacement and illumination change compensation is per-
formed as follows:
represents the displacement and illumina-
(6)
where
tion-compensated residual signal which is reconstructed by in-
verse quantization and inverse transform in the
represents the displacement and illumina-
th MB of
the current frame,
MB of the current frame, and
To enable the illumination change compensation in the de-
coder, the DVIC should be transmitted to the decoder in an ap-
propriate way. For encoding the DVIC, the prediction error of
represents the reconstructedth
,.

Page 4

HUR et al.: ILLUMINATION CHANGE COMPENSATION METHOD FOR H.264/AVC-BASED MULTIVIEW VIDEO CODING1499
Fig.3. (a)AveragevalueoftheMSEbetweentheDVICsofthecurrentMBand
each neighbor MBforvarious testsequences.(b)Causalneighbor MBs usedfor
the prediction of the current DVIC.
the DVIC is coded by means of the differential pulse code mod-
ulation (DPCM) of the DVIC and its prediction value, which
is calculated by using the DVICs of the neighboring blocks of
the current block. In the decoder, the DVIC is decoded through
inverse DPCM with the prediction value which is calculated in
the same way as that in the encoder. The prediction process of
DVIC is explained in the next subsection.
B. Prediction Process of DVIC
The DVIC corresponds to a local illumination change af-
fecting the DC component in each MB. Since the area over
which the local illumination-change occurs is usually larger
thantheareaofone MB,themagnitudeofthelocal illumination
change of the current MB may have a high correlation with
that of the neighboring MBs. Therefore, in order to reduce the
number of bits required to encode the DVIC, the DVIC is coded
by DPCM with the predictor (
DVIC of the causal neighbor MBs, as shown in Fig. 3.
The prediction process was developed by analyzing the
average value of the mean squared error (MSE) between the
DVICs of the current MB and each neighboring MB for various
test sequences. Fig. 3(a) shows the MSE average value of the
DVICs between the current MB and neighboring MBs. There-
fore, the neighbor block search order for the DVIC prediction
for eachneighboring blockin Fig.3(b) becomesA,B, C,and D,
according to the smaller MSE average value of the neighboring
blocks in Fig. 3(a). If there are no neighboring MBs whose
reference index is equal to that of the current MB even if the
mb_ic_flags of the neighboring MBs are equal to 1, median fil-
tering is performed to calculate the value of
three neighboring MBs (MBs A, B and C shown in Fig. 3(b)).
The prediction process of DVIC is as follows:
Step 1) If the upper MB, A, of the current MB in Fig. 3(b)
has a DVIC and its reference index is the same as
that of the current MB,
of MB A and the process is finalized.
Step 2) If the left MB, B, of the current MB has a DVIC and
its reference index is the same as that of the current
MB,
is set to the DVIC of MB B and the
process is finalized.
Step 3) If the upper-right MB, C, of the current MB has a
DVIC and its reference index is the same as that of
the current MB,
C and the process is finalized.
Step 4) If the upper-left MB, D, of the current MB has a
DVIC and its reference index is the same as that of
) obtained from the
by using
is set to the DVIC
is set to the DVIC of MB
Fig. 4. Probability distribution of the DVIC and the prediction error of DVIC
(dpcm_of_dvic) in the “V” picture set. (a) Race1. (b) Akko & Kayo.
the current MB,
D and the process is finalized.
Step 5) If MB A, MB B and MB C have their own DVICs,
then median filtering is performed with the three
neighboring MBs.
the median filtering and the process is finalized.
Step 6)
is set to 0.
After obtaining the prediction value of DVIC through the
above steps,
is subtracted from the DVIC and its pre-
diction error (dpcm_of_dvic
the entropy coding method.
Probability distribution of the DVIC and the prediction error
of DVIC (dpcm_of_dvic) in the “V” picture set for Race1 and
Akko & Kayo sequences are plotted in Fig. 4. As shown in
Fig. 4, the distribution of dpcm_of_dvic is concentrated at zero,
as if it were a zero-mean Laplacian distribution, while the dis-
tribution of DVIC is irregular. Therefore, the prediction error of
DVIC is coded instead of the DVIC in this paper.
is set to the DVIC of MB
is set to the result of
DVIC-) is coded by

Page 5

1500 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 17, NO. 11, NOVEMBER 2007
C. Integration of the MVC Reference Model and the Proposed
MBAIC
To evaluate its performance, the proposed MBAIC method
is integrated into the MVC reference model described in
Section II. The proposed method is only applied to each
16
16 block for a luminance component (Y), even though the
variable block-size adaptive illumination change compensation,
viz. 16
8, 816, 8 8, 8 4, 4
could be applied. This is because additional information, such
as the illumination compensation indication bit per block (not
MB) and dpcm_of_dvic bits per block (not MB), would have
to be transmitted to the decoder, with the result that no coding
gain would be achieved, due to the increase in the number of
these information-signalingbits. Also, thereis verylittle coding
gain when the proposed method is applied to the chrominance
components (U and V).
The proposed method is applied to the three MB modes ex-
isting in H.264/AVC, viz. Inter 16
Skip mode (in P slice), and Direct 16
In the case of Skip mode, the DV is derived by the existing
derivation process of the motion vector predictor specified in
[2] and the DVIC is calculated using the average value of the
DVICs of the neighboring MBs. This idea was proposed by
Thomson/USC in [11], and was submitted to the MVC stan-
dardization of JVT as a joint proposal with our algorithm. In
the case of Direct 16
16 mode, the DV that is derived by the
existing derivation process of Direct mode [12] is used and the
DVIC is calculated by (4), in which the mean value of the ref-
erence block is calculated with the derived DV. As a reference,
the spatial Direct mode is used for our experiments.
Inordertocompensateforthelocalilluminationchangesadap-
tively, mb_ic_flag is needed for each Inter 16
16
16 block mode. On the encoder side, the rate-constrained
codercontrol[13]isusedtodecidewhetherthelocalillumination
changecompensationisneededforeachMBornot.Althoughthe
current MB is determined as “Direct Skip”, which is a skipped
MB mode in the B slice, mb_ic_flag and dpcm_of_dvic need to
be transmitted to the decoder for the current MB. mb_ic_flag
anddpcm_of_dvicareencodedusingthecontext-basedadaptive
binary arithmetic coding (CABAC) method [14].
8, and 44, in (2) and (3)
16 mode (in P or B slice),
16 mode (in B slice).
16 and Direct
IV. EXPERIMENTAL RESULTS
To demonstrate the effectiveness of the proposed MBAIC
method,severalexperimentalresultsareprovided.First,theper-
formance of DVE is demonstrated by comparing DVE using
SAD with DVE using MRSAD. Next, the rate-distortion (
performance is shown for each picture set depicted in Fig. 1 and
for various MVC test sequences, in which the reference model
(RM)withtheproposedMBAICmethodiscomparedtotheRM
with variable block-size adaptive illumination change compen-
sation, the RM without any illumination change compensation,
and the RM with the weighted prediction (WP) [15] adopted in
H.264/AVC. Finally, a comparison of the encoding time com-
plexity is shown.
For our experiments, we used the JSVM 6.5 reference soft-
ware1that supports MVC mode and performed the experiments
)
1From CVS Repository of JSVM.
TABLE I
TEST SEQUENCES WITH DIFFERENT IMAGE PROPERTIES AND DIFFERENT
CAMERA ARRANGEMENTS FOR MULTIVIEW VIDEO CODING
on various MVC test sequences with different image properties
and camera arrangements of the YUV 4:2:0 format, as shown in
Table I. Basically, the proposed method is performed for tem-
poral prediction as well as inter-view prediction, and the coding
tools specified in the Main and FRExt profiles, such as the de-
blocking filter, CABAC, and adaptive block transform (4
8
8 size) are used for the encoding simulation. For more in-
formation regarding the test conditions, refer to [3] and [16].
4 or
A. Comparison of the Performances of DVE Using SAD and
MRSAD
To compare the DVE using SAD with the DVE using
MRSAD, the disparity vectors are extracted in the 16
block unit by using the T0/V2 picture of Fig. 1 as the reference
frame and the T0/V3 picture of Fig. 1 as the current frame in the
“Race1” sequence, as shown in Fig. 5(a) and (b), respectively.
Fig. 5(c) and (d) depict two kinds of disparity vector field.
Fig. 5(c) and (d) depict the disparity vector fields extracted by
the DVEs using SAD and MRSAD, respectively. The disparity
vector field depicted in Fig. 5(c) is very irregular and rough in
some local areas. As we predicted, it can be seen that the local
illumination change deteriorates the encoding performance by
disturbing the DVE using SAD. On the other hand, the disparity
vector field depicted in Fig. 5(d) is relatively smoother than
that in Fig. 5(c) and looks more like a real disparity vector.
Through these results, we can intuitively conclude that the
DVE using MRSAD can improve the disparity vector coding
by using a more smooth disparity vector field, and enhance the
overall encoding performance by compensating for the local
illumination changes when they occur.
16
B. Comparison of
As already mentioned in Section II, the pictures coded by the
H.264/AVC-based MVC reference model can be classified into
threepicturesets,asshowninFig.1,viz.the“V”,“T”,and“V/T”
picture sets. The three
-curves per sequence are shown
in Fig. 6 for the RM and the RM with the proposed MBAIC
(RM+MBAIC). In the “V” picture set, the highest coding gain
-Performance According to Picture Set