Multiview Video Coding Using View Interpolation and Color Correction

Universal Media Res. Center, Tokyo
IEEE Transactions on Circuits and Systems for Video Technology (Impact Factor: 2.26). 12/2007; DOI: 10.1109/TCSVT.2007.903802
Source: IEEE Xplore

ABSTRACT Neighboring views must be highly correlated in multiview video systems. We should therefore use various neighboring views to efficiently compress videos. There are many approaches to doing this. However, most of these treat pictures of other views in the same way as they treat pictures of the current view, i.e., pictures of other views are used as reference pictures (inter-view prediction). We introduce two approaches to improving compression efficiency in this paper. The first is by synthesizing pictures at a given time and a given position by using view interpolation and using them as reference pictures (view-interpolation prediction). In other words, we tried to compensate for geometry to obtain precise predictions. The second approach is to correct the luminance and chrominance of other views by using lookup tables to compensate for photoelectric variations in individual cameras. We implemented these ideas in H.264/AVC with inter-view prediction and confirmed that they worked well. The experimental results revealed that these ideas can reduce the number of generated bits by approximately 15% without loss of PSNR.

  • [Show abstract] [Hide abstract]
    ABSTRACT: Multi-view video plus depth (MVD) format is considered as the next-generation standard for advanced 3D video systems. MVD consists of multiple color videos with a depth value associated with each texture pixel. Relying on this representation and by using depth-image-based rendering techniques, new viewpoints for multi-view video applications can be generated. However, since MVD is captured from different viewing angles with different cameras, significant illumination and color differences can be observed between views. These color mismatches degrade the performance of view rendering algorithms by introducing visible artifacts leading to a reduced view synthesis quality. To cope with this issue, we propose an effective method for correcting color inconsistencies in MVD. Firstly, to avoid occlusion problems and allow performing correction in the most accurate way, we consider only the overlapping region when calculating the color mapping function. These common regions are determined using a reliable feature matching technique. Also, to maintain the temporal coherence, correction is applied on a temporal sliding window. Experimental results show that the proposed method reduces the color difference between views and improves view rendering process providing high-quality results.
    Signal Image and Video Processing 01/2015; DOI:10.1007/s11760-015-0761-9 · 1.02 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: View synthesis (VS) and depth map generation (DMG) are two of the most crucial operations for facilitating the development of free-viewpoint 3D video systems. This demand for high computational complexity has motivated the development of the proposed design. Conventional designs have directly adopted dedicated hardware to increase the speed of these operations independently, which has resulted in algorithmic-level redundancies. Thus, the conventional per-pixel computation results in high redundancy in both VS and DMG operations. Accordingly, this paper proposes a shape-adaptive low-complexity (SALC) technique for removing such algorithmic-level redundancies for both VS and DMG operations. Moreover, adopting shape-adaptive searching blocks can improve the accuracy of stereo matching (SM), which subsequently enhances the resulting VS. To overcome the problem of border misalignment in VS, a novel edge alignment technique is presented for detecting and aligning misaligned borders. In addition, an unreliable region repairing (URR) technique was adopted to enhance the accuracy of DMG operations, thereby optimizing the image quality. The proposed SALC technique was verified using \(C\) /Matlab simulations. Through the presented complexity reduction without any hardware acceleration, this design can achieve a high performance of 48 fps for synthesizing the virtual views of the CIF format (352 \(\times \) 288) based on a portable platform equipped with a 1 GHz ARM Cortex-A9 RISC processor only. The experimental results revealed that the proposed SALC technique saves up to 96 % computational complexity for VS, and the computation time of DMG operations is reduced to approximately 2 % of that required for conventional SM. The average improved image quality was 32.46 dB (i.e., 7.33 % improvement from the original 30.46 dB) when using the presented edge alignment technique. When considering the overall efficiency evaluation, the proposed SALC-based SM method is 172 times more efficient than the conventional fixed block-sized SM based on the combined indices of the BPCT product. Compared with the designs of previous studies, the proposed SALC-based design obtained superior efficiency in both VS and DMG as a result of the computation reuse property.
    Circuits Systems and Signal Processing 02/2014; 34(2). DOI:10.1007/s00034-014-9870-x · 1.26 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: When compared to conventional 2-D video, multiview video can significantly enhance the visual 3-D experience in 3-D applications by offering horizontal parallax. However, when processing images originating from different views, it is common that the colors between the different cameras are not well- calibrated . To solve this problem, a novel energy function -based color correction method for multiview camera setups is proposed to enforce that colors are as close as possible to those in the reference image but also that the overall structural information is well-preserved. The proposed system introduces a spatio-temporal correspondence matching method to ensure that each pixel in the input image gets bijectively mapped to a reference pixel. By combining this mapping with the original structural information, we construct a global optimization algorithm in a Laplacian matrix formulation and solve it using a sparse matrix solver. We further introduce a novel forward-reverse objective evaluation model to overcome the problem of lack of ground truth in this field. The visual comparisons are shown to outperform state-of-the-art multiview color correction methods, while the objective evaluation reports PSNR gains of up to 1.34 dB and SSIM gains of up to 3.2%, respectively.
    IEEE Transactions on Multimedia 05/2015; 17(5):577-590. DOI:10.1109/TMM.2015.2412879 · 1.78 Impact Factor