No full-text available
To read the full-text of this research,
you can request a copy directly from the author.
Evaluation of 3D CG Image Colorization Quality Using Visible Digital Watermarking after Noise Removal Based on Sparse Dictionary Learning
ResearchGate has not been able to resolve any citations for this publication.
It has become a trend in recent years to use deep neural networks for colorization. However, previous methods often encounter problems with edge color leakage and difficulties in obtaining a plausible color output from the Euclidean distance. To solve these problems, we propose a new adversarial edge-aware image colorization method with multitask output combined with semantic segmentation. The system uses a generator with a deep semantic fusion structure to infer semantic clues in a given grayscale image under chroma conditions and learns colorization by simultaneously predicting color information and semantic information. In addition, we also use a specific color difference loss with characteristics of human visual observation that is combined with semantic segmentation loss and adversarial loss for training. The experimental results show that our method is superior to existing methods in terms of different quality metrics and achieves good results in image colorization.
Techniques of using convolutional neural networks (CNNs) to colorize monochrome still images have been widely researched. However, the results of automatic colorization are often different from the user's intentions and historical fact. A lot of color correction work still needs to be done in order to produce a colorized video. This is a major problem in situations such as broadcasting production where footage must be appropriately colorized in accordance with historical fact. In this article, we propose a practical video colorization framework that can easily reflect the user's intentions. The proposed framework uses a combination of two CNNs--a user-guided still-image-colorization CNN and a color-propagation CNN--that allows the correction work to be performed efficiently. The user-guided still-image-colorization CNN produces key frames by colorizing several monochrome frames from the target video on the basis of user-specified colors and color-boundary information. The color-propagation CNN automatically colorizes the entire video on the basis of the key frames, while suppressing discontinuous changes in color between frames. A quantitative evaluation showed that it is possible to produce color video reflecting the user's intention with less effort than with earlier methods.
Legacy grayscale aerial photographs represent one of the main available sources for studying the past state of the environment and its relationship to its current state. However, these photos lack spectral information, thereby hindering their use in current remote sensing approaches that rely on spectral data for characterizing surfaces. This paper proposes a Conditional Generative Adversarial Network, a deep learning model, to enrich legacy photographs by colorizing an input grayscale image. The technique was used to colorize two orthophotographs (taken in 1956 and 1978) covering the entire Eurometropole de Strasbourg. To assess the model's performances, two strategies were proposed: (1) colorized photographs were evaluated with metrics such as mean squared error, peak signal-to-noise ratio, and structural similarity index; (2) Random Forest classifications were performed to automatically extract land cover classes from grayscale and colorized photographs respectively. The results revealed the strong performances of the colorization system, indicating that the model successfully learned the mapping between grayscale and color aerial over a large territory. More nuanced results were observed for specific land cover classes such as crops. Moreover, land cover classifications performed on colorized data showed significant improvements over grayscale photos; respectively +6% and +17% for 1956 and 1978. It was concluded that deep learning models are powerful tools for improving radiometric properties of old aerial photographs and land cover mapping. We also argue that the proposed approach could serve as a basis for further developments aiming to promote the use of National Aerial Archives for landscapes reconstruction.
Image colorization aims to produce a natural looking color image from a given grayscale image, which remains a challenging problem. In this paper, we propose a novel examplebased image colorization method exploiting a new locality consistent sparse representation. Given a single reference color image, our method automatically colorizes the target grayscale image by sparse pursuit. For efficiency and robustness, our method operates at the superpixel level. We extract low-level intensity features, mid-level texture features and high-level semantic features for each superpixel, which are then concatenated to form its descriptor. The collection of feature vectors for all the superpixels from the reference image composes the dictionary. We formulate colorization of target superpixels as a dictionary-based sparse reconstruction problem. Inspired by the observation that superpixels with similar spatial location and/or feature representation are likely to match spatially close regions from the reference image, we further introduce a locality promoting regularization term into the energy formulation which substantially improves the matching consistency and subsequent colorization results. Target superpixels are colorized based on the chrominance information from the dominant reference superpixels. Finally, to further improve coherence while preserving sharpness, we develop a new edge-preserving filter for chrominance channels with the guidance from the target grayscale image. To the best of our knowledge, this is the first work on sparse pursuit image colorization from single reference images. Experimental results demonstrate that our colorization method outperforms state-ofthe- art methods, both visually and quantitatively using a user study.
The work of Levin et al. (2004) popularised strokebased methods that add color to gray value images according to a small amount of user-specified color samples. Even though such reconstructions from sparse data suggest a possible use in compression, only few attempts were made so far in this direction. Diffusion-based compression methods pursue a similar idea: They store only few image pixels and inpaint the missing regions. Despite this close relation and a lack of diffusion-based color codecs, colorization ideas were so far only integrated into transform-based approaches such as JPEG. We address this missing link with two contributions. First, we show the relation between the discrete colorization of Levin et al. and continuous diffusion-based inpainting in the YCbCr color space. It decomposes the image into a luma (brightness) channel and two chroma (color) channels. Our luma-guided diffusion framework steers the diffusion inpainting in the chroma channels according to the structure in the luma channel. We show that making the luma-guided colorization anisotropic outperforms the method of Levin et al. significantly. Secondly, we propose a new luma preference codec that invests a large fraction of the bit budget into an accurate representation of the luma channel. This allows a high-quality reconstruction of color data with our colorization technique. Simultaneously we exploit the fact that the human visual system is more sensitive to structural than to color information. Our experiments demonstrate that our new codec outperforms the state-of-the-art in diffusion-based image compression and is competitive to transform-based codecs.
Thus far, we have studied multi-view 3D image quality evaluation, including visible digital watermarking, in the case of considering coded degradation, region, resistance, and number of viewpoints. As a result, the more we process watermarking in the low frequency domain, the more the assessment tends toward independence from image or video patterns. However, we do not consider color information in the case where we generate digital watermarking images. In this paper, first, for the 3D CG images encoded and decoded by H.265/HEVC with 8 Viewpoints Parallax Barrier Method, we process the wavelet transformation, and perform embedding for the arrangement of the high and low frequency domain (LL3, HL3, LH3) in the same frequency level per viewpoint. Next, we evaluate the generated image, analyze the results, and classify for RGB pattern and coded degradation using SVM (Support Vector Machine).
We propose a new technique for video colorization based on spatiotemporal colour propagation in the three dimensional video volume, utilizing the dominant orientation response obtained from the steerable pyramid decomposition of the video. The volumetric colour diffusion from the sources which are marked by scribbles occurs across spatiotemporally smooth regions, and the prevention of leakage is facilitated by the spatiotemporal discontinuities in the output of steerable filters, representing the object boundaries and motion boundaries. Unlike most of the existing methods, our approach dispenses with the need of motion vectors for interframe colour transfer, and provides a general framework for image and video colorization. Experimental results establish the effectiveness of the proposed approach in colorizing videos having different types of motion and visual content even in the presence of occlusion, in terms of accuracy and computational requirements.
We present a new scheme for video colorization using optimization in rotation-aware Gabor feature space. Most current methods of video colorization incur temporal artifacts and prohibitive processing costs, while this approach is designed in a spatiotemporal manner to preserve temporal coherence. The parallel implementation on graphics hardware is also facilitated to achieve realtime performance of color optimization. By adaptively clustering video frames and extending Gabor filtering to optical flow computation, we can achieve real-time color propagation within and between frames. Temporal coherence is further refined through user scribbles in video frames. The experimental results demonstrate that our proposed approach is efficient in producing high-quality colorized videos.
Colorization is a method that adds color components to a grayscale image using only a few representative pixels provided by the user. A novel approach to image compression called colorization-based coding has recently been proposed. It automatically extracts representative pixels from an original color image at an encoder and restores a full color image by using colorization at a decoder. However, previous studies on colorization-based coding extract redundant representative pixels and do not extract the pixels required for suppressing coding error. This paper focuses on the colorization basis that restricts the decoded color components. From this viewpoint, we propose a new colorization-based coding method. Experimental results revealed that our method can drastically suppress the information amount (number of representative pixels) compared conventional colorization based-coding while objective quality is maintained.
Colonization is a computer-assisted process of adding color to a monochrome image or movie. The process typically involves segmenting images into regions and tracking these regions across image sequences. Neither of these tasks can be performed reliably in practice; consequently, colonization requires considerable user intervention and remains a tedious, time-consuming, and expensive task. In this paper we present a simple colorization method that requires neither precise image segmentation, nor accurate region tracking. Our method is based on a simple premise: neighboring pixels in space-time that have similar intensities should have similar colors. We formalize this premise using a quadratic cost function and obtain an optimization problem that can be solved efficiently using standard techniques. In our approach an artist only needs to annotate the image with a few color scribbles, and the indicated colors are automatically propagated in both space and time to produce a fully colorized image or sequence. We demonstrate that high quality colorizations of stills and movie clips may be obtained from a relatively modest amount of user input.
In this paper, first, we carried out dictionary learning to process the sparse coding in advance, and then, we added six types of noise for 3D CG images. Next, we processed noise removal based on sparse coding theory and dictionary learning. Before and after image processing, we discussed improvement of image quality evaluation value eventually by measuring PSNR.
In this paper, we propose a fully automatic image colorization method for grayscale images using neural network and optimization. For a determined training set including the gray images and its corresponding color images, our method segments grayscale images into superpixels and then extracts features of particular points of interest in each superpixel. The obtained features and their RGB values are given as input for, the training colorization neural network of each pixel. To achieve a better image colorization effect in shorter time, our method further propagates the resulting color points to neighboring pixels for improved colorization results. In the propagation of color, we present a cost function to formalize the premise that neighboring pixels should have the maximum positive similarity of intensities and colors; we then propose our solution to solving the optimization problem. At last, a guided image filter is employed to refine the colorized image. Experiments on a wide variety of images show that the proposed algorithms can achieve superior performance over the state-of-the-art algorithms.
Image colorization refers to a computer-assisted process that adds colors to grayscale images. It is a challenging task since there is usually no one-to-one correspondence between color and local texture. In this paper, we tackle this issue by exploiting weighted nonlocal self-similarity and local consistency constraints at the resolution of superpixels. Given a grayscale target image, we first select a color source image containing similar segments to target image and extract multi-level features of each superpixel in both images after superpixel segmentation. Then a set of color candidates for each target superpixel is selected by adopting a top-down feature matching scheme with confidence assignment. Finally, we propose a variational approach to determine the most appropriate color for each target superpixel from color candidates. Experiments demonstrate the effectiveness of the proposed method and show its superiority to other state-of-the-art methods. Furthermore, our method can be easily extended to color transfer between two color images.
In our previous studies, we studied on the multi-view 3D CG image quality evaluation including visible digital watermarking. Particularly, we verified for the multimedia evaluation including both the coded image quality and watermark quality. Actually, the image quality of watermarking is not always better in case we carried out the visible digital watermarking. Therefore, depending on the situation, we need to change the coded image quality of watermarking. In this paper, we used 3D CG images with 8 viewpoints parallax barrier method, which embedded the watermarking image encoded and decoded by H.265/HEVC in advance by transforming frequency domain for the generated images. And then, we composed the generated images. We carried out the subjective quality evaluation of these images, and then we analyzed results, and classified evaluation values by using Support Vector Machine (SVM). Furthermore, we considered for the application procedure of watermarking by comparing mutually to the case of considering the coded image quality of watermarking.
In our previous studies, we studied on the multi-view 3D CG image quality evaluation including visible digital watermarking. Particularly, we verified for the multimedia evaluation including both the coded image quality and watermark quality. Actually, the image quality of watermarking is not always better in case we carried out the visible digital watermarking. Therefore, depending on the situation, we need to change the coded image quality of watermarking. In this paper, we used 3D CG images with 8 viewpoints parallax barrier method, which embedded the watermarking image encoded and decoded by H.265/HEVC in advance by transforming frequency domain for the generated images. And then, we composed the generated images. We carried out the subjective quality evaluation of these images, and then we analyzed results, and classified evaluation values by using Support Vector Machine (SVM). Furthermore, we considered for the application procedure of watermarking by comparing mutually to the case of considering the coded image quality of watermarking.
Previously, it is not obvious to what extent was accepted for the assessors when we see the 3D image (including multi-view 3D) which the luminance change may affect the stereoscopic effect and assessment generally. We think that we can conduct a general evaluation, along with a subjective evaluation, of the luminance component using both the S-CIELAB color space and CIEDE2000. In this study, first, we performed three types of subjective evaluation experiments for contrast enhancement in an image by using the eight viewpoints parallax barrier method. Next, we analyzed the results statistically by using a support vector machine (SVM). Further, we objectively evaluated the luminance value measurement by using CIEDE2000 in the S-CIELAB color space. Then, we checked whether the objective evaluation value was related to the subjective evaluation value. From results, we were able to see the characteristic relationship between subjective assessment and objective assessment.
In this paper, we experimented the subjective evaluation for 3D CG image including the gray scale region with 8 viewpoints parallax barrier method, and we analyzed this result statistically. Next, we measured about the relation between the luminance change and the color difference by using S-CIELAB color space and CIEDE2000. As a result, we obtained knowledge about the relation among the coded image quality, the contrast enhancement, and gray scale.
In this paper, we formulate the colorization based coding problem into an optimization problem, i.e., an L1 minimization problem. In colorization based coding, the encoder chooses a few representative pixels for which the chrominance values and the positions are sent to the decoder, while in the decoder, the chrominance values for all the pixels are reconstructed by colorization methods. The main issue in colorization based coding is how to extract the representative pixels well so that the compression rate and the quality of the reconstructed color image becomes good. By formulating the colorization based coding into an L1 minimization problem, it is guaranteed that, given the colorization matrix, the chosen set of representative pixels becomes the optimal set in the sense that it minimizes the error between the original and the reconstructed color image. In other words, for a fixed error value and a given colorization matrix, the chosen set of representative pixels is the smallest set possible. We also propose a method to construct the colorization matrix which colorizes the image in a multi-scale manner. This, combined with the proposed representative pixels extraction method, allows us to choose a very small set of representative pixels. It is shown experimentally that the proposed method outperforms conventional colorization based coding methods as well as the JPEG standard and is comparable to the JPEG2000 compression standard, both in terms of the compression rate and the quality of the reconstructed color image.
Colorization is the process that restores colors on a grayscale image from user-generated color-assignation information. A novel approach to image compression has recently been proposed that extracts such color assignation from an input color image (we called this inverse colorization). Previous studies on inverse colorization have represented color assignation as a set of color points. However, in regions with flat color and fluctuating luminance, numerous color points are needed to correctly resore the color. Therefore, we propose a novel method of inverse colorization that represents color assignation in a more human-like manner - as a set of color line segments. To extract the minimum necessary line segments from an input image, iterative updating of tentative color assignation was introduced. The experimental results revealed that our proposed method can drastically suppress color information compared to either conventional inverse colorization or JPEG.
Colorization, the task of coloring a grayscale image or video, involves assigning from the single dimension of intensity or luminance a quantity that varies in three dimensions, such as red, green, and blue channels. Mapping between intensity and color is, therefore, not unique, and colorization is ambiguous in nature and requires some amount of human interaction or external information. A computationally simple, yet effective, approach of colorization is presented in this paper. The method is fast and it can be conveniently used "on the fly," permitting the user to interactively get the desired results promptly after providing a reduced set of chrominance scribbles. Based on the concepts of luminance-weighted chrominance blending and fast intrinsic distance computations, high-quality colorization results for still images and video are obtained at a fraction of the complexity and computational cost of previously reported techniques. Possible extensions of the algorithm introduced here included the capability of changing the colors of an existing color image or video, as well as changing the underlying luminance, and many other special effects demonstrated here.
Colorization refers to an image processing task which recovers color in grayscale images when only small regions with color are given. We propose a couple of variational models using chromaticity color components to colorize black and white images. We first consider total variation minimizing (TV) colorization which is an extension from TV inpainting to color using chromaticity model. Second, we further modify our model to weighted harmonic maps for colorization. This model adds edge information from the brightness data, while it reconstructs smooth color values for each homogeneous region. We introduce penalized versions of the variational models, we analyze their convergence properties, and we present numerical results including extension to texture colorization.
A Practical Monochrome Video Colorization Framework for Broadcast Program Production
- R Endo
- Y Kawai
- T Mochizuki
R. Endo, Y. Kawai, and T. Mochizuki: "A Practical Monochrome
Video Colorization Framework for Broadcast Program Production,"
IEEE Trans. on Broadcasting, Vol.67, No.1, pp.225-237, March 2021.
3D CG Image Noise Removal and Quality Assessment Fig. 11. Experimental result (Poisson noise)
- N Kawabata
N. Kawabata: "3D CG Image Noise Removal and Quality Assessment
Fig. 11. Experimental result (Poisson noise)
(In horizontal order from top left to bottom, Museum (Q = 0.01, 0.05, 0.1),
Wonder World (Q = 0.01, 0.05, 0.1))