Figure 1 - uploaded by Sotirios Tsaftaris
Content may be subject to copyright.
Pablo Picasso, La Joie de Vivre, 1946 (Musée Picasso, Antibes, France, 1946.1.4); oleoresinous enamel paint and charcoal on fibrocement, 120 × 250 cm. Present state ©Emilie Hubert, CICRP, Marseille (Photo) ©Estate of Pablo Picasso/Artists Rights Society (ARS), New York. 

Pablo Picasso, La Joie de Vivre, 1946 (Musée Picasso, Antibes, France, 1946.1.4); oleoresinous enamel paint and charcoal on fibrocement, 120 × 250 cm. Present state ©Emilie Hubert, CICRP, Marseille (Photo) ©Estate of Pablo Picasso/Artists Rights Society (ARS), New York. 

Source publication
Article
Full-text available
This paper describes the use of a customized algorithm for the colorization of historical black and white photographs documenting earlier states of paintings. This study specifically focuses on Pablo Picasso's mid-century Mediterranean masterpiece La Joie de Vivre, 1946 (Musée Picasso, Antibes, France). The custom-designed algorithm allows computer...

Contexts in source publication

Context 1
... woman, 40 years his junior, who would soon make him a father again (Gilot & Lake, 1964). That is, in essence, the Joy of Life (La Joie de Vivre) (LJDV): Françoise Gilot, depicted as the central nymph dancing in front of an Eden-like landscape of water and earth, accompanied by a faun and a centaur playing the double flute and two small goats (Fig. 1). The title given to this large fibrocement panel (120 × 250 cm) by Dor de la Souchère, director of the museum at the time and the first to exhibit this major Picasso work (Giraudy, 1981-2, p. 2) perfectly encapsulates this celebration of a newfound freedom and of creative renewal. This painting is also quite unique among the ...
Context 2
... colorization experiment, but ethically this would be hard to justify given the exceptional condition of the work. In any case, the images illustrated in Figs. 2 and 3 remain extremely valuable close approximations that for the first time allow full appreciation of the rhythmic chromatic per- mutations of this masterpiece. By way of comparison, Fig. 10 illustrates a detail of S 4 colorized with the auto- mated algorithm described here (at left), as opposed to the more traditional, more subjective, and labor inten- sive colorization using Adobe Photoshop ® tools by a manual operator. It is apparent that the manually colored image results in a rather flat, dull, and less color-accurate ...
Context 3
... linear optimization problem is cast with respect to the pixels with unknown color component (e.g. U) in a 3 × 3 pixel neighborhood around the pixels with known color (Fig. 11 illustrates an example). Once all pixels in the B&W image have been considered and equations have been cast for all of them, the sol- ution to the problem can be had by solving the system of linear equations. This process thus allows the U and V chrominance components to be estimated for all the pixels in the historical photographs, ...
Context 4
... wise addition), where X is the S 4 image, and X' is the desired corrected image. The pixels in the S 4 image above a certain threshold, set as 70% of the maximum intensity were selected to estimate the field F. Upon estimation, the corrected image could be recovered based on the following equation: X ′ = X − F. The estimated field F is shown in Fig. 12. At this point, a further normalization step of the intensities of X ′ was required. This was achieved by comparing the average intensity of the hand-drum held up by Françoise in the final state with that of Figure 11 The colorization algorithm develops a set of equations for each 3 × 3 neighborhood of pixels (right) in the unknown ...
Context 5
... this point, a further normalization step of the intensities of X ′ was required. This was achieved by comparing the average intensity of the hand-drum held up by Françoise in the final state with that of Figure 11 The colorization algorithm develops a set of equations for each 3 × 3 neighborhood of pixels (right) in the unknown chrominance matrix. Here for illustration purposes the top-right pixel (x i−1,j+1 ) is shown in gray, to denote that for this pixel the color information is known. ...
Context 6
... ′′ is the S 4 image where all illumination distortions are fully corrected, and it is the image that was colorized with the method described in the paper. Figure 12 Illumination field as estimated using the approach described in text corresponding to the fourth state, shown in Fig. 3. ...

Citations

... Grandin and Centauro 2015 Bartoletti et al. 2020Mecklenburg and Tumosa 1991Pietro and Ligterink 1999 Frøysaker 2003Ružić et al. 2011Iedema et al. 2014Tsaftaris et al. 2014Oakley et al. 2015Hendriks et al. 2017 ...
Conference Paper
In conservation, mock-ups are routinely used as surrogate works of art that can be subjected to treatments proposed for use on the original objects. This paper investigates the role of mock-ups in conservation research, specifically, research into dirt removal from the monumental Aula paintings by Edvard Munch housed at the University of Oslo. The mock-ups, prepared to support an empirical evaluation of a selection of novel cleaning systems, inspired reflections on the broader socio-material role of mock-ups in research. This paper relates the philosophical basis for the use of mock-ups in conservation to aspects of perspectivism and applies categories and terminology borrowed from the medical sciences to paintings research. Through the case study, the research context, notion of material agency and the roles of mock-ups in conservation ethics, research and practice are explored.
... Image colorization is a challenging, inherently stochastic task that requires a semantic understanding of the scene as well as knowledge of the world. Core immediate applications of the technique include producing organic new colorizations of existing image and video content as well as giving life to originally grayscale media, such as old archival images (Tsaftaris et al., 2014), videos (Geshwind, 1986) and black-and-white cartoons (Sỳkora et al., 2004;Qu et al., 2006;Cinarel & Zhang, 2017). Colorization also has important technical uses as a way to learn meaningful representations without explicit supervision (Zhang et al., 2016;Larsson et al., 2016;Vondrick et al., 2018) or as an unsupervised data augmentation technique, whereby diverse semantics-preserving colorizations of labelled images are produced with a colorization model trained on a potentially much larger set of unlabelled images. ...
... Colorization methods have initially relied on human-in-the-loop approaches to provide hints in the form of scribbles (Levin et al., 2004;Ironi et al., 2005;Huang et al., 2005;Yatziv & Sapiro, 2006;Qu et al., 2006;Luan et al., 2007;Tsaftaris et al., 2014;Ci et al., 2018) and exemplar-based techniques that involve identifying a reference source image to copy colors from (Reinhard et al., 2001;Welsh et al., 2002;Tai et al., 2005;Ironi et al., 2005;Pitié et al., 2007;Morimoto et al., 2009;Gupta et al., 2012;Xiao et al., 2020). Exemplar based techniques have been recently extended to video as well (Zhang et al., 2019a). ...
Preprint
We present the Colorization Transformer, a novel approach for diverse high fidelity image colorization based on self-attention. Given a grayscale image, the colorization proceeds in three steps. We first use a conditional autoregressive transformer to produce a low resolution coarse coloring of the grayscale image. Our architecture adopts conditional transformer layers to effectively condition grayscale input. Two subsequent fully parallel networks upsample the coarse colored low resolution image into a finely colored high resolution image. Sampling from the Colorization Transformer produces diverse colorings whose fidelity outperforms the previous state-of-the-art on colorising ImageNet based on FID results and based on a human evaluation in a Mechanical Turk test. Remarkably, in more than 60% of cases human evaluators prefer the highest rated among three generated colorings over the ground truth. The code and pre-trained checkpoints for Colorization Transformer are publicly available at https://github.com/google-research/google-research/tree/master/coltran
... Given colorization's importance across multiple applications (e.g., historical photographs and videos [117], artist assistance [95,113]), much research strives to make it cheaper and less time-consuming [13,18,19,23,38,54,71,83,125]. However, most methods still require some level of user input [13,19,38,54,71,103]. ...
Article
The success of deep learning in computer vision is rooted in the ability of deep networks to scale up model complexity as demanded by challenging visual tasks. As complexity is increased, so is the need for large amounts of labeled data to train the model. This is associated with a costly human annotation effort. To address this concern, with the long-term goal of leveraging the abundance of cheap unlabeled data, we explore methods of unsupervised "pre-training." In particular, we propose to use self-supervised automatic image colorization. We show that traditional methods for unsupervised learning, such as layer-wise clustering or autoencoders, remain inferior to supervised pre-training. In search for an alternative, we develop a fully automatic image colorization method. Our method sets a new state-of-the-art in revitalizing old black-and-white photography, without requiring human effort or expertise. Additionally, it gives us a method for self-supervised representation learning. In order for the model to appropriately re-color a grayscale object, it must first be able to identify it. This ability, learned entirely self-supervised, can be used to improve other visual tasks, such as classification and semantic segmentation. As a future direction for self-supervision, we investigate if multiple proxy tasks can be combined to improve generalization. This turns out to be a challenging open problem. We hope that our contributions to this endeavor will provide a foundation for future efforts in making self-supervision compete with supervised pre-training.
... Building a computer system that can automatically convert a black and white image to a plausible color image is useful for restoring old photographs, videos [34], or even assisting cartoon artists [26,32]. From a computer vision perspective, this may appear like a straightforward image-toimage mapping problem, amenable to a convolutional neural network (CNN). ...
Article
We propose a novel approach to automatically produce multiple colorized versions of a grayscale image. Our method results from the observation that the task of automated colorization is relatively easy given a low-resolution version of the color image. We first train a conditional PixelCNN to generate a low resolution color for a given grayscale image. Then, given the generated low-resolution color image and the original grayscale image as inputs, we train a second CNN to generate a high-resolution colorization of an image. We demonstrate that our approach produces more diverse and plausible colorizations than existing methods, as judged by human raters in a "Visual Turing Test".
... Building a computer system that can automatically convert a black and white image to a plausible color image is useful for restoring old photographs, videos [34], or even assisting cartoon artists [26,32]. From a computer vision perspective, this may appear like a straightforward image-to-image mapping problem, amenable to a convolutional neural network (CNN). ...
... Given its importance across multiple applications (e.g. historical photographs and videos [1], artist assistance [2,3]), much research strives to make colorization cheaper and less time-consuming [4][5][6][7][8][9][10][11][12]. However, most methods still require some level of user input [5-7, 9, 10, 13] Fig. 2: System overview. ...
... Deshpande et al. [11], test a separate task where the ground-truth histogram in the two non-lightness color channels of the original color image is made available. 1 In order to compare, we propose two histogram transfer methods. We refer to the predicted image as the source and the ground-truth image as the target. ...
Article
We develop a fully automatic image colorization system. Our approach leverages recent advances in deep networks, exploiting both low-level and semantic representations during colorization. As many scene elements naturally appear according to multimodal color distributions, we train our model to predict per-pixel color histograms. This intermediate output can be used to automatically generate a color image, or further manipulated prior to image formation; our experiments consider both scenarios. On both fully and partially automatic colorization tasks, our system significantly outperforms all existing methods.
Full-text available
Article
Image colorization is an important and difficult problem in image processing with various applications including image stylization and heritage restoration. Most existing image colorization methods utilize feature matching between the reference color image and the target grayscale image. The effectiveness of features is often significantly affected by the characteristics of the local image region. Traditional methods usually combine multiple features to improve the matching performance. However, the same set of features is still applied to the whole images. In this paper, based on the observation that local regions have different characteristics and hence different features may work more effectively, we propose a novel image colorization method using automatic feature selection with the results fused via a Markov Random Field (MRF) model for improved consistency. More specifically, the proposed algorithm automatically classifies image regions as either uniform or non-uniform, and selects a suitable feature vector for each local patch of the target image to determine the colorization results. For this purpose, a descriptor based on luminance deviation is used to estimate the probability of each patch being uniform or non-uniform, and the same descriptor is also used for calculating the label cost of the MRF model to determine which feature vector should be selected for each patch. In addition, the similarity between the luminance of the neighborhood is used as the smoothness cost for the MRF model which enhances the local consistency of the colorization results. Experimental results on a variety of images show that our method outperforms several state-of-the-art algorithms, both visually and quantitatively using standard measures and a user study.
Conference Paper
We develop a fully automatic image colorization system. Our approach leverages recent advances in deep networks, exploiting both low-level and semantic representations. As many scene elements naturally appear according to multimodal color distributions, we train our model to predict per-pixel color histograms. This intermediate output can be used to automatically generate a color image, or further manipulated prior to image formation. On both fully and partially automatic colorization tasks, we outperform existing methods. We also explore colorization as a vehicle for self-supervised visual representation learning.
Conference Paper
Image colorization has been a topic of interest since the mid 70's and several algorithms have been proposed that given a grayscale image and color scribbles (hints) produce a colorized image. Recently, this approach has been introduced in the field of art conservation and cultural heritage, where B&W photographs of paintings at previous stages have been colorized. However, the questions of what is the minimum number of scribbles necessary and where they should be placed in an image remain unexplored. Here we address this limitation using an iterative algorithm that provides insights as to the relationship between locally vs. globally important scribbles. Given a color image we randomly select scribbles and we attempt to color the grayscale version of the original. We define a scribble contribution measure based on the reconstruction error. We demonstrate our approach using a widely used colorization algorithm and images from a Picasso painting and the peppers test image. We show that areas isolated by thick brushstrokes or areas with high textural variation are locally important but contribute very little to the overall representation accuracy. We also find that for the case of Picasso on average 10% of scribble coverage is enough and that flat areas can be presented by few scribbles. The proposed method can be used verbatim to test any colorization algorithm.