No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Flow-Based Image Abstraction
Abstract
We present a non-photorealistic rendering technique that automatically delivers a stylized abstraction of a photograph. Our approach is based on shape/color filtering guided by a vector field that describes the flow of salient features in the image. This flow-based filtering significantly improves the abstraction performance in terms of feature enhancement and stylization. Our method is simple, fast, and easy to implement. Experimental results demonstrate the effectiveness of our method in producing stylistic and feature-enhancing illustrations from photographs.
... Artistic sketch is a genre of fine arts that has long been beloved. Many researchers have proposed various schemes for synthesizing sketches from images [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19]. However, many of them focus on synthesizing portrait sketches [10,11,[13][14][15]20]. ...
... Various sketch synthesizing methods can be categorized into four groups: (1) pre-deep learning methods [1][2][3]28] that use mathematical principles to extract prominent lines and to create hatching patterns using kernel-based techniques, (2) general deep learning schemes [4][5][6][7][8][9][10][11] including neural style transfer (NST) or generative adversarial networks (GAN) that synthesize sketch images from input images, (3) specialized deep learning schemes [12][13][14][15][16][17][18][19][20] designed for sketch synthesis, such as portrait sketch synthesis, forensic facial sketch synthesis, and architectural sketch synthesis, and (4) diffusion model-based methods [24][25][26][29][30][31][32][33][34], which are relatively less explored for sketch synthesis due to the dearth of high-quality sketch datasets. ...
... A successive approach [2] describes an image abstraction and stylization method that acknowledges the direction of the local image structure in shape or color filtering. Notably, the Flow-based bilateral (FBL) smoothing and Edge Tangent Flow (ETF) presented in this work enhance the quality of the extracted lines. ...
We propose a framework that synthesizes artistic landscape sketches using a diffusion model-based approach. Furthermore, we suggest a three-channel perspective map (3CPM) that mimics the artistic skill used by real artists. We employ Stable Diffusion, which leads us to use ControlNet to process 3CPM in Stable Diffusion. Additionally, we adopt the Low Rank Adaptation (LoRA) method to fine-tune our framework, thereby enhancing the quality of sketch and resolving the color-remaining problem, which is a frequently observed artifact in the sketch images using diffusion models. We implement a bimodal sketch generation interface: text to sketch and image to sketch. In producing a sketch, a guide token is used so that our method synthesizes an artistic sketch in both cases. Finally, we evaluate our framework using quantitative and quantitative schemes. Various sketch images synthesized by our framework demonstrate the excellence of our study.
... In the fields of image processing, computer vision, and computer graphics, many authors have presented line extraction studies that express the shape of objects by using lines [7][8][9][10][11]. Recently, line extraction studies have been accelerated on a large scale, due to the progress of deep learning technology. ...
... Line sketching is a drawing technique where the shape of a character is depicted using only lines of various lengths and widths. Before deep-learning-based techniques, various techniques for detecting edges and lines, based on the high-frequency components of images, had been proposed [7][8][9]. However, these techniques were limited, as they did not consider the content information of the image. ...
... A visual comparison of our proposed method to tonal sketch generation techniques [21,22,24], as well as to line-based sketch generation techniques [8,10,11], was also conducted. As depicted in Figure 7, our results present reduced noise and clearly portray various elements, such as facial contours and hair, resulting in a better representation of the input photo's identity. ...
A deep-learning-based model for generating line-based portrait sketches from portrait photos is proposed in this paper. The misalignment problem is addressed by the introduction of a novel loss term, designed to tolerate misalignments between Ground Truth sketches and generated sketches. Artists’ sketching strategies are mimicked by dividing the portrait into face and hair regions, with separate models trained for each region, and the outcomes subsequently combined. Our contributions include the resolution of misalignment between photos and artist-created sketches, and high-quality sketch results via region-based model training. The experimental results show the effectiveness of our approach in generating convincing portrait sketches, with both quantitative and visual comparisons to State-of-the-Art techniques. The quantitative comparisons demonstrate that our method preserves the identity of the input portrait photos, while applying the style of Ground Truth sketch.
... We chose three most representative artistic media effects used in computer graphics and computer vision society: watercolor, color pencil drawing and abstraction with lines. We use Kang et al.'s work [8] for abstraction with lines, Bousseau et al.'s work [2] for watercolor and Yang et al.'s work [3] for color pencil drawing. ...
... Abstraction is a drawing technique that removes tiny textures and abstracts complex colors to present an object's coarse shape. Many researchers [1,8,[13][14][15][16] present abstraction techniques for photograph colors and tiny textures. They also presented various line drawing schemes for clearly presenting the distinguishing shape of objects [17][18][19]. ...
... We train our model by artistic images synthesized from existing non-photorealistic (NPR) studies. Kang et al.'s work [8] generates abstracted images by integrating color along smooth flow embedded in an image. This scheme can generate both abstracted images and abstracted images with lines. ...
We present a generative model with spatial control to synthesize dual-artistic media effects. It generates different artistic media effects on the foreground and background of an image. In order to apply a distinct artistic media effect to a photograph, deep learning-based models require a training dataset composed of pairs of a photograph and its corresponding artwork images. To build the dataset, we apply some existing techniques that generate an artwork image including colored pencil, watercolor and abstraction from a photograph. In order to produce a dual artistic effect, we apply a semantic segmentation technique to separate the foreground and background of a photograph. Our model applies different artistic media effects on the foreground and background using space control module such as SPADE block.
... In our approach, stylization schemes are employed to render images in some game scene style. The stylization schemes we employ include flow-based image abstraction with coherent lines [37], color abstraction using bilateral filters [38] and deep cartoon-styled rendering [39]. ...
... In our approach, we augmented 3000 images by applying three stylization schemes [37][38][39] and retrained object detection algorithms. Some of the augmented images are suggested in Figure 6. ...
... Figure 7 illustrates the comparison of three approaches: (i) trained with Pascal VOC, (ii) retrained with augmentation and (iii) retrained with game scenes. [37], (c) is produced by color abstraction using a bilateral filter [38] and (d) is produced by deep cartoon-styled rendering [39]. ...
The advancement and popularity of computer games make game scene analysis one of the most interesting research topics in the computer vision society. Among the various computer vision techniques, we employ object detection algorithms for the analysis, since they can both recognize and localize objects in a scene. However, applying the existing object detection algorithms for analyzing game scenes does not guarantee a desired performance, since the algorithms are trained using datasets collected from the real world. In order to achieve a desired performance for analyzing game scenes, we built a dataset by collecting game scenes and retrained the object detection algorithms pre-trained with the datasets from the real world. We selected five object detection algorithms, namely YOLOv3, Faster R-CNN, SSD, FPN and EfficientDet, and eight games from various game genres including first-person shooting, role-playing, sports, and driving. PascalVOC and MS COCO were employed for the pre-training of the object detection algorithms. We proved the improvement in the performance that comes from our strategy in two aspects: recognition and localization. The improvement in recognition performance was measured using mean average precision (mAP) and the improvement in localization using intersection over union (IoU).
... This is followed by color quantization technique to get partial stylization image which reduces the over illumination in the image. Finally, Kang proposed FBL [22] filtering is applied over the desired focused region to obtain the artistic stylization output. ...
... For underexposed, low-illuminated and standard complex images Laplacian filtering on its own may not give the desired result. Hence, many scholars have attempted to develop various edge and structure aware filtering approaches such as bilateral filtering approach by Tomasi and Manduchi [53]1998, Sylvain Paris et al., [51] local laplacian filtering, Bilateral texture filtering by Cho [52], Geodesic Filtering by David Mould [20], Domain transform filtering by Gastal et al., [54], Tree filtering by Linchao Bao [21], guided image filtering by Sun et al., [56], Anisotropic filtering by Perona and Malik [10], fast local laplacian filtering by Aubry et al., [55], Flow based difference of gaussian and flow-based -Bilateral filtering by H Kang et al., [22]. Although all these edge and structure aware filtering approaches may perform well for standard adequate data set pictures sample, they do not give desired results for underexposed and low-illuminated images especially for images with rich texture patterns. ...
... Accepted manuscript to appear in VJCS blurring. In this work, we adopted the Henry kang proposed FBL filtering [22] and Kang FBL is based on ETF and if we increase the radii of the filtering kernel, then image gets smoothened. Hence, this filtering is most useful in line extraction and region smoothening and it produces the effective artistic stylization output. ...
This work identifies the strong dominant features by its location and extracts the image features for the purpose of automatic desire focusing on prominent structure and artistic stylization of images. At the pre-processing level, dataset image is treated using refined structure preserving image abstraction framework which can deliver the best effectual structure preserved abstracted results by utilizing visual attributes from 2D color image. The presented framework efficiently conserves the structural characteristics in the foreground of an input image by exhaustively amalgamate the series of non-photorealistic rendering image filters over meticulous investigational work and it also reduces the background substance of an image. The framework assesses image and object space details to generate structure preserved image abstraction thus distinguishing the accentuated elements of an enhanced structures using Harris key-point feature detector and chooses the 100 major unique dominant feature locations among available features. This work automatically selects the unique location from the extracted features using polynomial region of interest and unselected image regions and its background are blurred using Gaussian motion blurring with point spread function. Deblurring the selected region using wiener filtering to get the desire focusing on prominent structure followed by color quantization and flow-based bilateral filtering is applied over focused structural region to achieve artistic stylization. Efficiency of the framework has been validated by carrying out the trials on the selected Flickr repository, David Mould and Ruixing Wang dataset. In addition, user’s visual opinion and the image quality estimation methods were also utilized to appraise the proposed pre-processing framework. This work lists the structure preserving image abstraction framework applications, limitation, execution difficulties and future work in the field of Non-photorealistic rendering domain.
... To minimize this problem and obtain more clear boundaries and abstraction results, some studies have proposed new smoothing techniques [5,43]. However, region segmentation using mean-shift filters is useful for merging regions, but is not suitable for generating sophisticated lines [20]. ...
... Methods for extracting lines from an image include Canny [1], Mean-shift segmentation [4], and DoG filtering [26]. The proposed framework uses techniques of Winnemoller et al. [44] and Kang et al. [20] based on the DoG filter. As a result of applying various line extraction techniques, we obtained the best results using Kang et al [20]'s method of adjusting the DoG filter according to the ETF (Edge Tangent Flow) to improve the quality of the line. ...
... The proposed framework uses techniques of Winnemoller et al. [44] and Kang et al. [20] based on the DoG filter. As a result of applying various line extraction techniques, we obtained the best results using Kang et al [20]'s method of adjusting the DoG filter according to the ETF (Edge Tangent Flow) to improve the quality of the line. ...
In this paper, we provide a layered non-photorealistic rendering (NPR) technique that automatically extracts the depth of field (DoF) shown in the picture and adjusts the degree of abstraction accordingly. We use an RGB channel to efficiently classify the DoF region anisotropically. Based on the DoF values, we abstract the color and adjust the thickness of the line. We use anisotropic DoF-based filtering to improve the abstraction quality by finding the blur region using cross-correlation filtering and anisotropically calculating the weight map. Our approach has greatly improved the quality of abstraction in terms of performance and design. The algorithm is also fast and simple to implement. Experimental results show well the characteristics and style of the DoF of the original photograph.
... We extend GIST to artistic Style Transfer by amplifying the textural details of the style image and integrating them with the content details in the representation space. Specifically, we extract the Edge Tangent Flow (ETF) [40] of the style image, compute its multiscale representation, and align the content detail subbands with it using a fusion-based strategy. ...
... where Ω x denotes the neighborhood of coordinate x ∈ Z 2 , τ a normalization factor, ϕ x,y ∈ {−1, 1} a vector alignment function, w s x,y a spatial weight function, w m x,y a magnitude weight function, and w d x,y a direction weight function. For a comprehensive explanation, refer to Kang et al. [28,40]. ...
State-of-the-art Style Transfer methods often leverage pre-trained encoders optimized for discriminative tasks, which may not be ideal for image synthesis. This can result in significant artifacts and loss of photorealism. Motivated by the ability of multiscale geometric image representations to capture fine-grained details and global structure, we propose GIST: Geometric-based Image Style Transfer, a novel Style Transfer technique that exploits the geometric properties of content and style images. GIST replaces the standard Neural Style Transfer autoencoding framework with a multiscale image expansion, preserving scene details without the need for post-processing or training. Our method matches multiresolution and multidirectional representations such as Wavelets and Contourlets by solving an optimal transport problem, leading to an efficient texture transferring. Experiments show that GIST is on-par or outperforms recent photorealistic Style Transfer approaches while significantly reducing the processing time with no model training.
... Therefore, the stroke direction should follow the tangent of the edges. Specifically, we use the ETF vector field (Edge Tangent Flow [25,26]) to find the tangent of the edges and use the anchor's ETF direction as . Here briefly reviews the calculation of ETF: compute the input image's gradient vector field (modulus and direction at each pixel); rotate the directions by 2 ; tend the direction of the pixels with a small modulus to the direction of its nearby pixels with a large modulus, while the modulus is not changed. ...
... As described in Section 3.2, we use the Edge Tangent Flow vector field [25,26] as the stroke direction guidance. The ablation study of ETF is shown in Figure 6: (a1) is the visualization of the ETF direction, where the short lines denote the local ETF direction; (a2) is the oil painting result using ETF as its stroke direction guidance; (b1) is the visualization of a constant vector field with the direction of 45 • ; (b2) is the oil painting result using 45 • as its stroke direction guidance; (c1) is the visualization of a random vector field; (c2) is the oil painting result using this random vector field as its stroke direction guidance. ...
This paper proposes a novel stroke-based rendering (SBR) method that translates images into vivid oil paintings. Previous SBR techniques usually formulate the oil painting problem as pixel-wise approximation. Different from this technique route, we treat oil painting creation as an adaptive sampling problem. Firstly, we compute a probability density map based on the texture complexity of the input image. Then we use the Voronoi algorithm to sample a set of pixels as the stroke anchors. Next, we search and generate an individual oil stroke at each anchor. Finally, we place all the strokes on the canvas to obtain the oil painting. By adjusting the hyper-parameter maximum sampling probability, we can control the oil painting fineness in a linear manner. Comparison with existing state-of-the-art oil painting techniques shows that our results have higher fidelity and more realistic textures. A user opinion test demonstrates that people behave more preference toward our oil paintings than the results of other methods. More interesting results and the code are in https://github.com/TZYSJTU/Im2Oil.
... Cartoons have unique visual features characterized by clear edges and smooth color shading in non-edge areas. The problem of reproducing cartoon-like effect on real photos is widely explored in early time from the perspective of image abstraction [1], [2], [3], [4]. These methods model cartoon styles with established image processing techniques, which could be summarized into three aspects: (i) image smoothing; (ii) color quantization; (iii) edges enhancement. ...
... Some qualitative results of our model trained over different cartoon datasets and evaluated over high-resolution input pictures are shown in Fig. 5, our method reproduces vivid cartoon effects on high-resolution real-world-scene photos. For qualitative method comparison, we divide related methods into general image stylization or abstraction methods including neural style transfer (NST) [8], flow-based image abstraction (FBIA) [4], and CycleGAN [38], as well as advanced image cartoonization methods including CartoonGAN [6], AnimeGAN [7], and WhiteBox [9]. Results of these models trained over "The Wind Rises" dataset are shown in Fig. 6 and Fig. 7. NST globally and randomly transfers low-level texture features, the produced results suffer from unappealing artifacts and structure distortions. ...
Image cartoonization is recently dominated by generative adversarial networks (GANs) from the perspective of unsupervised image-to-image translation, in which an inherent challenge is to precisely capture and sufficiently transfer characteristic cartoon styles (e.g., clear edges, smooth color shading, abstract fine structures, etc.). Existing advanced models try to enhance cartoonization effect by learning to promote edges adversarially, introducing style transfer loss, or learning to align style from multiple representation space. This paper demonstrates that more distinct and vivid cartoonization effect could be easily achieved with only basic adversarial loss. Observing that cartoon style is more evident in cartoon-texture-salient local image regions, we build a region-level adversarial learning branch in parallel with the normal image-level one, which constrains adversarial learning on cartoon-texture-salient local patches for better perceiving and transferring cartoon texture features. To this end, a novel cartoon-texture-saliency-sampler (CTSS) module is proposed to dynamically sample cartoon-texture-salient patches from training data. With extensive experiments, we demonstrate that texture saliency adaptive attention in adversarial learning, as a missing ingredient of related methods in image cartoonization, is of significant importance in facilitating and enhancing image cartoon stylization, especially for high-resolution input pictures.
... Cartoons have unique visual features characterized by clear edges and smooth color shading in non-edge areas. The problem of reproducing cartoon-like effect on real photos is widely explored in early time from the perspective of image abstraction [1], [2], [3], [4]. These methods model cartoon styles with established image processing techniques, which could be summarized into three aspects: (i) image smoothing; ...
... Some qualitative results of our model trained over different cartoon datasets and evaluated over high-resolution input pictures are shown in Fig. 5, our method reproduces vivid cartoon effects on high-resolution real-world-scene photos. For qualitative method comparison, we divide related methods into general image stylization or abstraction methods including neural style transfer (NST) [8], flow-based image abstraction (FBIA) [4], and CycleGAN [29], as well as advanced image cartoonization methods including CartoonGAN [6], AnimeGAN [7], and WhiteBox [9]. Results of these models trained over "The Wind Rises" dataset are shown in Fig. 6 and Fig. 7. NST globally and randomly transfers low-level texture features, the produced results suffer from unappealing artifacts and structure distortions. ...
Image cartoonization is recently dominated by generative adversarial networks (GANs) from the perspective of unsupervised image-to-image translation , in which an inherent challenge is to precisely capture and sufficiently transfer characteristic cartoon styles (e.g., clear edges, smooth color shading, abstract fine structures, etc.). Existing advanced models try to enhance cartoonization effect by learning to promote edges adversarially, introducing style transfer loss, or learning to align style from multiple representation space. This paper demonstrates that more distinct and vivid cartoonization effect could be easily achieved with only basic adversarial loss. Observing that cartoon style is more evident in cartoon-texture-salient local image regions, we build a region-level adversarial learning branch in parallel with the normal image-level one, which constrains ad-versarial learning on cartoon-texture-salient local patches for better perceiving and transferring cartoon texture features. To this end, a novel cartoon-texture-saliency-sampler (CTSS) module is proposed to dynamically sample cartoon-texture-salient patches from training data. With extensive experiments, we demonstrate that texture saliency adaptive attention in adversarial learning, as a missing ingredient of related methods in image cartoonization, is of significant importance in facilitating and enhancing image cartoon stylization, especially for high-resolution input pictures.
... There are some methods for automatically generating a line drawing, such as flow-based Difference of Gaussian filter [13,33,37], likelihood-function estimation [30,41], datadriven method [2,26,40] and so on. However, these traditional methods did not consider the style of line drawings. ...
... It is obvious that the width in the area A changes significantly. The sharpness w end is defined as: (13) where Len end is the length from the point to the end of the vector curve, end th is the threshold at which the line begins to sharpen, when the length from the point to the end of the vector curve is less than this value, the line begins to sharpen. λ indicates the rate the sharpness changes, the larger the λ, the faster the sharpness changes. ...
Line drawing is a means of superior visual communication, which is made up of lines. Artists usually show their unique styles while creating a line drawing. However, traditional methods can not effectively simulate this free-hand style of artists. Though the data-driven method can generate abundant styles, it is complex and time-consuming. To this end, a new style enhanced line drawing method was proposed. First, lines in images were extracted based on edge detection and edge tracking methods. Then, the global drawing features were simulated, including length measurement, overlap measurement and offset measurement. Finally, the local drawing features that contain width measurement, sharpness measurement and depth measurement were simulated. The results showed that our method can generate stylized line drawings that are more similar to free-hand drawings of artists than the state-of-the-art methods.
... This seminal work achieved stylization by creating painterly images from a collection of brush strokes that were computed using local attributes of an image. This work was extended in the research community by many others ( [5][6][7][8][9]). Automated stylization enabled practical application of stylization to video, for the first time used in the movie What Dreams May Come [10]. ...
... Kyprianidis and Döllner [7] used oriented separable filters and XDoG to achieve a high level of image abstraction. Kang et al. [8] further improved the level of abstraction by adding a flow-based step. Other more complex algorithms simplify images using advanced multi-scale detail image decomposition [16]. ...
The authors present a framework for interactive design of new image stylisations using a wide range of predefined filter blocks. Both novel and off‐the‐shelf image filtering and rendering techniques are extended and combined to allow the user to unleash their creativity to intuitively invent, modify, and tune new styles from a given set of filters. In parallel to this manual design, they propose a novel procedural approach that automatically assembles sequences of filters, leading to unique and novel styles. An important aim of the authors’ framework is to allow for interactive exploration and design, as well as to enable videos and camera streams to be stylised on the fly. In order to achieve this real‐time performance, they use the Best Linear Adaptive Enhancement (BLADE) framework – an interpretable shallow machine learning method that simulates complex filter blocks in real time. Their representative results include over a dozen styles designed using their interactive tool, a set of styles created procedurally, and new filters trained with their BLADE approach.
... This seminal work achieved stylization by creating painterly images from a collection of brush strokes that were computed using local attributes of an image. This work was extended in the research community by many others ( [5][6][7][8][9]). Automated stylization enabled practical application of stylization to video, for the first time used in the movie What Dreams May Come [10]. ...
... Kyprianidis and Döllner [7] used oriented separable filters and XDoG to achieve a high level of image abstraction. Kang et al. [8] further improved the level of abstraction by adding a flow-based step. Other more complex algorithms simplify images using advanced multi-scale detail image decomposition [16]. ...
We present a framework for interactive design of new image stylizations using a wide range of predefined filter blocks. Both novel and off-the-shelf image filtering and rendering techniques are extended and combined to allow the user to unleash their creativity to intuitively invent, modify, and tune new styles from a given set of filters. In parallel to this manual design, we propose a novel procedural approach that automatically assembles sequences of filters, leading to unique and novel styles. An important aim of our framework is to allow for interactive exploration and design, as well as to enable videos and camera streams to be stylized on the fly. In order to achieve this real-time performance, we use the \textit{Best Linear Adaptive Enhancement} (BLADE) framework -- an interpretable shallow machine learning method that simulates complex filter blocks in real time. Our representative results include over a dozen styles designed using our interactive tool, a set of styles created procedurally, and new filters trained with our BLADE approach.
... This process involves the integration of various technologies, such as machine learning combined with other techniques, deep learning, CNN, generative models, image recognition, image transfer, and more. Examples of these techniques include conditional GANs [108], flow-based models [82], diffusion models [69], Transformer [153], variational auto-encoders (VAE) [88], and others. As a graph neural network (GNN) framework, Graph2Plan [72] aims to automatically generate floor plans under user guidance. ...
Landscape design is a complex process that requires designers to engage in intricate planning, analysis, and decision-making. This process involves the integration and reconstruction of science, art, and technology. Traditional landscape design methods often rely on the designer's personal experience and subjective aesthetics, with design standards rooted in subjective perception. As a result, they lack scientific and objective evaluation criteria and systematic design processes. Data-driven artificial intelligence (AI) technology provides an objective and rational design process. With the rapid development of different AI technologies, AI-generated content (AIGC) has permeated various aspects of landscape design at an unprecedented speed, serving as an innovative design tool. This article aims to explore the applications and opportunities of AIGC in landscape design. AIGC can support landscape design in areas such as site research and analysis, design concepts and scheme generation, parametric design optimization, plant selection and visual simulation, construction management, and process optimization. However, AIGC also faces challenges in landscape design, including data quality and reliability, design expertise and judgment, technical challenges and limitations, site characteristics and sustainability, user needs and participation, the balance between technology and creativity, ethics, and social impact. Finally, this article provides a detailed outlook on the future development trends and prospects of AIGC in landscape design. Through in-depth research and exploration in this review, readers can gain a better understanding of the relevant applications, potential opportunities, and key challenges of AIGC in landscape design.
... Texture filtering is a hot topic in image processing, trying to smooth out textural details to more effectively present the structures in images, by which many subsequent image processing tasks can be improved, like saliency detection [PKPH12] and image abstraction [KLC09]. For texture filtering, two fundamental procedures are required, one for determining whether pixels are inside textured regions, called texture measurement, and the other for smoothing the gradients of pixels in textured regions while maintaining the gradients of pixels on structural edges, called the smoothing procedure. ...
It is still challenging with existing methods to distinguish structures from texture details, and so preventing texture filtering. Considering that the textures on both sides of a structural edge always differ much from each other in appearances, we determine whether a pixel is on a structure edge by exploiting the appearance contrast between patches around the pixel, and further propose an efficient implementation method. We demonstrate that our proposed method is more effective than existing methods to distinguish structures from texture details, and our required patches for texture measurement can be smaller than the used patches in existing methods by at least half. Thus, we can improve texture filtering on both quality and efficiency, as shown by the experimental results, e.g., we can handle the textured images with a resolution of 800 × 600 pixels in real‐time. (The code is available at https://github.com/hefengxiyulu/MLPC)
... Third, when reducing noise, BLF has been shown to preserve the sharp edges of the image. The BLF has been used for a variety of tasks, including enhancement of image [186], artistic rendering [187], editing of image [188], feature recognition [189], optical flow estimation [190,191], medical image denoising [192,193], and retinal layer segmentation of 3-D OCT [194]. ...
Automated retinal image analysis holds prime significance in the accurate diagnosis of various critical eye diseases that include diabetic retinopathy (DR), age-related macular degeneration (AMD), atherosclerosis, and glaucoma. Manual diagnosis of retinal diseases by ophthalmologists takes time, effort, and financial resources, and is prone to error, in comparison to computer-aided diagnosis systems. In this context, robust classification and segmentation of retinal images are primary operations that aid clinicians in the early screening of patients to ensure the prevention and/or treatment of these diseases. This paper conducts an extensive review of the state-of-the-art methods for the detection and segmentation of retinal image features. Existing notable techniques for the detection of retinal features are categorized into essential groups and compared in depth. Additionally, a summary of quantifiable performance measures for various important stages of retinal image analysis, such as image acquisition and preprocessing, is provided. Finally, the widely used in the literature datasets for analyzing retinal images are described and their significance is emphasized.
... Basically the development of curiosity has been done by many previous researchers including using inquiry-based learning approaches such as problem-based learning [20], building four models of logical associations between inquiry questions as a framework for open inquiry plans on subjects [21], the ability to discuss things with others will strengthen the curiosity of individuals [22], openness in thinking students will foster curiosity [23] and the state of flow in conducting activities can lead to curiosity. [24] [25]. ...
In this study we described the curiosity of education faculty students in learning based on aspects of interest, noveltyseeking, the openness of experience and exploration. We used cross-sectional survey with a quantitative approach to collect data and to measure curiosity in learning we used questionnaire. The participants were 286 spread across 9 study programs in the faculty of education. The results showed that students sometimes have a curiosity in learning. This condition explained the curiosity of students in learning tends to be in the medium category and tends to be low because the number of students who are rare and never curiosity in learning more than students who often and always curiosity in learning.
... We show two simple examples: pen and color-pencil drawing effects in Fig. 12. One can also combine our semi-sparse smoothing filter model with more complex configurations as illustrated in [21], [30] to produce more reasonable and aesthetic stylized results. (a) Bilateral Filter (BF) [50] (b) Guided Filter (GF) [19] (c) WLS [14] (d) L0 Minimization [57] (e) Relative Total Variation (RTV) [58] (f) Our Result ...
In this paper, we propose an interesting semi-sparsity smoothing algorithm based on a novel sparsity-inducing optimization framework. This method is derived from the multiple observations, that is, semi-sparsity prior knowledge is more universally applicable, especially in areas where sparsity is not fully admitted, such as polynomial-smoothing surfaces. We illustrate that this semi-sparsity can be identified into a generalized -norm minimization in higher-order gradient domains, thereby giving rise to a new ``feature-aware'' filtering method with a powerful simultaneous-fitting ability in both sparse features (singularities and sharpening edges) and non-sparse regions (polynomial-smoothing surfaces). Notice that a direct solver is always unavailable due to the non-convexity and combinatorial nature of -norm minimization. Instead, we solve the model based on an efficient half-quadratic splitting minimization with fast Fourier transforms (FFTs) for acceleration. We finally demonstrate its versatility and many benefits to a series of signal/image processing and computer vision applications.
... In the final stage of the stylization framework, ETF is applied over quantized output. ETF is basically a vector field [87,88] which protects the significant tangent features, thereby driving the weak features to follow the nearest neighboring dominant features, enhance the sharp corner, image direction and minimize the spin artifact impressionism. This work adopted ETF proposed by Kang [87] which is mathematically given by: ...
idp.springer.com
Structure-preserving NPR framework for image abstraction and stylization
MP Pavan Kumar, B Poornima, HS Nagendraswamy, C Manjunath
The Journal of Supercomputing, 1-69, 2021
This work presents a structure-preserving non-photorealistic rendering (NPR) framework that can produce an effective structure-preserving abstracted and stylized output by manipulating visual features from 2D color image. The proposed framework distills the prominent structural features, dominant edges, medium-scale details, curved discontinued edges, silhouette, dendritic structures and curved boundaries and suppresses the superfluous details like noise, texture, irregular gradients, small-scale details and block artifacts. This framework effectively enhanced the significant image properties such as color, contrast, edge strength and sharpness at every stage based on the obtained statistical features availability information and the predefined conditions. This leads to enhancement of quality assessment features such as PSNR, SSIM and suppressing the image complexity and noise. It considers image and object space information to produce abstraction and stylization, thereby identifying emphasized elements of the structure using Harris feature detector algorithm. The proposed framework effectively preserves the structural features in the foreground of an image by comprehensively integrating the sequence of NPR image filters through rigorous experimental analysis simultaneously diminishing the background content of an image. Implementation of the proposed work is carried out in MATLAB 2018 with high-performance computer of 6.6 teraflop/s computing environment and Nvidia Tesla P100 GPU. The proposed framework evaluates every stage output with various subjective matters and quality assessment techniques with various statistical essences. By this manner, contextual features in an image have been identified and well preserved. Effectiveness of the proposed work has been validated by conducting the experiments taking David Mould dataset and Flickr images as references and comparing the obtained results with similar contemporary work cited in the literature. In addition, user’s visual feedback and the standard quality assessment techniques were also used to evaluate the work. Finally, this work lists out the structures preserving applications, constraints, framework implementation challenges and future work in the fields of image abstraction and stylization.
... It is an automatic non-photo-realistic image processing technique that creates simplified stylistic illustrations from color images, videos, and 3D renderings [26]. Most previous approaches [20,21,26,46] utilize the Winnemöller et al.'s [45] image abstract framework, where input image contrast is iteratively exaggerated using a filter and highlighted edges are also added to increase local contrast. The pencil drawing is also one of the image decomposition applications by separating the input image into structure and texture. ...
Image decomposition, which separates a given input image into structure and texture images, has been used for various applications in the fields of computer graphics and image processing. Most previous image decomposition algorithms and applications start with high-quality images, but it requires numerous steps to separate and edit the high-resolution structure and texture images from the low-resolution input counterparts. This paper proposes a simple but effective end-to-end deep neural image decomposition network, which is called ”scalable image decomposition”, by decomposing and upscaling structure and texture images from the degraded input image at the same time. We train the deep neural network to automatically estimate high-resolution structure and texture from the low-resolution input image by designing a shared feature extractor, and structure and texture upscaling networks which have a powerful capability to establish and distinguish a complex mapping between the low-resolution input image and high-quality structure and texture, while preserving more contextual information without any prior information regarding the low-resolution input image. Quantitative and qualitative analyses of the proposed scalable image decomposition network validate that the proposed method is stable and robust against blurring and staircase effects by separating texture and structure upscaling networks in real-time. The predicted upscaled structure and texture images can be used to a variety of applications, such as image abstraction, detail enhancement, and pencil sketching.
... Thirdly, the fact that BLF can maintain narrow edges while removing noise in the image has been validated. BLF has been applied to a wide range of tasks, including Image Enhancement [44], Artistic Rendering [45], Image Editing [46], Optical Flow Estimate [47,48], Feature Recognition [49], Medical Image Denoising [50,51], and 3D Optical Coherence Tomography Retinal Layer Segmentation [52]. ...
Automated detection of vision threatening eye disease based on high resolution retinal fundus images requires accurate segmentation of the blood vessels. In this regard, detection and segmentation of finer vessels, which are obscured by a considerable degree of noise and poor illumination, is particularly challenging. These noises include (systematic) additive noise and multiplicative (speckle) noise, which arise due to various practical limitations of the fundus imaging systems. To address this inherent issue, we present an efficient unsupervised vessel segmentation strategy as a step towards accurate classification of eye diseases from the noisy fundus images. To that end, an ensemble block matching 3D (BM3D) speckle filter is proposed for removal of unwanted noise leading to improved detection. The BM3D-speckle filter, despite its ability to recover finer details (i.e., vessels in fundus images), yields a pattern of checkerboard artifacts in the aftermath of multiplicative (speckle) noise removal. These artifacts are generally ignored in the case of satellite images; however, in the case of fundus images, these artifacts have a degenerating effect on the segmentation or detection of fine vessels. To counter that, an ensemble of BM3D-speckle filter is proposed to suppress these artifacts while further sharpening the recovered vessels. This is subsequently used to devise an improved unsupervised segmentation strategy that can detect fine vessels even in the presence of dominant noise and yields an overall much improved accuracy. Testing was carried out on three publicly available databases namely Structured Analysis of the Retina (STARE), Digital Retinal Images for Vessel Extraction (DRIVE) and CHASE_DB1. We have achieved a sensitivity of 82.88, 81.41 and 82.03 on DRIVE, SATARE, and CHASE_DB1, respectively. The accuracy is also boosted to 95.41, 95.70 and 95.61 on DRIVE, SATARE, and CHASE_DB1, respectively. The performance of the proposed methods on images with pathologies was observed to be more convincing than the performance of similar state-of-the-art methods.
... Contour Generation In order to make it ready to be drawn by the robotic arm, the generated avatar is processed to generate contours with the Flow-based Difference-of-Gaussian (FDoG) filtering (Kang, Lee, and Chui 2009). During this process, the Edge Tangent Flow (ETF) is first constructed from the input image; then a DoG filter is applied to the local edge flow in the perpendicular directions. ...
Robotic drawing has become increasingly popular as an entertainment and interactive tool. In this paper we present RoboCoDraw, a real-time collaborative robot-based drawing system that draws stylized human face sketches interactively in front of human users, by using the Generative Adversarial Network (GAN)-based style transfer and a Random-Key Genetic Algorithm (RKGA)-based path optimization. The proposed RoboCoDraw system takes a real human face image as input, converts it to a stylized avatar, then draws it with a robotic arm. A core component in this system is the AvatarGAN proposed by us, which generates a cartoon avatar face image from a real human face. AvatarGAN is trained with unpaired face and avatar images only and can generate avatar images of much better likeness with human face images in comparison with the vanilla CycleGAN. After the avatar image is generated, it is fed to a line extraction algorithm and converted to sketches. An RKGA-based path optimization algorithm is applied to find a time-efficient robotic drawing path to be executed by the robotic arm. We demonstrate the capability of RoboCoDraw on various face images using a lightweight, safe collaborative robot UR5.
... The direction of the brush is determined by the angleWeight parameter, and the direction of the input image is computed locally by the edge tangent flow (ETF) [41]. The dot product is obtained between the gradient vector of the ETF and the direction vector of the brush, and the brush having the closest angular value is selected. ...
When painters draw, they do not draw what they see; they reflect their emotions through specific forms of artistic expression. Through this empathy, humans derive aesthetic satisfaction. Psychological studies on art and emotions have been extensively conducted since the 20th century, and it has been found that artistic activities such as painting and writing have a positive effect on human emotions. However, it is difficult for a non-professional painter to express human emotions in a picture using artistic elements. The aim of this study was to propose a “user emotion-inspired painterly rendering process” to help non-professionals reflect their emotions through painting. To accomplish this aim, a painter-rendering system that automatically creates digital paintings with parameterized painting features was used. We analyzed the correlation between human emotions and the rendering parameters affecting the painting style through a user study, and we estimated the rendering parameter values for specific emotions. The results showed that colors and emotions were closely linked, and color scheme was an influential factor in the paintings. Furthermore, we developed a recoloring algorithm based on Kobayashi’s three color systems classified as human emotions, and we created paintings that better reflect specific emotions. The proposed algorithm was validated through a user satisfaction survey. Hence, the proposed system can improve a user’s understanding of painting and create paintings that reflect specific emotions.
... Image abstraction consists of the clustering of local pixels with similar properties into perceptually homogeneous regions [31]. It is often pursued for an aesthetic effect while another goal is data reduction [17]. Edge-preserving smoothing filters are an important part of image abstraction where they are used to extract sparse salient structures with the bilateral filter being the most popular [3,10,20]. ...
Edge-preserving smoothing filters have been shown to improve generalisation performance on the HOG features with a SVM classifier. However, not all smoothing filters and parameters lead to better performance. The effects of smoothing filters are studied on the Faster R-CNN detector using generic object and human detection datasets, namely the PASCAL VOC and KITTI respectively. The total variation (TV) smoothing filter was used for this study. It was found that the TV smoothing removed details the CNN was using for detection which degraded performance for both datasets. The results are consistent with previous observations that CNNs tend to learn weak visual features. The performance loss, however, was moderate and could be justified in the context of improving robustness to perturbations. The PASCAL VOC and KITTI datasets had comparable performance loss despite the latter having many more small objects that tend to blend into the background when smoothing is applied.
... We retain it and refer to it as BILATERAL. Most relevant applications apply the bilateral filter in a multi-stage process for realtime rendering with a cartoon look [WOG06], and enhance it by flow-based implementations adapted to the local image structure [KD08,KLC09], in particular to reduce smoothing across falsely detected edges. Providing smooth outputs at curved boundaries of delicate structures, we thus consider the flow-based variant [KD08], and name it FLOWABS. ...
We present the first empirical study on using colour manipulation and stylization to make surgery images/videos more palatable. While aversion to such material is natural, it limits many people's ability to satisfy their curiosity, educate themselves and make informed decisions. We selected a diverse set of image processing techniques to test them both on surgeons and lay people. While colour manipulation techniques and many artistic methods were found unusable by surgeons, edge‐preserving image smoothing yielded good results both for preserving information (as judged by surgeons) and reducing repulsiveness (as judged by lay people). We then conducted a second set of interview with surgeons to assess whether these methods could also be used on videos and derive good default parameters for information preservation. We provide extensive supplemental material at osf.io/4pfes/.
... SAICSIT '19, local pixels with similar properties into perceptually homogeneous regions [33]. Often pursued for an aesthetic effect while another goal is data reduction [18]. ...
Edge-preserving smoothing filters have had many applications in the image processing community, such as image compression, restoration, deblurring and abstraction. However, their potential application in computer vision and machine learning has never been fully studied. The most successful feature descriptors for image classification use gradient images for extracting the overall shapes of objects, thus edge preserving filters could improve their quality. The effects of various edge-preserving filters were evaluated as a pre-processing step inhuman detection. In this work, three smoothing filters were tested, namely the total variation (TV), relative total variation (RTV) and L0 smoothing. Significant performance gains were realised with TV and RTV for both colour and thermal images while the L0 smoothing filter only realised a slight improvement on thermal images and poorer performance on colour images. These results show that smoothing filters have a potential to improve the robustness of common statistical learning classifiers.
Real-world paintings are made, by artists, using brush strokes as the rendering primitive to depict semantic content. The bulk of the Neural Style Transfer (NST) is known transferring style using texture patches, not strokes. The output looks like the content image, but some are traced over using the style texture: it does not look painterly. We adopt a very different approach that uses strokes. Our contribution is to analyse paintings to learn
stroke families
—that is, distributions of strokes based on their shape (a dot, straight lines, curved arcs,
etc.
). When synthesising a new output, these distributions are sampled to ensure the output is painted with the correct style of stroke. Consequently, our output looks more “painterly” than NST output based on texture. Furthermore, where strokes are placed is an important contributing factor in determining output quality, and we have also addressed this aspect. Humans place strokes to emphasize salient semantically meaningful image content. Conventional NST uses a content loss premised on filter responses that is agnostic to salience. We show that replacing that loss with one based on the language-image model benefits the output through greater emphasis of salient content.
Image abstraction is increasingly used in the field of computer graphics and computer vision. Existing methods divide the input images into disjoint regions and generate the abstract images with a reduced color palette. However, the generated images are not satisfactory due to the presence of blurred and unsmooth boundaries, etc. To this end, a pixelated image abstraction method via power diagrams is introduced in this paper to generate high-quality abstract images. Firstly, a pre-processing step is used to capture the significant information in the input image, which is divided into core and non-core parts according to the importance of each pixel. Then, a density map is calculated on basis of the division and grayscale value of each pixel. Furthermore, the primal image is partitioned into polygonal subregions with equal area constraint based on power diagrams. Based on the average RGB color of the grids centered at its centroid, the pixels in each polygonal subregion are colored to produce the output abstract image. Experimental results demonstrate that the proposed method could generate high-quality images with abstraction controllable and smooth boundaries.KeywordsImage abstractionSegmentationPixelatedPower diagram
Image-based artistic rendering can synthesize a variety of expressive styles using algorithmic image filtering. In contrast to deep learning-based methods, these heuristics-based filtering techniques can operate on high-resolution images, are interpretable, and can be parameterized according to various design aspects. However, adapting or extending these techniques to produce new styles is often a tedious and error-prone task that requires expert knowledge. We propose a new paradigm to alleviate this problem: implementing algorithmic image filtering techniques as differentiable operations that can learn parametrizations aligned to certain reference styles. To this end, we present WISE, an example-based image-processing system that can handle a multitude of stylization techniques, such as watercolor, oil or cartoon stylization, within a common framework. By training parameter prediction networks for global and local filter parameterizations, we can simultaneously adapt effects to reference styles and image content, e.g., to enhance facial features. Our method can be optimized in a style-transfer framework or learned in a generative-adversarial setting for image-to-image translation. We demonstrate that jointly training an XDoG filter and a CNN for postprocessing can achieve comparable results to a state-of-the-art GAN-based method. https://github.com/winfried-loetzsch/wise.
In this paper, we present an end-to-end neural solution to model portrait bas-relief from a single photograph, which is cast as a problem of image-to-depth translation. The main challenge is the lack of bas-relief data for network training. To solve this problem, we propose a semi-automatic pipeline to synthesize bas-relief samples. The main idea is to first construct normal maps from photos, and then generate bas-relief samples by reconstructing pixel-wise depths. In total, our synthetic dataset contains 23k pixel-wise photo/bas-relief pairs. Since the process of bas-relief synthesis requires a certain amount of user interactions, we propose end-to-end solutions with various network architectures, and train them on the synthetic data. We select the one that gave the best results through qualitative and quantitative comparisons. Experiments on numerous portrait photos, comparisons with state-of-the-art methods and evaluations by artists have proven the effectiveness and efficiency of the selected network.
Depth quality of stereoscopic three-dimensional (S3D) videos is a significant factor which directly affects the quality of experience (QoE) associated with 3D video applications and services. Nevertheless, there remain limited reports on the investigation of depth perception and depth quality evaluation of S3D videos, which impedes further advancement and deployment of 3D video technology. This paper reports a series of subjective experiments which have been conducted to investigate the depth perception and its related properties of the human visual system (HVS) using S3D video compressed by the H.264/AVC standard. The experimental results reveal that the HVS response in depth perception varies at different frequencies and in varying orientations, and the distortions introduced by video coding can cause the loss of and/or variation in depth perception. By integration of binocular and monocular features (BM) extracted from left and right views of S3D video with respect to depth perception, a depth quality assessment model, herein referred to as BM-DQAM, is devised by training these stereoscopic and spatial orientation structural features with a support vector regression model. It is shown that the BM-DQAM provides a novel no-reference metric for evaluation of the depth quality of S3D videos. Based on two publicly available 3D video databases and the proposed depth perception assessment database, the experimental results show that the BM-DQAM has demonstrated better performance in assessing the depth quality in S3D video viewing than that of other metrics reported in the published literatures, correlating well with the HVS response in the depth perception assessment experiment.
Line drawing is a means of superior visual communication which can effectively convey useful information to viewers. Artists usually draw what they see rather than what exists, which means the most attractive object is emphasized while the rest are inhibited. Moreover, artists draw the whole object with coherent lines instead of fractured lines, which also contribute to the outstanding visual effect. From these perspectives, generating line drawings with saliency and coherence remains a great challenge. Existing line drawing generation methods ignore these important properties and cannot generate coherent lines in some cases since they do not take into account how artists draw a picture. To this end, a novel saliency-aware line drawing method was proposed to better grasp the viewer’s attention on an image. First, a saliency enhanced line extraction method combining saliency map and edge tangent flow was proposed to ensure the saliency and coherence of lines, especially in salient but low contrast areas. Then, the salient information was highlighted while irrelevant details were eliminated by inhibiting lines in less salient areas to enhance the saliency of the line drawing. Finally, the transparency of lines was adjusted to further emphasize important information. Various results showed that our method can generate line drawings with better visual saliency and coherence than the state-of-the-art methods.
This work extricates the image characteristic features for the classification of archeological monument images. At the pre-processing stage, archeological dataset sample images are treated by using structure safeguarding image abstraction framework, which can deliver the most effective image abstraction output by manipulating the perceptible features in the given low-illuminated and underexposed color image samples. Proposed abstraction-framework effectively boosted the significant image property features like color, edge, sharpness, contrast and suppresses complexity and noise. The image properties were also refined at each phase based on the attained statistical feature disposal information. The work adopted the Harris feature identification technique to identify the most significant image features in the input and enhanced images. The framework also preserves significant features in the foreground of an image by intelligently integrating the series of filters during rigorous experimental work and also diminishes the background content of an input image. The proposed archeological system evaluates every stage of the result with assorted subjective matters and calculates the image quality and properties assessment statistical attributes. By this way prominent features in an image have been recognized. The efficiency of this work has been corroborated by performing the trials on the selected archeological dataset. In addition, user’s visual feedback and the standard image quality assessment techniques were also used to evaluate the proposed pre-processing framework. Based on the obtained abstraction images from the framework, this work extracts the image gray color texture features using GLCM, color texture from CTMs and deep-learning features from AlexNet for the classification of archeological monument classification. This work adopted a support vector machine as a classifier. To corroborate the efficiency of the proposed method, an experiment was conducted on our own data set of Chalukya, Kadamba, Hoysala and new engraving monuments, each domain consisting of 500 archeological data set samples with large intra-class variation, with different environmental lighting condition, low-illumination and different pose. Implementation of this work was carried out in MATLAB-2020 with HPC Nvidia Tesla P100 GPU, and obtained results show that combination of multiple features significantly improves the performance to the extent of 98.10%.
Thangka is a religious scroll painting. As a window to Tibetan traditional culture and religious customs, the segmentation of important semantic objects in the images can help the public understand the image content and realize high-level semantic cognition of the visual content. This paper proposes a semantic segmentation network for the central figures’ headdresses in the portrait-type Thangka images based on the rich structure and standard composition of Thangka images. Meanwhile, it has also constructed the first pixel-level semantic annotation data set of the central figures’ headdresses in Thangka images. Firstly, an original Thangka image witnesses edge augmentation through the line drawing augmentation module. Then the Feature Extraction Network is used to extract the feature map. After that, it uses the Half-RPN network to get the region of interest (ROI). Finally, Mask R-CNN Head completes segmentation and class prediction. The experimental results show that the performance of the proposed model is 10% to 19% better than that of the state-of-the-art models such as Mask R-CNN and DeepLab V3.
While modern digital entertainment has seen a major shift toward photorealism in animation, there is still significant demand for stylized rendering tools. Stylized, or non-photorealistic rendering (NPR), applications generally sacrifice physical accuracy for artistic or functional visual output. Oftentimes, NPR applications focus on extracting specific features from a 3D environment and highlighting them in a unique manner. One application of interest involves recreating 2D hand-drawn art styles in a 3D-modeled environment. This task poses challenges in the form of spatial coherence, feature extraction, and stroke line rendering. Previous research on this topic has also struggled to overcome specific performance bottlenecks, which have limited use of this technology in real-time applications. Specifically, many stylized rendering techniques have difficulty operating on large-scale scenes, such as open-world terrain environments. In this paper, we describe various novel rendering techniques for mimicking hand-drawn art styles in a large-scale 3D environment, including modifications to existing methods for stroke rendering and hatch-line texturing. Our system focuses on providing various complex styles while maintaining real-time performance, to maximize user-interactability. Our results demonstrate improved performance over existing real-time methods, and offer a few unique style options for users, though the system still suffers from some visual inconsistencies.
Texture plays an important role in cartoon images to represent materials of objects and enrich visual attractiveness. However, manually crafting a cartoon texture is not easy, so amateurs usually directly use cartoon textures downloaded from the Internet. Unfortunately, Internet resources are quite limited and often patented, which restrict the users from generating visually pleasant and personalized cartoon textures. In this paper, we propose a deep learning based method to generate cartoon textures from natural textures. Different from the existing photo cartoonization methods that only aim to generate cartoonic images, the key to our method is to generate cartoon textures that are both cartoonic and regular. To achieve this goal, we propose a regularization module to generate a regular natural texture with similar appearance as the input, and a cartoonization module to cartoffonize the regularized natural texture into a regular cartoon texture. Our method successfully produces cartoonic and regular textures from various natural textures.
Line drawing and screentoning are two distinct areas of study in non-photorealistic rendering, where the former emphasizes object contours, while the latter conveys tone and shading information on object surfaces. As these two problems are concerned with different yet equally important features, either method seldom delivers a complete description of the scene when used alone. Yet, research community has largely treated them as separate problems and thus resulted in two entirely different sets of solutions, complicating both implementation and usage. In this paper, we present a stylistic image binarization method called hybrid difference of Gaussians (HDoG) that performs both line drawing and screentoning in a unified framework. Our method is based upon two different extensions of DoG operator: one for line extraction, and the other for tone description. In particular, we propose an extension called adaptive DoG, that uses luminance as weight to automatically generate screentone that adapts to the local tone. Experimental results demonstrate that our hybrid method effectively generates aesthetically pleasing image binarizations that encompass both line drawing and screentoning, closely resembling professional pen-and-ink illustrations. Also, being based on Gaussian filtering, our method is very fast and also easy to implement.
Cartoon is highly abstracted with clear edges, which makes it unique from the other art forms. In this paper, we focus on the essential cartoon factors of abstraction and edges, aiming to cartoonize real‐world photographs like an artist. To this end, we propose a two‐stage network, each stage explicitly targets at producing abstracted shading and crisp edges respectively. In the first abstraction stage, we propose a novel unsupervised bilateral flattening loss, which allows generating high‐quality smoothing results in a label‐free manner. Together with two other semantic‐aware losses, the abstraction stage imposes different forms of regularization for creating cartoon‐like flattened images. In the second stage we draw lines on the structural edges of the flattened cartoon with the fully supervised line drawing objective and unsupervised edge augmenting loss. We collect a cartoon‐line dataset with line tracing, and it serves as the starting point for preparing abstraction and line drawing data. We have evaluated the proposed method on a large number of photographs, by converting them to three different cartoon styles. Our method substantially outperforms state‐of‐the‐art methods in terms of visual quality quantitatively and qualitatively.
The screen content (SC) pictures, such as webpages, serve as a visible and convenient medium to well-represent the Internet information, and therefore, the visual quality of SC pictures is highly significant and has attained a growing amount of attention. Accurate quality evaluation of SC pictures not only provides the fidelity of the conveyed information, but also contributes to the improvement of the user experience. In practical applications, a reliable estimation of SC pictures plays a considerably critical role for the optimization of the processing systems as the guidance. Based on these motivations, this paper proposes a novel method for precisely assessing the quality of SC pictures using very sparse reference information. Specifically, the proposed quality method separately extracts the macroscopic and microscopic structures, followed by comparing the differences of macroscopic and microscopic features between a pristine SC picture and its corrupted version to infer the overall quality score. By studying the feature histogram for dimensionality reduction, the proposed method merely requires two features as the reference information that can be exactly embedded in the file header with very few bits. Experiments manifest the superiority of our algorithm as compared with state-of-the-art relevant quality metrics when applied to the visual quality evaluation of SC pictures.
In this article, we present a framework which transfers video to dynamic relief‐structures with water interaction. Our method enables the interaction between video content and waterflows. The key idea is that the dynamic 2.5D structures of the objects in video are constructed and then waterflows are simulated to interact with the 2.5D structures. We performed experiments for still images, and video in which objects had various motion styles. User interaction with our system was also demonstrated. The result of a user study indicated that our method could produce visually acceptable results. This article presents a framework which transfers video to dynamic relief‐structures with water interaction. The proposed method enables the interaction between video content and waterflows. The key idea is that the dynamic 2.5D structures of the objects in video are constructed and then waterflows are simulated to interact with the 2.5D structures.
This work proposed a semi-supervised method for automatic texture filtering. Our method leveraged a limited amount of labeled data and a large amount of unlabeled data to train Generative Adversarial Networks (GANs). Separate loss functions were designed for both labeled and unlabeled datasets. Our main contribution is the introduction of knowledge extracted from shallow and deep layers in neural networks. Loss defined within shallow layers preserves the edge, while loss defined within the deep layers identifies the semantic content and conversely removes the small-scale texture variations. This contribution directly addresses the major challenge for texture filtering, distinguishing the structural content from non-structural textures at the pixel level. The extracted information, in our study, improved the content and color consistency before and after the process of filtering, for unlabeled samples in particular. The proposed method offers twofold benefits: first, significant reductions in the amounts of time and effort expended in reconstructing the labeled dataset, especially given the delicate operations required at the pixel level; second, a reduction in over-fitting, in supervised learning with a small amount of labeled data, by utilizing a large amount of unlabeled data. The results confirm that our method can perform comparably with non-learning-based methods, alleviating the demand for the determination of optimal parameter values.
Tangent vector fields are an essential ingredient in controlling surface appearance for applications ranging from anisotropic shading to texture synthesis and non-photorealistic rendering. To achieve a desired effect one is typically interested in smoothly varying fields that satisfy a sparse set of user-provided constraints. Using tools from Discrete Exterior Calculus, we present a simple and efficient algorithm for designing such fields over arbitrary triangle meshes. By representing the field as scalars over mesh edges (i.e., discrete 1-forms), we obtain an intrinsic, coordinate-free formulation in which field smoothness is enforced through discrete Laplace operators. Unlike previous methods, such a formulation leads to a linear system whose sparsity permits efficient pre-factorization. Constraints are incorporated through weighted least squares and can be updated rapidly enough to enable interactive design, as we demonstrate in the context of anisotropic texture synthesis.
A stable algorithm is proposed for image restoration based on the "mean curvature motion" equation. Existence and uniqueness of the "viscosity" solution of the equation are proved, a L(infinity) stable algorithm is given, experimental results are shown, and the subjacent vision model is compared with those introduced recently by several vision researchers. The algorithm presented appears to be the sharpest possible among the multiscale image smoothing methods preserving uniqueness and stability.
Digital photography has made it possible to quickly and easily take a pair of images of low-light environments: one with flash to capture detail and one without flash to capture ambient illumination. We present a variety of applications that analyze and combine the strengths of such flash/no-flash image pairs. Our applications include denoising and detail transfer (to merge the ambient qualities of the no-flash image with the high-frequency flash detail), white-balancing (to change the color tone of the ambient image), continuous flash (to interactively adjust flash intensity), and red-eye removal (to repair artifacts in the flash image). We demonstrate how these applications can synthesize new images that are of higher quality than either of the originals.
This paper describes a computational approach to edge detection. The success of the approach depends on the definition of a comprehensive set of goals for the computation of edge points. These goals must be precise enough to delimit the desired behavior of the detector while making minimal assumptions about the form of the solution. We define detection and localization criteria for a class of edges, and present mathematical forms for these criteria as functionals on the operator impulse response. A third criterion is then added to ensure that the detector has only one response to a single edge. We use the criteria in numerical optimization to derive detectors for several common image features, including step edges. On specializing the analysis to step edges, we find that there is a natural uncertainty principle between detection and localization performance, which are the two main goals. With this principle we derive a single operator shape which is optimal at any scale. The optimal detector has a simple approximate implementation in which edges are marked at maxima in gradient magnitude of a Gaussian-smoothed image. We extend this simple detector using operators of several widths to cope with different signal-to-noise ratios in the image. We present a general method, called feature synthesis, for the fine-to-coarse integration of information from operators at different scales. Finally we show that step edge detector performance improves considerably as the operator point spread function is extended along the edge.
Abstract In this paper, we introduce an NPR system to generate color sketches in a free-hand drawing style. From observation, we found out that in a free-hand style both shape and color of image regions are stylized, which contributes cooperatively to the visual effect of a sketch. With the color regions obtained by our interactive segmentation algorithm, we create a stylized color sketch in two steps. First, in order to manifest the lighting and shading of each region, we shrink the boundary,of the color regions with a luminance-based algorithm emphasizing the highlight parts, which enforces consistency with the free-hand artist style of the sketch. Second, the proposed color shift algorithm is applied to image regions to emphasize the main content of the sketch and acquires a visually pleasing combination,of color. We choose the optimal combination,by minimizing the energy function based on an artist drawn color database, artistic drawing rules and input image colors. The generated results show that our system can produce aesthetically pleasing sketches. Keywords: non-photorealistic rendering, boundary shrinking,
We present techniques for transforming images and videos into painterly animations depicting different artistic styles. Our techniques rely on image and video analysis to compute appearance and motion properties. We also determine and apply motion information from different (user-specified) sources to static and moving images. These properties that encode spatio-temporal variations are then used to render (or paint) effects of selected styles to generate images and videos with a painted look. Painterly animations are generated using a mesh of brush stroke objects with dynamic spatio-temporal properties. Styles govern the behavior of these brush strokes as well as their rendering to a virtual canvas. We present methods for modifying the properties of these brush strokes according to the input images, videos, or motions. Brush stroke color, length, orientation, opacity, and motion are determined and the brush strokes are regenerated to fill the canvas as the video changes. All brush stroke properties are temporally constrained to guarantee temporally coherent non-photorealistic animations.
. We present an interactive system, which allows the user to produce drawings in a variety of traditional styles. It takes as input an image and performs semi-automatic tonal modeling. Our system shifts tedious technical aspects to the computer side, while providing the user with freedom on the creative and aesthetic side. The user has low-level control over stroke placement, and high-level control over the tone, smudging and amount of detail. The drawing is rendered in real-time. The basic component is a thresholding model of strokes that can simulate a large class of styles (e.g. pencil, charcoal, engraving). It provides a controllable simulation of the variation of pencil pressure or stroke thickness traditionally used in tonal modeling. We introduce a novel fast equilibration approach for the resulting thresholding structure. The user can specify smudging and control the amount of detail over each part of the drawing. 1
We present a new technique for the display of high-dynamic-range images, which reduces the contrast while preserving detail. It is based on a two-scale decomposition of the image into a base layer, encoding large-scale variations, and a detail layer. Only the base layer has its contrast reduced, thereby preserving detail. The base layer is obtained using an edge-preserving filter called the bilateral filter. This is a non-linear filter, where the weight of each pixel is computed using a Gaussian in the spatial domain multiplied by an influence function in the intensity domain that decreases the weight of pixels with large intensity differences. We express bilateral filtering in the framework of robust statistics and show how it relates to anisotropic diffusion. We then accelerate bilateral filtering by using a piecewise-linear approximation in the intensity domain and appropriate subsampling. This results in a speed-up of two orders of magnitude. The method is fast and requires no parameter setting.
We present a new technique for the display of high-dynamic-range images, which reduces the contrast while preserving detail. It is based on a two-scale decomposition of the image into a base layer, encoding large-scale variations, and a detail layer. Only the base layer has its contrast reduced, thereby preserving detail. The base layer is obtained using an edge-preserving filter called the bilateral filter. This is a non-linear filter, where the weight of each pixel is computed using a Gaussian in the spatial domain multiplied by an influence function in the intensity domain that decreases the weight of pixels with large intensity differences. We express bilateral filtering in the framework of robust statistics and show how it relates to anisotropic diffusion. We then accelerate bilateral filtering by using a piecewise-linear approximation in the intensity domain and appropriate subsampling. This results in a speed-up of two orders of magnitude. The method is fast and requires no parameter setting.
Image analysis and enhancement tasks such as tone mapping, colorization, stereo depth, and photomontage, often require computing a solution (e.g., for exposure, chromaticity, disparity, labels) over the pixel grid. Computational and memory costs often require that a smaller solution be run over a downsampled image. Although general purpose upsampling methods can be used to interpolate the low resolution solution to the full resolution, these methods generally assume a smoothness prior for the interpolation.
We demonstrate that in cases, such as those above, the available high resolution input image may be leveraged as a prior in the context of a joint bilateral upsampling procedure to produce a better high resolution solution. We show results for each of the applications above and compare them to traditional upsampling methods.
We present an automatic, real-time video and image abstraction framework that abstracts imagery by modifying the contrast of visually important features, namely luminance and color opponency. We reduce contrast in low-contrast regions using an approximation to anisotropic diffusion, and artificially increase contrast in higher contrast regions with difference-of-Gaussian edges. The abstraction step is extensible and allows for artistic or data-driven control. Abstracted images can optionally be stylized using soft color quantization to create cartoon-like effects with good temporal coherence. Our framework design is highly parallel, allowing for a GPU-based, real-time implementation. We evaluate the effectiveness of our abstraction framework with a user-study and find that participants are faster at naming abstracted faces of known persons compared to photographs. Participants are also better at remembering abstracted images of arbitrary scenes in a memory task.
The ultimate goal of augmented reality is to provide the user with a view of the surroundings enriched by virtual objects. Practically all augmented reality systems rely on standard real-time rendering methods for generating the images of virtual scene elements. Although such conventional computer graphics algorithms are fast, they often fail to produce sufficiently realistic renderings. The use of simple lighting and shading methods, as well as the lack of knowledge about actual lighting conditions in the real surroundings, cause virtual objects to appear artificial. In this paper, we propose an entirely novel approach for generating augmented reality images in video see-through systems. Our method is based on the idea of applying stylization techniques for reducing the visual realism of both the camera image and the virtual graphical objects. A special painterly image filter is applied to the camera video stream. The virtual scene elements are generated using a non-photorealistic rendering method. Since both the camera image and the virtual objects are stylized in a corresponding "cartoon-like" or "sketch-like" way, they appear very similar. As a result, the graphical objects seem to be an actual part of the real surroundings. We describe both the new painterly filter for the camera image and the non-photorealistic rendering method for virtual scene elements, which has been adapted for this purpose. Both are fast enough for generating augmented reality images in real-time and are highly customizable. The results obtained using our method are very promising and show that it improves immersion in augmented reality.
This thesis concerns the use of spatial and tonal adaptivity in improving the resolution of aliased image sequences under scene or camera motion. Each of the five content chapters focuses on a different subtopic of super-resolution: image registration (chapter 2), image fusion (chapter 3 and 4), super-resolution restoration (chapter 5), and super-resolution synthesis (chapter 6). Chapter 2 derives the Cramer-Rao lower bound of image registration and shows that iterative gradient-based estimators achieve this performance limit. Chapter 3 presents an algorithm for image fusion of irregularly sampled and uncertain data using robust normalized convolution. The size and shape of the fusion kernel is adapted to local curvilinear structures in the image. Each data sample is assigned an intensity-related certainty value to limit the influence of outliers. Chapter 4 presents two fast implementations of the signal-adaptive bilateral filter. The xy-separable implementation filters the image along sampling axes, while the uv-separable implementation filters the image along gauge coordinates. Chapter 5 presents a robust image restoration method using Gaussian error norm rather than quadratic error norm. The robust solution resembles the maximum-likelihood solution under Gaussian noise, and it is not susceptible to outliers. A series of objective quality measures confirms the superiority of this solution to current super-resolution algorithms in the literature. Chapter 6 proposes a super-resolution synthesis algorithm in the DCT domain. The algorithm requires a high-resolution image of similar content to be used as texture source for the low-resolution input image.
This contribution introduces the basic techniques for digital facial engraving, which imitates traditional copperplate engraving. Inspired by traditional techniques, we first establish a set of basic rules thanks to which separate engraving layers are built on the top of the original photo. Separate layers are merged according to simple merging rules and according to range shift/scale masks specially introduced for this purpose. We illustrate the introduced technique by a set of black/white and color engravings, showing different features such as engraving-specific image enhancements, mixing different regular engraving lines with mezzotint, irregular perturbations of engraving lines etc. We introduce the notion of engraving style which comprises a set of separate engraving layers together with a set of associated range shift/scale masks. The engraving style helps to port the look and feel of one engraving to another. Once different libraries of pre-defined mappable engraving styles and an appropriate user interface are added to the basic system, producing a decent gravure starting from a simple digital photo will be a matter of seconds. The engraving technique described in this contribution opens new perspectives for digital art, adding unprecedented power and precision to the engraver's work.
A theory of edge detection is presented. The analysis proceeds in two parts. (1) Intensity changes, which occur in a natural image over a wide range of scales, are detected separately at different scales. An appropriate filter for this purpose at a given scale is found to be the second derivative of a Gaussian, and it is shown that, provided some simple conditions are satisfied, these primary filters need not be orientation-dependent. Thus, intensity changes at a given scale are best detected by finding the zero values of delta 2G(x,y)*I(x,y) for image I, where G(x,y) is a two-dimensional Gaussian distribution and delta 2 is the Laplacian. The intensity changes thus discovered in each of the channels are then represented by oriented primitives called zero-crossing segments, and evidence is given that this representation is complete. (2) Intensity changes in images arise from surface discontinuities or from reflectance or illumination boundaries, and these all have the property that they are spatially. Because of this, the zero-crossing segments from the different channels are not independent, and rules are deduced for combining them into a description of the image. This description is called the raw primal sketch. The theory explains several basic psychophysical findings, and the operation of forming oriented zero-crossing segments from the output of centre-surround delta 2G filters acting on the image forms the basis for a physiological model of simple cells (see Marr & Ullman 1979).
The contribution of this paper is a novel framework for synthesizing nonphotorealistic animations from real video sequences. We demonstrate that, through automated mid-level analysis of the video sequence as a spatiotemporal volume--a block of frames with time as the third dimension--we are able to generate animations in a wide variety of artistic styles, exhibiting a uniquely high degree of temporal coherence. In addition to rotoscoping, matting, and novel temporal effects unique to our method, we demonstrate the extension of static nonphotorealistic rendering (NPR) styles to video, including painterly, sketchy, and cartoon shading. We demonstrate how this novel coherent shading framework may be combined with our earlier motion emphasis work to produce a comprehensive "Video Paintbox" capable of rendering complete cartoon-styled animations from video clips.
Bilateral filtering is an edge-preserving filtering technique that employs both geometric closeness and photometric similarity of neighboring pixels to construct its filter kernel. Multi-dimensional bilateral filtering is computationally expensive because the adaptive kernel has to be recomputed at every pixel. In this paper, we present a separable implementation of the bilateral filter. The separable implementation offers equivalent adaptive filtering capability at a fraction of execution time compared to the traditional filter. Because of this efficiency, the separable bilateral filter can be used for fast preprocessing of images and videos. Experiments show that better image quality and higher compression efficiency is achievable if the original video is preprocessed with the separable bilateral filter.
Tangent vector fields are an essential ingredient in controlling surface appearance for applications ranging from anisotropic shading to texture synthesis and non-photorealistic rendering. To achieve a desired effect one is typically interested in smoothly varying fields that satisfy a sparse set of user-provided constraints. Using tools from Discrete Exterior Calculus, we present a simple and efficient algorithm for designing such fields over arbitrary triangle meshes. By representing the field as scalars over mesh edges (i.e., discrete 1-forms), we obtain an intrinsic, coordinate-free formulation in which field smoothness is enforced through discrete Laplace operators. Unlike previous methods, such a formulation leads to a linear system whose sparsity permits efficient pre-factorization. Constraints are incorporated through weighted least squares and can be updated rapidly enough to enable interactive design, as we demonstrate in the context of anisotropic texture synthesis.
We present an image-based modeling and editing system that takes a single photo as input. We represent a scene as a layered collection of depth images, where each pixel encodes both color and depth. Starting from an input image, we employ a suite of user-assisted techniques, based on a painting metaphor, to assign depths and extract layers. We introduce two specific editing operations. The first, a "clone brushing tool," permits the distortion-free copying of parts of a picture, by using a parameterization optimization technique. The second, a "texture-illuminance decoupling filter," discounts the effect of illumination on uniformly textured areas, by decoupling large- and small-scale features via bilateral filtering. Our system enables editing from different viewpoints, extracting and grouping of image-based objects, and modifying the shape, color, and illumination of these objects.
We present a method for creating black-and-white illustrations from photographs of human faces. In addition an interactive technique is demonstrated for deforming these black-and-white facial illustrations to create caricatures which highlight and exaggerate representative facial features. We evaluate the effectiveness of the resulting images through psychophysical studies to assess accuracy and speed in both recognition and learning tasks. These studies show that the facial illustrations and caricatures generated using our techniques are as effective as photographs in recognition tasks. For the learning task we find that illustrations are learned two times faster than photographs and caricatures are learned one and a half times faster than photographs. Because our techniques produce images that are effective at communicating complex information, they are useful in a number of potential applications, ranging from entertainment and education to low bandwidth telecommunications and psychology research.
In this work a scale-space framework has been presented which does not require any monotony assumption (comparison principle). We have seen that, besides the fact that many global smoothing scale-space properties are maintained, new possibilities with respect to image restoration appear. Rather than deducing a unique equation from first principles, we have analyzed well-posedness and scale-space properties of a general family of regularized anisotropic diffusion filters. Existence and uniqueness results, continuous dependence of the solution on the initial image, maximum-minimum principles, invariances, Lyapunov functionals, and convergence to a constant steady state have been established. The large class of Lyapunov functionals permits to regard these filters in numerous ways as simplifying, information-reducing transformations. These global smoothing properties do not contradict seemingly opposite local effects such its edge enhancement. For this reason it is possible to design scale-spaces with restoration properties giving segmentation-like results. Prerequisites have been stated under which one can prove well-posedness and scale-space results in the continuous, semidiscrete and discrete setting. Each of these frameworks stands on its own and does not require the others. On the other hand, the prerequisites in all three settings reveal many similarities and, as a consequence, representatives of the semidiscrete class can be obtained by suitable spatial discretizations of the continuous class, while representatives of the discrete class may arise from time discretizations of semidiscrete filters. The degree of freedom within the proposed class of filters can be used to tailor the filters towards specific restoration tasks. Therefore, these scale-spaces do not need to be uncommitted; they give the user the liberty to incorporate a-priori knowledge, for instance concerning size and contrast of especially interesting features. The analyzed class comprises linear diffusion filtering and the nonlinear isotropic model of Catté, Lions, Morel, Coll and Whitaker and Pizer, but also novel approaches have been proposed: The use of diffusion tensors instead of scalar-valued diffusivities puts us in a position to design real anisotropic diffusion processes which may reveal advantages at noisy edges. Last but not least, the fact that these filters are steered by the structure tensor instead of the regularized gradient allows to adapt them to more sophisticated tasks such as the enhancement of coherent flow-like structures. In view of these results, anisotropic diffusion deserves to be regarded as much more than all ad-hoc strategy for transforming a degraded image into a more pleasant looking one. It is a flexible and mathematically sound class of methods which ties the advantages of two worlds: scale-space analysis and image restoration.
This paper describes a technique that transforms ordinary video segments into animations that have a hand-painted look. Our method is the first to exploit temporal coherence in video clips to design an automatic filter with a hand-drawn animation quality, in this case, one that produces an Impressionist effect. Off-the-shelf image processing and rendering techniques are employed, modified and combined in a novel way. This paper proceeds through the process step by step, providing helpful hints for tuning the off-the-shelf parts as well as describing the new techniques and bookkeeping used to glue the parts together.
This paper presents models for graphite pencil, drawing paper, blenders, and kneaded eraser that produce realistic looking pencil marks, textures, and tones. Our models are based on an observation of how lead pencils interact with drawing paper, and on the absorptive and dispersive properties of blenders and erasers interacting with lead material deposited over drawing paper. The models consider parameters such as the particle composition of the lead, the texture of the paper, the position and shape of the pencil materials, and the pressure applied to them. We demonstrate the capabilities of our approach with a variety of images and compare them to digitized pencil drawings. We also present image-based rendering results implementing traditional graphite pencil tone rendering methods.
We are interested in diffusion PDE’s for smoothing multi-valued images in an anisotropic manner. By pointing out the pros
and cons of existing tensor-driven regularization methods, we introduce a new constrained diffusion PDE that regularizes image
data while taking curvatures of image structures into account. Our method has a direct link with a continuous formulation
of the Line Integral Convolutions, allowing us to design a very fast and stable algorithm for its implementation. Besides,
our smoothing scheme numerically performs with a sub-pixel accuracy and is then able to preserves very thin image structures
contrary to classical PDE discretizations based on finite difference approximations. We illustrate our method with different
applications on color images.
This paper presents a comprehensive scheme for automatically generating a broad class of artistic illustrations from photographs.
Using strokes as the major building blocks, our system optimizes the stroke attributes subject to the desired rendering style.
The stroke attributes are computed adaptively to enable importance-based control of the abstraction level at each pixel. We
propose a novel outline detection and refinement paradigm called edge painting to construct an outline map, and from which to derive the pixel-wise importance. We also introduce an adaptive bilateral filter to adaptively guide the curved stroke directions based on the importance map. Given the outline, importance, and direction
maps, the system creates the illustration via selecting the representative colors, setting the style parameters, and optimizing
the stroke attributes based on simulated annealing. The experimental results show that our scheme facilitates automatic production
of artistic illustrations in a wide range of rendering styles.
This paper presents an approach for determining stroke thickness in computer-generated illustrations of smooth surfaces. We assume that dark strokes are drawn to approximate the dark regions of the shaded surface. This assumption leads to a simple formula for thickness of contours and suggestive contours; this formula depends on depth, radial curvature, and light direction in a manner that reproduces aspects of thickness observed in hand-made drawings. These strokes convey local shape and depth relationships, and produce appealing imagery. Our method is simple to implement, provides temporally-coherent strokes, and runs at interactive rates.
This paper presents a non-photorealistic rendering technique that automatically generates a line drawing from a photograph. We aim at extracting a set of coherent, smooth, and stylistic lines that ef- fectively capture and convey important shapes in the image. We rst develop a novel method for constructing a smooth direction eld that preserves the o w of the salient image features. We then introduce the notion of o w-guided anisotropic ltering for detect- ing highly coherent lines while suppressing noise. Our method is simple and easy to implement. A variety of experimental results are presented to show the effectiveness of our method in producing self-contained, high-quality line illustrations.
Recent work has shown that sparse lines defined on 3D shapes, including occluding contours and suggestive contours, are effective at conveying shape. We introduce two new families of lines called suggestive highlights and principal highlights, based on definitions related to suggestive contours and geometric creases. We show that when these are drawn in white on a gray background they are naturally interpreted as highlight lines, and complement contours and suggestive contours. We provide object-space definitions and algorithms for extracting these lines, explore several stylization possibilities, and compare the lines to ridges and valleys of intensity in diffuse-shaded images.
Imaging vector fields has applications in science, art, image pro- cessing and special effects. An effective new approach is to use linear and curvilinear filtering techniques to locally blur textures along a vector field. This approach builds on several previous tex- ture generation and filtering techniques(8, 9, 11, 14, 15, 17, 23). It is, however, unique because it is local, one-dimensional and inde- pendent of any predefined geometry or texture. The technique is general and capable of imaging arbitrary two- and three-dimen- sional vector fields. The local one-dimensional nature of the algo- rithm lends itself to highly parallel and efficient implementations. Furthermore, the curvilinear filter is capable of rendering detail on very intricate vector fields. Combining this technique with other rendering and image processing techniques — like periodic motion filtering — results in richly informative and striking images. The technique can also produce novel special effects.
We present an interactive system for creating pen-and-ink illustrations. The system uses stroke textures—collections of strokes arranged in different patterns—to generate texture and tone. The user “paints” with a desired stroke texture to achieve a desired tone, and the computer draws all of the individual strokes.The system includes support for using scanned or rendered images for reference to provide the user with guides for outline and tone. By following these guides closely, the illustration system can be used for interactive digital halftoning, in which stroke textures are applied to convey details that would otherwise be lost in this black-and-white medium.By removing the burden of placing individual strokes from the user, the illustration system makes it possible to create fine stroke work with a purely mouse-based interface. Thus, this approach holds promise for bringing high-quality black-and-white illustration to the world of personal computing and desktop publishing.
Good information design depends on clarifying the meaningful structure in an image. We describe a computational approach to stylizing and abstracting photographs that explicitly responds to this design goal. Our system transforms images into a line-drawing style using bold edges and large regions of constant color. To do this, it represents images as a hierarchical structure of parts and boundaries computed using state-of-the-art computer vision. Our system identifies the meaningful elements of this structure using a model of human perception and a record of a user's eye movements in looking at the photo; the system renders a new image using transformations that preserve and highlight these visual elements. Our method thus represents a new alternative for non-photorealistic rendering both in its visual style, in its approach to visual form, and in its techniques for interaction.
We are interested in regularizing fields of orthonormal vector sets, using constraint-preserving anisotropic diffusion PDE's. Each point of such a field is defined by multiple orthogonal and unitary vectors and can indeed represent a lot of interesting orientation features such as direction vectors or orthogonal matrices (among other examples). We first develop a general variational framework that solves this regularization problem, thanks to a constrained minimization of -functionals. This leads to a set of coupled vector-valued PDE's preserving the orthonormal constraints. Then, we focus on particular applications of this general framework, including the restoration of noisy direction fields, noisy chromaticity color images, estimated camera motions and DT-MRI (Di usion Tensor MRI) datasets.
We present a method for rendering 3D models in the traditional line-drawing style used in artistic and scientific illustrations. The goal is to suggest the 3D shape of the object using a small number of lines drawn with carefully chosen line qualities. The system combines several known techniques into a simple yet effective non-photorealistic line renderer. Feature edges related to the outline and interior of a given 3D mesh are extracted, segmented, and smoothed, yielding chains of lines with varying path, length, thickness, gaps and enclosures. The paper includes sample renderings obtained for a variety of models.
We present a new data structure---the bilateral grid, that enables fast edge-aware image processing. By working in the bilateral grid, algorithms such as bilateral filtering, edge-aware painting, and local histogram equalization become simple manipulations that are both local and independent. We parallelize our algorithms on modern GPUs to achieve real-time frame rates on high-definition video. We demonstrate our method on a variety of applications such as image editing, transfer of photographic look, and contrast enhancement of medical images.
We describe a GPU-based algorithm for rendering a 3D model as a line drawing, based on the insight that a line drawing can be understood as an abstraction of a shaded image. We thus render lines along tone boundaries or thin dark areas in the shaded image. We extend this notion to the dual: we render highlight lines along thin bright areas and tone boundaries. We combine the lines with toon shading to capture broad regions of tone.
The resulting line drawings effectively convey both shape and material cues. The lines produced by the method can include silhouettes. creases, and ridges, along with a generalization of suggestive contours that responds to lighting as well as viewing changes. The method supports automatic level of abstraction, where the size of depicted shape features adjusts appropriately as the camera zooms in or out. Animated models can be rendered in real time because costly mesh curvature calculations are not needed.
In this paper, we describe a non-photorealistic rendering system that conveys shape using lines. We go beyond contours and creases by developing a new type of line to draw: the suggestive contour. Suggestive contours are lines drawn on clearly visible parts of the surface, where a true contour would first appear with a minimal change in viewpoint. We provide two methods for calculating suggestive contours, including an algorithm that finds the zero crossings of the radial curvature. We show that suggestive contours can be drawn consistently with true contours, because they anticipate and extend them. We present a variety of results, arguing that these images convey shape more effectively than contour alone.
Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2007. Includes bibliographical references (p. 69-72). Non-photorealistic line drawing depicts 3D shapes through the rendering of feature lines. A number of characterizations of relevant lines have been proposed but none of these definitions alone seem to capture all visually-relevant lines. We introduce a new definition of feature lines based on two perceptual observations. First, human perception is sensitive to the variation of shading, and since shape perception is little affected by lighting and reflectance modification, we should focus on normal variation. Second, view-dependent lines better convey the shape of smooth surfaces better than view-independent lines. From this we define view-dependent curvature as the variation of the surface normal with respect to a viewing screen plane, and apparent ridges as the locus points of the maximum of the view-dependent curvature. We derive the equation for apparent ridges and present a new algorithm to render line drawings of 3D meshes. We show that our apparent ridges encompass or enhance aspects of several other feature lines. by Tilke Judd. S.M.
Median filtering is a cornerstone of modern image processing and is used extensively in smoothing and de-noising applications. The fastest commercial implementations (e.g. in Adobe ® Pho- toshop ® CS2) exhibit O(r) runtime in the radius of the filter, which limits their usefulness in realtime or resolution-independent contexts. We introduce a CPU-based, vectorizable O(log r) algo- rithm for median filtering, to our knowledge the most efficient yet developed. Our algorithm extends to images of any bit-depth, and can also be adapted to perform bilateral filtering. On 8-bit data our median filter outperforms Photoshop's implementation by up to a factor of fifty.
Good information design depends on clarifying the meaningful structure in an image. We describe a computational approach to stylizing and abstracting photographs that explicitly responds to this design goal. Our system transforms images into a line-drawing style using bold edges and large regions of constant color. To do this, it represents images as a hierarchical structure of parts and boundaries computed using state-of-the-art computer vision. Our system identifies the meaningful elements of this structure using a model of human perception and a record of a user's eye movements in looking at the photo; the system renders a new image using transformations that preserve and highlight these visual elements. Our method thus represents a new alternative for non-photorealistic rendering both in its visual style, in its approach to visual form, and in its techniques for interaction.
We describe a way to render stylized silhouettes of animated 3D models with temporal coherence. Coherence is one of the central challenges for non-photorealistic rendering. It is especially difficult for silhouettes, because they may not have obvious correspondences between frames. We demonstrate various coherence effects for stylized silhouettes with a robust working system. Our method runs in real-time for models of moderate complexity, making it suitable for both interactive applications and offline animation.
Visual content is often better communicated by simplified or exaggerated images than by the “real world like” images. In this paper, we offer a tool for creating such enhanced representations of photographs in a way consistent with the original image content. To do so, we develop a method to identify the relevant image strucures and their importance. Our approach (a) uses edges as the basic structural unit in the image, (b) proposes tools to manipulate this structure in a flexible way, and (c) employs gradient domain image processing techniques to reconstruct the final image from a “cropped” gradient information. This edge-based approach to non-photorealistic image processing is made feasible by two new techniques we introduce: an addition to the Gaussian scale space theory to compute a perceptually meaningful hierarchy of structures, and a contrast estimation method necessary for faithful gradient-based reconstructions. We finally present various applications that manipulate image structure in different ways.
Traditional toon shading uses a 1D texture that describes how tone varies with surface orientation relative to a given light source. In this paper we describe two extensions to the basic algorithm that support view-dependent effects. First, we replace the 1D texture with a 2D texture whose second dimension corresponds to the desired ``tone detail'', which can vary with depth or surface orientation. This supports effects such as levels-of-detail, aerial perspective, depth-of-field, backlighting, and specular highlights. Second, to control the amount of surface detail depicted by the shader, we further extend the toon shader to use a modified normal field that can range from the original normals to a simpler set of normals taken from an ``abstracted shape.'' A global shape detail parameter determines the degree of interpolation used between the original and abstracted normal fields. We explain how to implement these ideas efficiently on the GPU via vertex and fragment shaders, and discuss ways to extend our approach to alternative tone and shape detail maps.
Hair is a major feature of digital characters. Unfortunately, it has a complex geometry which challenges standard modeling tools. Some dedicated techniques exist, but creating a realistic hairstyle still takes hours. Complementary to user-driven methods, we here propose an image-based approach to capture the geometry of hair. The novelty of this work is that we draw information from the scattering properties of the hair that are normally considered a hindrance. To do so, we analyze image sequences from a fixed camera with a moving light source. We first introduce a novel method to compute the image orientation of the hairs from their anisotropic behavior. This method is proven to subsume and extend existing work while improving accuracy. This image orientation is then raised into a 3D orientation by analyzing the light reflected by the hair fibers. This part relies on minimal assumptions that have been proven correct in previous work. Finally, we show how to use several such image sequences to reconstruct the complete hair geometry of a real person. Results are shown to illustrate the fidelity of the captured geometry to the original hair. This technique paves the way for a new approach to digital hair generation.
The bilateral filter is a nonlinear filter that smoothes a signal while preserving strong edges. It has demonstrated great effectiveness for a variety of problems in computer vision and computer graphics, and fast versions have been proposed. Unfortunately, little is known about the accuracy of such accelerations. In this paper, we propose a new signal-processing analysis of the bilateral filter which complements the recent studies that analyzed it as a PDE or as a robust statistical estimator. The key to our analysis is to express the filter in a higher-dimensional space where the signal intensity is added to the original domain dimensions. Importantly, this signal-processing perspective allows us to develop a novel bilateral filtering acceleration using downsampling in space and intensity. This affords a principled expression of accuracy in terms of bandwidth and sampling. The bilateral filter can be expressed as linear convolutions in this augmented space followed by two simple nonlinearities. This allows us to derive criteria for downsampling the key operations and achieving important acceleration of the bilateral filter. We show that, for the same running time, our method is more accurate than previous acceleration techniques. Typically, we are able to process a 2~megapixel image using our acceleration technique in less than a second, and have the result be visually similar to the exact computation that takes several tens of minutes. The acceleration is most effective with large spatial kernels. Furthermore, this approach extends naturally to color images and cross bilateral filtering.
We've built a system for creating pen-and-ink illustrations that uses three components: darkness, texture, and orientation. Previous systems have used the first two. To allow the user to specify the third one, orientation, we have built an interactive direction field editor in which the user effectively 'paints' directions onto the illustration. The system then automatically creates the illustration at the requested output scale.
Diffusions provide a convenient way of smoothing noisy brightness images, of analyzing images at multiple scales, and of enhancing discontinuities. Some quantities of interest in computer vision are defined on curved manifolds; typical examples are orientation and hue that are defined on the circle. Generalizing diffusions to orientation is not straightforward, especially in the case where a discrete implementation is sought. An example of what may go wrong is presented.
A method is proposed to define diffusions of orientation-like quantities. First a definition in the continuum is discussed, then a discrete orientation diffusion is proposed. The behavior of such diffusions is explored both analytically and experimentally. It is shown how such orientation diffusions contain a nonlinearity that is reminiscent of edge-process and anisotropic diffusion. A number of open questions are proposed at the end.