Conference PaperPDF Available

BeCasso: Image Stylization by Interactive Oil Paint Filtering on Mobile Devices

Authors:

Abstract

BeCasso is a mobile app that enables users to transform photos into an oil paint look that is inspired by traditional painting elements. In contrast to stroke-based approaches, the app uses state-of-the-art nonlinear image filtering techniques based on smoothed structure information to interactively synthesize oil paint renderings with soft color transitions. BeCasso empowers users to easily create aesthetic oil paint renderings by implementing a two-fold strategy. First, it provides parameter presets that may serve as a starting point for a custom stylization based on global parameter adjustments. Second, it introduces a novel interaction approach that operates within the parameter spaces of the stylization effect to facilitate creative control over the visual output: on-screen painting enables users to locally adjust the appearance in image regions, e.g., to vary the level of abstraction, brush and stroke direction. This way, the app provides tools for both higher-level interaction and low-level control to serve the different needs of non-experts and digital artists.
BeCasso: Image Stylization by Interactive Oil Paint Filtering on Mobile Devices *
Amir Semmo J¨
urgen D¨
ollner
Hasso Plattner Institute, University of Potsdam, Germany
Frank Schlegel
Digital Masterpieces GmbH
Figure 1: A result interactively produced with our mobile app for an input image with a resolution of 2,560 ×1,800 pixels (left). The stylized
output (right) is based on real-time color grading, multi-stage flow-based nonlinear filtering, and a paint texture synthesis.
Abstract
BeCasso is a mobile
app that enables users
to transform photos into
an oil paint look that is
inspired by traditional
painting elements. In
contrast to stroke-based
approaches, the app uses
state-of-the-art nonlinear
image filtering tech-
niques based on smoothed structure information to interactively
synthesize oil paint renderings with soft color transitions. BeCasso
empowers users to easily create aesthetic oil paint renderings by
implementing a two-fold strategy. First, it provides parameter
presets that may serve as a starting point for a custom stylization
based on global parameter adjustments. Second, it introduces
a novel interaction approach that operates within the parameter
spaces of the stylization effect to facilitate creative control over the
visual output: on-screen painting enables users to locally adjust the
appearance in image regions, e.g., to vary the level of abstraction,
brush and stroke direction. This way, the app provides tools for
both higher-level interaction and low-level control [Isenberg 2016]
to serve the different needs of non-experts and digital artists.
Keywords: mobile, image filtering, stylization, interaction, GPU
Concepts: Computing methodologies Image manipulation;
http://www.hpi3d.de |http://www.digitalmasterpieces.com
This is the authors’ version of the work. It is posted here for your personal
use. Not for redistribution. The definitive version will be published in Pro-
ceedings of the 43rd International Conference and Exhibition on Computer
Graphics & Interactive Techniques (SIGGRAPH ’16).
c
2016 Copyright held by the owner/author(s).
SIGGRAPH ’16, July 24-28, 2016, Anaheim, CA,
ISBN: 978-1-4503-4376-3/16/07
DOI: http://dx.doi.org/10.1145/2936744.2936750
1 Technical Approach
Image stylization enjoys a growing popularity on mobile devices
to foster casual creativity [Winnem¨
oller 2013]. However, the pro-
vision of high-quality image effects for artistic rendering is still
faced by the inherent limitations of mobile graphics hardware such
as computing power and memory resources. In particular, the in-
teractive processing of high-resolution image data becomes an in-
creasingly challenging task for image-based artistic rendering that
requires several passes of (non-)linear filtering.
This work presents BeCasso, a mobile app based on the oil paint ef-
fect of [Semmo et al. 2016] that provides several enhancements to
achieve the objective of an interactive image processing: (1) real-
time color grading using lookup tables [Selan 2004] is employed
to simulate rendering with a reduced color palette, (2) a multi-
scale approach is used that processes images on downsampled ver-
sions and performs joint bilateral upsampling [Kopf et al. 2007] to
achieve deliberate levels of abstraction at interactive frame rates,
and (3) a graph of the processing chains is maintained to dynami-
cally trigger only invalidated filtering stages. These enhancements
significantly facilitate the implementation of interactive tools that
can adjust the filtering effects at run-time—a contemporary field of
research [Isenberg 2016]—which is demonstrated by an on-screen
painting interface for per-pixel parameterization, e.g., to locally
vary the level of abstraction, brush and stroke direction.
Acknowledgments. This work was partly funded by the Federal Min-
istry of Education and Research (BMBF), Germany, within the InnoProfile
Transfer research group “4DnD-Vis” (www.4dndvis.de).
References
ISENBERG, T. 2016. Interactive NPAR: What Type of Tools Should We Create? In
Proc. NPAR, The Eurographics Association, Goslar, Germany, 89–96.
KOPF, J ., COHE N, M . F., LISCHINSKI, D ., AND UY TT END AEL E, M . 2007. Joint
Bilateral Upsampling. ACM Trans. Graph. 26, 3.
SELA N, J. 2004. Using Lookup Tables to Accelerate Color Transformations. In GPU
Gems. Addison-Wesley, 381–392.
SEMM O, A ., LIMBERGER, D., K YPRIANIDIS, J. E ., AND D¨
OLL NE R, J. 2016. Image
Stylization by Interactive Oil Paint Filtering. Computers & Graphics 55, 157–171.
WINNEM ¨
OLL ER , H . 2013. NPR in the Wild. In Image and Video-Based Artistic
Stylisation. Springer, 353–374.
... In the literature of both methods, there are methods that rendering or transferring the direction strokes or direction texture along the direction field of the content image in a direction-aware way. Many works have proved that the directionaware method has a better effect on the art created by directional strokes, such as oil painting and embroidery [12][13][14][15][16][17] . ...
... Then they introduce flowguided anisotropic filtering for detecting highly coherent lines to produce a coherent line drawing effect. Semmo et al. [6,15] propose a similar direction field extraction method based on adaptively smoothed structure tensor. They use the direction field to generate paint texture that looks like directional painting strokes. ...
Article
Neural learning methods have been shown to be effective in style transfer. These methods, which are called NST, aim to synthesize a new image that retains the high-level structure of a content image while keeps the low-level features of a style image. However, these models using convolutional structures only extract local statistical features of style images and semantic features of content images. Since the ab- sence of low-level features in the content image, these methods would synthesize images that look un- natural and full of traces of machines. In this paper, we find that direction, that is, the orientation of each painting stroke, can capture the soul of image style preferably and thus generates much more natural and vivid stylizations. According to this observation, we propose a direction-aware neural style transfer with texture enhancement. There are four major innovations. First, we separate the style transfer method into two stage, namely, NST stage and texture enhancement stage. Second, for the NST stage, a novel direction field loss is proposed to steer the direction of strokes in the synthesized image. And to build this loss function, we propose novel direction field loss networks to generate and compare the direction fields of content image and synthesized image. By incorporating the direction field loss in neural style transfer, we obtain a new optimization objective. Through minimizing this objective, we can produce synthesized images that better follow the direction field of the content image. Third, our method provides a simple interaction mechanism to control the generated direction fields, and further control the texture direction in synthesized images. Fourth, with a texture enhancement module, our method can add vivid texture details to the synthesized image. Experiments show that our method outperforms state-of-the-art style transfer method.
... In the literatures of both methods, there are methods that rendering or transferring the direction strokes or direction texture along the direction field of the content image in a direction-aware way. Many works have proved that the directionaware method has better effect on the art created by directional strokes, such as oil painting and embroidery [7,15,16,18,23,24]. ...
... Then they introduce a flow-guided anisotropic filtering for detecting highly coherent lines to produce a coherent line drawing effect. Amir et al. [18,20] propose a similar direction field extraction method based on adaptively smoothed structure tensor. They use direction field to generate paint texture that looks like directional painting strokes. ...
Conference Paper
Neural learning methods have been shown to be effective in style transfer. These methods, which are called NST, aim to synthesize a new image that retains the high-level structure of a content image while keeps the low-level features of a style image. However, these models using convolutional structures only extract local statistical features of style images and semantic features of content images. Since the absence of low-level features in the content image, these methods would synthesize images that look unnatural and full of traces of machines. In this paper, we find that direction, that is, the orientation of each painting stroke, can capture the soul of image style preferably and thus generates much more natural and vivid stylizations. According to this observation, we propose a Direction-aware Neural Style Transfer (DaNST) with two major innovations. First, a novel direction field loss is proposed to steer the direction of strokes in the synthesized image. And to build this loss function, we propose novel direction field loss networks to generate and compare the direction fields of content image and synthesized image. By incorporating the direction field loss in neural style transfer, we obtain a new optimization objective. Through minimizing this objective, we can produce synthesized images that better follow the direction field of the content image. Second, our method provides a simple interaction mechanism to control the generated direction fields, and further control the texture direction in synthesized images. Experiments show that our method outperforms state-of-the-art in most styles such as oil painting and mosaic.
... (2) global parameters; and (3) local parameter adjustments using on-screen painting metaphors [Semmo et al. 2016a]. Based on that framework, this paper presents a system that enables its users to easily modify and recombine IVOs, i.e., to create and share new image stylization components that support these three interaction concepts. ...
... More complex stylization effects are provided by neural style transfer apps [Gatys et al. 2016;Johnson et al. 2016] such as Prisma, which simulate artistic styles of famous artists and epochs, but typically do not provide explicit parameterizations of style components or phenomena. Conversely, this can be achieved by mask-based parameter painting apps such as BeCasso [Pasewaldt et al. 2016;Semmo et al. 2016a] and PaintCan [Benedetti et al. 2014], where image filtering techniques that simulate watercolor, oil paint, and cartoon styles can be adjusted with a high, medium and low level-of-control [Isenberg 2016]. The combination of neural style transfer with post processing via image filtering can be done with Pictory [Semmo et al. 2017b]. ...
Conference Paper
Full-text available
With the continuous advances of mobile graphics hardware, high-quality image stylization—e.g., based on image filtering, stroke-based rendering, and neural style transfer—is becoming feasible and increasingly used in casual creativity apps. The creative expression facilitated by these mobile apps, however, is typically limited with respect to the usage and application of pre-defined visual styles, which ultimately does not include their design and composition—an inherent requirement of prosumers. We present ProsumerFX, a GPU-based app that enables to interactively design parameterizable image stylization components on-device by reusing building blocks of image processing effects and pipelines. Furthermore, the presentation of the effects can be customized by modifying the icons, names, and order of parameters and presets. Thereby, the customized visual styles are defined as platform-independent effects and can be shared with other users via a web-based platform and database. Together with the presented mobile app, this system approach supports collaborative works for designing visual styles, including their rapid prototyping, A/B testing, publishing, and distribution. Thus, it satisfies the needs for creative expression of both professionals as well as the general public.
... With the continuous development of mobile graphics hardware, interactive high-quality image and video processing, such as based on nonlinear filtering for oil paint stylization [45,47] and a MapReduce approach [22], is becoming feasible and thus of particular interest for industrial and educational purposes [52], e. g., when used for implementing casual creativity applications. At this, popular applications such as BeCasso [38,25] and Pictory [44,26] typically employ a user-centric approach for assisted image stylization targeting mobile artists and users seeking casual creativity, thereby integrating user experience concepts for making image filters usable in their daily life [16]. This is technically achieved by parameterizing image filters at three levels of control, i. e., using presets, global parameter adjustments and complementary on-screen painting that operates within the filters' parameter spaces for local adjustments [39]. ...
Thesis
Full-text available
With the improvement of cameras and smartphones, more and more people can now take high-resolution pictures. Especially in the field of advertising and marketing, images with extremely high resolution are needed, e. g., for high quality print results. Also, in other fields such as art or medicine, images with several billion pixels are created. Due to their size, such gigapixel images cannot be processed or displayed similar to conventional images. Processing methods for such images have high performance requirements. Especially for mobile devices, which are even more limited in screen size and memory than computers, processing such images is hardly possible. In this thesis, a service-based approach for processing gigapixel images is presented to approach this problem. Cloud-based processing using different microservices enables a hardware-independent way to process gigapixel images. Therefore, the concept and implementation of such an integration into an existing service-based architecture is presented. To enable the exploration of gigapixel images, the integration of a gigapixel image viewer into a web application is presented. Furthermore, the design and implementation will be evaluated with regard to advantages, limitations, and runtime.
... Prisma uses neural style transfer [25,63] to create known artistic effects which are not parameterizable. BeCasso [50,62] uses GPU-based image filtering techniques to create watercolor, oil paint, and cartoon styles with a high, medium as well as low level of control [31]. The combination of neural style transfer with post processing via image filtering can be done with Pictory [67]. ...
Thesis
Full-text available
With the continuous advances of mobile graphics hardware, high-quality image stylization, e.g., based on image filtering, stroke-based rendering, and neural style transfer, is becoming feasible and increasingly used in casual creativity apps. Nowadays, users want to create and distribute their own works and become a prosumer, i.e., being both consumer and producer. However, the creativity facilitated by contemporary mobile apps, is typically limited with respect to the usage and application of pre-defined visual styles, that ultimately does not include their design and composition – an inherent requirement of prosumers. This thesis presents the concept and implementation of a GPU-based mobile application that enables to interactively design parameterizable image stylization effects on-device, by reusing building blocks of image processing effects and pipelines. The parameterization is supported by three levels of control: (1) convenience presets, (2) global parameters, and (3) local parameter adjustments using on-screen painting. Furthermore, the created visual styles are defined using a platform-independent document format and can be shared with other users via a web-based community platform. The presented app is evaluated with regard to variety and visual quality of the styles, run-time performance measures, memory consumption, and implementation complexity metrics to demonstrate the feasibility of the concept. The results show that the app supports the interactive combination of complex effects such as neural style transfer, watercolor filtering, oil paint filtering, and pencil hatching filtering to create unique high-quality effects. This approach supports collaborative works for designing visual styles, including their rapid prototyping, A/B testing, publishing, and distribution. Hence, it satisfies the needs for creative expression of both professionals and novice users, i.e., the general public.
Conference Paper
Full-text available
Photo filtering apps successfully deliver image-based stylization techniques to a broad audience, in particular in the ubiquitous domain (e.g., smartphones, tablet computers). Interacting with these inherently complex techniques has so far mostly been approached in two different ways: (1) by exposing many (technical) parameters to the user, resulting in a professional application that typically requires expert domain knowledge, or (2) by hiding the complexity via presets that only allows the application of filters but prevents creative expression thereon. In this work, we outline challenges of and present approaches for providing interactive image filtering on mobile devices, thereby focusing on how to make them usable for people in their daily life. This is discussed by the example of BeCasso, a user-centric app for assisted image stylization that targets two user groups: mobile artists and users seeking casual creativity. Through user research, qualitative and quantitative user studies, we identify and outline usability issues that showed to prevent both user groups from reaching their objectives when using the app. On the one hand, user-group-targeting has been improved by an optimized user experience design. On the other hand, multiple level of controls have been implemented to ease the interaction and hide the underlying complex technical parameters. Evaluations underline that the presented approach can increase the usability of complex image stylization techniques for mobile apps.
Conference Paper
Full-text available
With the continuous development of mobile graphics hardware, interactive high-quality image stylization based on nonlinear filtering is becoming feasible and increasingly used in casual creativity apps. However, these apps often only serve high-level controls to parameterize image filters and generally lack support for low-level (artistic) control, thus automating art creation rather than assisting it. This work presents a GPU-based framework that enables to parameterize image filters at three levels of control: (1) presets followed by (2) global parameter adjustments can be interactively refined by (3) complementary on-screen painting that operates within the filters' parameter spaces for local adjustments. The framework provides a modular XML-based effect scheme to effectively build complex image processing chains-using these interactive filters as building blocks-that can be efficiently processed on mobile devices. Thereby, global and local parameterizations are directed with higher-level algorithmic support to ease the interactive editing process, which is demonstrated by state-of-the-art stylization effects, such as oil paint filtering and watercolor rendering.
Article
Full-text available
This paper presents an interactive system for transforming images into an oil paint look. The system comprises two major stages. First, it derives dominant colors from an input image for feature-aware recolorization and quantization to conform with a global color palette. Afterwards, it employs non-linear filtering based on the smoothed structure adapted to the main feature contours of the quantized image to synthesize a paint texture in real-time. Our filtering approach leads to homogeneous outputs in the color domain and enables creative control over the visual output, such as color adjustments and per-pixel parametrizations by means of interactive painting. To this end, our system introduces a generalized brush-based painting interface that operates within parameter spaces to locally adjust the level of abstraction of the filtering effects. Several results demonstrate the various applications of our filtering approach to different genres of photography.
Article
Full-text available
Image analysis and enhancement tasks such as tone mapping, col- orization, stereo depth, and photomontage, often require computing a solution (e.g., for exposure, chromaticity, disparity, labels) over the pixel grid. Computational and memory costs often require that a smaller solution be run over a downsampled image. Although general purpose upsampling methods can be used to interpolate the low resolution solution to the full resolution, these methods gener- ally assume a smoothness prior for the interpolation. We demonstrate that in cases, such as those above, the available high resolution input image may be leveraged as a prior in the con- text of a joint bilateral upsampling procedure to produce a better high resolution solution. We show results for each of the applica- tions above and compare them to traditional upsampling methods. CR Categories: I.3.7 (Computer Graphics): Three-Dimensional Graphics and Realism—Color, shading, shadowing, and texture
Chapter
During the early years of computer graphics, the results were arguably not as realistic as the intended goal set forth. However, it was not until sometime later that non-realism was accepted as a goal worthwhile pursuing. Since then, NPR has matured considerably and found uses in numerous applications, ranging from movies and television, production tools, and games, to novelty and casual creativity apps on mobile devices. This chapter presents examples from each of these categories within their historical and applied context.
  • J Cohen
  • M F Lischinski
  • D And
KOPF, J., COHEN, M. F., LISCHINSKI, D., AND UYTTENDAELE, M. 2007. Joint Bilateral Upsampling. ACM Trans. Graph. 26, 3.
NPR in the Wild In Image and Video-Based Artistic Stylisation
  • H Winnemöller
Interactive NPAR: What Type of Tools Should We Create?
  • T Isenberg
ISENBERG, T. 2016. Interactive NPAR: What Type of Tools Should We Create? In Proc. NPAR, The Eurographics Association, Goslar, Germany, 89-96.
Using Lookup Tables to Accelerate Color Transformations
  • J Selan
SELAN, J. 2004. Using Lookup Tables to Accelerate Color Transformations. In GPU Gems. Addison-Wesley, 381-392.