Conference PaperPDF Available

BeCasso: Image Stylization by Interactive Oil Paint Filtering on Mobile Devices

July 2016

DOI:10.1145/2936744.2936750

Conference: SIGGRAPH Appy Hour

Project: Image and Video Abstraction

Authors:

Amir Semmo

Hasso Plattner Institute

Jürgen Döllner

Hasso Plattner Institute

Frank Schlegel

Digital Masterpieces GmbH

BeCasso is a mobile app that enables users to transform photos into an oil paint look that is inspired by traditional painting elements. In contrast to stroke-based approaches, the app uses state-of-the-art nonlinear image filtering techniques based on smoothed structure information to interactively synthesize oil paint renderings with soft color transitions. BeCasso empowers users to easily create aesthetic oil paint renderings by implementing a two-fold strategy. First, it provides parameter presets that may serve as a starting point for a custom stylization based on global parameter adjustments. Second, it introduces a novel interaction approach that operates within the parameter spaces of the stylization effect to facilitate creative control over the visual output: on-screen painting enables users to locally adjust the appearance in image regions, e.g., to vary the level of abstraction, brush and stroke direction. This way, the app provides tools for both higher-level interaction and low-level control to serve the different needs of non-experts and digital artists.

Content uploaded by Amir Semmo

Content may be subject to copyright.

BeCasso: Image Stylization by Interactive Oil Paint Filtering on Mobile Devices *

Amir Semmo J¨

urgen D¨

ollner

Hasso Plattner Institute, University of Potsdam, Germany∗

Frank Schlegel

Digital Masterpieces GmbH∗

Figure 1: A result interactively produced with our mobile app for an input image with a resolution of 2,560 ×1,800 pixels (left). The stylized

output (right) is based on real-time color grading, multi-stage ﬂow-based nonlinear ﬁltering, and a paint texture synthesis.

Abstract

BeCasso is a mobile

app that enables users

to transform photos into

an oil paint look that is

inspired by traditional

painting elements. In

contrast to stroke-based

approaches, the app uses

state-of-the-art nonlinear

image ﬁltering tech-

niques based on smoothed structure information to interactively

synthesize oil paint renderings with soft color transitions. BeCasso

empowers users to easily create aesthetic oil paint renderings by

implementing a two-fold strategy. First, it provides parameter

presets that may serve as a starting point for a custom stylization

based on global parameter adjustments. Second, it introduces

a novel interaction approach that operates within the parameter

spaces of the stylization effect to facilitate creative control over the

visual output: on-screen painting enables users to locally adjust the

appearance in image regions, e.g., to vary the level of abstraction,

brush and stroke direction. This way, the app provides tools for

both higher-level interaction and low-level control [Isenberg 2016]

to serve the different needs of non-experts and digital artists.

Keywords: mobile, image ﬁltering, stylization, interaction, GPU

Concepts: •Computing methodologies →Image manipulation;

∗http://www.hpi3d.de |http://www.digitalmasterpieces.com

This is the authors’ version of the work. It is posted here for your personal

use. Not for redistribution. The deﬁnitive version will be published in Pro-

ceedings of the 43rd International Conference and Exhibition on Computer

Graphics & Interactive Techniques (SIGGRAPH ’16).

2016 Copyright held by the owner/author(s).

SIGGRAPH ’16, July 24-28, 2016, Anaheim, CA,

ISBN: 978-1-4503-4376-3/16/07

DOI: http://dx.doi.org/10.1145/2936744.2936750

1 Technical Approach

Image stylization enjoys a growing popularity on mobile devices

to foster casual creativity [Winnem¨

oller 2013]. However, the pro-

vision of high-quality image effects for artistic rendering is still

faced by the inherent limitations of mobile graphics hardware such

as computing power and memory resources. In particular, the in-

teractive processing of high-resolution image data becomes an in-

creasingly challenging task for image-based artistic rendering that

requires several passes of (non-)linear ﬁltering.

This work presents BeCasso, a mobile app based on the oil paint ef-

fect of [Semmo et al. 2016] that provides several enhancements to

achieve the objective of an interactive image processing: (1) real-

time color grading using lookup tables [Selan 2004] is employed

to simulate rendering with a reduced color palette, (2) a multi-

scale approach is used that processes images on downsampled ver-

sions and performs joint bilateral upsampling [Kopf et al. 2007] to

achieve deliberate levels of abstraction at interactive frame rates,

and (3) a graph of the processing chains is maintained to dynami-

cally trigger only invalidated ﬁltering stages. These enhancements

signiﬁcantly facilitate the implementation of interactive tools that

can adjust the ﬁltering effects at run-time—a contemporary ﬁeld of

research [Isenberg 2016]—which is demonstrated by an on-screen

painting interface for per-pixel parameterization, e.g., to locally

vary the level of abstraction, brush and stroke direction.

Acknowledgments. This work was partly funded by the Federal Min-

istry of Education and Research (BMBF), Germany, within the InnoProﬁle

Transfer research group “4DnD-Vis” (www.4dndvis.de).

References

ISENBERG, T. 2016. Interactive NPAR: What Type of Tools Should We Create? In

Proc. NPAR, The Eurographics Association, Goslar, Germany, 89–96.

KOPF, J ., COHE N, M . F., LISCHINSKI, D ., AND UY TT END AEL E, M . 2007. Joint

Bilateral Upsampling. ACM Trans. Graph. 26, 3.

SELA N, J. 2004. Using Lookup Tables to Accelerate Color Transformations. In GPU

Gems. Addison-Wesley, 381–392.

SEMM O, A ., LIMBERGER, D., K YPRIANIDIS, J. E ., AND D¨

OLL NE R, J. 2016. Image

Stylization by Interactive Oil Paint Filtering. Computers & Graphics 55, 157–171.

WINNEM ¨

OLL ER , H . 2013. NPR in the Wild. In Image and Video-Based Artistic

Stylisation. Springer, 353–374.

Direction-aware Neural Style Transfer with Texture Enhancement

Article

Aug 2019
NEUROCOMPUTING

Neural learning methods have been shown to be effective in style transfer. These methods, which are called NST, aim to synthesize a new image that retains the high-level structure of a content image while keeps the low-level features of a style image. However, these models using convolutional structures only extract local statistical features of style images and semantic features of content images. Since the ab- sence of low-level features in the content image, these methods would synthesize images that look un- natural and full of traces of machines. In this paper, we find that direction, that is, the orientation of each painting stroke, can capture the soul of image style preferably and thus generates much more natural and vivid stylizations. According to this observation, we propose a direction-aware neural style transfer with texture enhancement. There are four major innovations. First, we separate the style transfer method into two stage, namely, NST stage and texture enhancement stage. Second, for the NST stage, a novel direction field loss is proposed to steer the direction of strokes in the synthesized image. And to build this loss function, we propose novel direction field loss networks to generate and compare the direction fields of content image and synthesized image. By incorporating the direction field loss in neural style transfer, we obtain a new optimization objective. Through minimizing this objective, we can produce synthesized images that better follow the direction field of the content image. Third, our method provides a simple interaction mechanism to control the generated direction fields, and further control the texture direction in synthesized images. Fourth, with a texture enhancement module, our method can add vivid texture details to the synthesized image. Experiments show that our method outperforms state-of-the-art style transfer method.

Direction-aware Neural Style Transfer

Conference Paper

Oct 2018

Neural learning methods have been shown to be effective in style transfer. These methods, which are called NST, aim to synthesize a new image that retains the high-level structure of a content image while keeps the low-level features of a style image. However, these models using convolutional structures only extract local statistical features of style images and semantic features of content images. Since the absence of low-level features in the content image, these methods would synthesize images that look unnatural and full of traces of machines. In this paper, we find that direction, that is, the orientation of each painting stroke, can capture the soul of image style preferably and thus generates much more natural and vivid stylizations. According to this observation, we propose a Direction-aware Neural Style Transfer (DaNST) with two major innovations. First, a novel direction field loss is proposed to steer the direction of strokes in the synthesized image. And to build this loss function, we propose novel direction field loss networks to generate and compare the direction fields of content image and synthesized image. By incorporating the direction field loss in neural style transfer, we obtain a new optimization objective. Through minimizing this objective, we can produce synthesized images that better follow the direction field of the content image. Second, our method provides a simple interaction mechanism to control the generated direction fields, and further control the texture direction in synthesized images. Experiments show that our method outperforms state-of-the-art in most styles such as oil painting and mosaic.

ProsumerFX: Mobile Design of Image Stylization Components

Conference Paper

Full-text available

Nov 2017

With the continuous advances of mobile graphics hardware, high-quality image stylization—e.g., based on image filtering, stroke-based rendering, and neural style transfer—is becoming feasible and increasingly used in casual creativity apps. The creative expression facilitated by these mobile apps, however, is typically limited with respect to the usage and application of pre-defined visual styles, which ultimately does not include their design and composition—an inherent requirement of prosumers. We present ProsumerFX, a GPU-based app that enables to interactively design parameterizable image stylization components on-device by reusing building blocks of image processing effects and pipelines. Furthermore, the presentation of the effects can be customized by modifying the icons, names, and order of parameters and presets. Thereby, the customized visual styles are defined as platform-independent effects and can be shared with other users via a web-based platform and database. Together with the presented mobile app, this system approach supports collaborative works for designing visual styles, including their rapid prototyping, A/B testing, publishing, and distribution. Thus, it satisfies the needs for creative expression of both professionals as well as the general public.

Service-based Out-of-Core Processing and Exploration of High-resolution Images

Thesis

Full-text available

Jul 2020

Florian Fregien

With the improvement of cameras and smartphones, more and more people can now take high-resolution pictures. Especially in the field of advertising and marketing, images with extremely high resolution are needed, e. g., for high quality print results. Also, in other fields such as art or medicine, images with several billion pixels are created. Due to their size, such gigapixel images cannot be processed or displayed similar to conventional images. Processing methods for such images have high performance requirements. Especially for mobile devices, which are even more limited in screen size and memory than computers, processing such images is hardly possible. In this thesis, a service-based approach for processing gigapixel images is presented to approach this problem. Cloud-based processing using different microservices enables a hardware-independent way to process gigapixel images. Therefore, the concept and implementation of such an integration into an existing service-based architecture is presented. To enable the exploration of gigapixel images, the integration of a gigapixel image viewer into a web application is presented. Furthermore, the design and implementation will be evaluated with regard to advantages, limitations, and runtime.

A Framework for Editing and Execution of Image and Video Processing Techniques on Mobile Devices

Thesis

Full-text available

Jul 2017

Tobias Dürschmid

With the continuous advances of mobile graphics hardware, high-quality image stylization, e.g., based on image filtering, stroke-based rendering, and neural style transfer, is becoming feasible and increasingly used in casual creativity apps. Nowadays, users want to create and distribute their own works and become a prosumer, i.e., being both consumer and producer. However, the creativity facilitated by contemporary mobile apps, is typically limited with respect to the usage and application of pre-defined visual styles, that ultimately does not include their design and composition – an inherent requirement of prosumers. This thesis presents the concept and implementation of a GPU-based mobile application that enables to interactively design parameterizable image stylization effects on-device, by reusing building blocks of image processing effects and pipelines. The parameterization is supported by three levels of control: (1) convenience presets, (2) global parameters, and (3) local parameter adjustments using on-screen painting. Furthermore, the created visual styles are defined using a platform-independent document format and can be shared with other users via a web-based community platform. The presented app is evaluated with regard to variety and visual quality of the styles, run-time performance measures, memory consumption, and implementation complexity metrics to demonstrate the feasibility of the concept. The results show that the app supports the interactive combination of complex effects such as neural style transfer, watercolor filtering, oil paint filtering, and pencil hatching filtering to create unique high-quality effects. This approach supports collaborative works for designing visual styles, including their rapid prototyping, A/B testing, publishing, and distribution. Hence, it satisfies the needs for creative expression of both professionals and novice users, i.e., the general public.

Challenges in User Experience Design of Image Filtering Apps

Conference Paper

Full-text available

Nov 2017

Photo filtering apps successfully deliver image-based stylization techniques to a broad audience, in particular in the ubiquitous domain (e.g., smartphones, tablet computers). Interacting with these inherently complex techniques has so far mostly been approached in two different ways: (1) by exposing many (technical) parameters to the user, resulting in a professional application that typically requires expert domain knowledge, or (2) by hiding the complexity via presets that only allows the application of filters but prevents creative expression thereon. In this work, we outline challenges of and present approaches for providing interactive image filtering on mobile devices, thereby focusing on how to make them usable for people in their daily life. This is discussed by the example of BeCasso, a user-centric app for assisted image stylization that targets two user groups: mobile artists and users seeking casual creativity. Through user research, qualitative and quantitative user studies, we identify and outline usability issues that showed to prevent both user groups from reaching their objectives when using the app. On the one hand, user-group-targeting has been improved by an optimized user experience design. On the other hand, multiple level of controls have been implemented to ease the interaction and hide the underlying complex technical parameters. Evaluations underline that the presented approach can increase the usability of complex image stylization techniques for mobile apps.

Interactive Image Filtering with Multiple Levels-of-Control on Mobile Devices

Conference Paper

Full-text available

Dec 2016

With the continuous development of mobile graphics hardware, interactive high-quality image stylization based on nonlinear filtering is becoming feasible and increasingly used in casual creativity apps. However, these apps often only serve high-level controls to parameterize image filters and generally lack support for low-level (artistic) control, thus automating art creation rather than assisting it. This work presents a GPU-based framework that enables to parameterize image filters at three levels of control: (1) presets followed by (2) global parameter adjustments can be interactively refined by (3) complementary on-screen painting that operates within the filters' parameter spaces for local adjustments. The framework provides a modular XML-based effect scheme to effectively build complex image processing chains-using these interactive filters as building blocks-that can be efficiently processed on mobile devices. Thereby, global and local parameterizations are directed with higher-level algorithmic support to ease the interactive editing process, which is demonstrated by state-of-the-art stylization effects, such as oil paint filtering and watercolor rendering.

Image stylization by interactive oil paint filtering

Article

Full-text available

Jan 2016
COMPUT GRAPH-UK

This paper presents an interactive system for transforming images into an oil paint look. The system comprises two major stages. First, it derives dominant colors from an input image for feature-aware recolorization and quantization to conform with a global color palette. Afterwards, it employs non-linear filtering based on the smoothed structure adapted to the main feature contours of the quantized image to synthesize a paint texture in real-time. Our filtering approach leads to homogeneous outputs in the color domain and enables creative control over the visual output, such as color adjustments and per-pixel parametrizations by means of interactive painting. To this end, our system introduces a generalized brush-based painting interface that operates within parameter spaces to locally adjust the level of abstraction of the filtering effects. Several results demonstrate the various applications of our filtering approach to different genres of photography.

Joint bilateral upsampling

Article

Full-text available

Jul 2007

Image analysis and enhancement tasks such as tone mapping, col- orization, stereo depth, and photomontage, often require computing a solution (e.g., for exposure, chromaticity, disparity, labels) over the pixel grid. Computational and memory costs often require that a smaller solution be run over a downsampled image. Although general purpose upsampling methods can be used to interpolate the low resolution solution to the full resolution, these methods gener- ally assume a smoothness prior for the interpolation. We demonstrate that in cases, such as those above, the available high resolution input image may be leveraged as a prior in the con- text of a joint bilateral upsampling procedure to produce a better high resolution solution. We show results for each of the applica- tions above and compare them to traditional upsampling methods. CR Categories: I.3.7 (Computer Graphics): Three-Dimensional Graphics and Realism—Color, shading, shadowing, and texture

NPR in the Wild

Chapter

Oct 2013

Holger Winnemoeller

During the early years of computer graphics, the results were arguably not as realistic as the intended goal set forth. However, it was not until sometime later that non-realism was accepted as a goal worthwhile pursuing. Since then, NPR has matured considerably and found uses in numerous applications, ranging from movies and television, production tools, and games, to novelty and casual creativity apps on mobile devices. This chapter presents examples from each of these categories within their historical and applied context.

Color transformations and lookup tables

Article

Jan 1992

Jan 2007
3

J Cohen
M F Lischinski
D And

KOPF, J., COHEN, M. F., LISCHINSKI, D., AND UYTTENDAELE, M. 2007. Joint Bilateral Upsampling. ACM Trans. Graph. 26, 3.

NPR in the Wild In Image and Video-Based Artistic Stylisation

H Winnemöller

Interactive NPAR: What Type of Tools Should We Create?

Jan 2016
89-96

T Isenberg

ISENBERG, T. 2016. Interactive NPAR: What Type of Tools Should We Create? In Proc. NPAR, The Eurographics Association, Goslar, Germany, 89-96.

Using Lookup Tables to Accelerate Color Transformations

Jan 2004
381-392

J Selan

SELAN, J. 2004. Using Lookup Tables to Accelerate Color Transformations. In GPU Gems. Addison-Wesley, 381-392.

BeCasso: Image Stylization by Interactive Oil Paint Filtering on Mobile Devices

Abstract

Recommendations

Image and Video Abstraction

Information Cartography

Real-Time Rendering Techniques

Interactive Image Filtering

MaeSTrO: A Mobile App for Style Transfer Orchestration using Neural Networks

Interactive Oil Paint Filtering On Mobile Devices

Interactive Multi-scale Oil Paint Filtering on Mobile Devices

Demo: Pictory - Neural Style Transfer and Editing with CoreML

BeCasso - Artistic Image Processing and Editing on Mobile Devices