Conference PaperPDF Available

BeCasso - Artistic Image Processing and Editing on Mobile Devices

Authors:

Abstract

BeCasso is a mobile app that enables users to transform photos into high-quality, high-resolution non-photorealistic renditions, such as oil and watercolor paintings, cartoons, and colored pencil drawings, which are inspired by real-world paintings or drawing techniques. In contrast to neuronal network and physically-based approaches, the app employs state-of-the-art nonlinear image filtering. For example, oil paint and cartoon effects are based on smoothed structure information to interactively synthesize renderings with soft color transitions. BeCasso empowers users to easily create aesthetic renderings by implementing a two-fold strategy: First, it provides parameter presets that may serve as a starting point for a custom stylization based on global parameter adjustments. Thereby, users can obtain initial renditions that may be fine-tuned afterwards. Second, it enables local style adjustments: using on-screen painting metaphors, users are able to locally adjust different stylization features, e.g., to vary the level of abstraction, pen, brush and stroke direction or the contour lines. In this way, the app provides tools for both higher-level interaction and low-level control to serve the different needs of non-experts and digital artists.
Demo: BeCasso - Artistic Image Processing and Editing on Mobile Devices
Sebastian Pasewaldt Amir Semmo J¨
urgen D¨
ollner
Hasso Plattner Institute, University of Potsdam, Germany
Frank Schlegel
Digital Masterpieces GmbH
Figure 1: Three results produced with BeCasso for an input image with a resolution of 2,400 ×1,600 pixels (displayed cropped): cartoon
(left), oil paint (middle) and watercolor style (right). The outputs are based on multi-stage, flow-based nonlinear filtering and color grading.
Abstract
BeCasso is a mobile app that enables users to transform photos into
high-quality, high-resolution non-photorealistic renditions, such as
oil and watercolor paintings, cartoons, and colored pencil drawings,
which are inspired by real-world paintings or drawing techniques.
In contrast to neuronal network and physically-based approaches,
the app employs state-of-the-art nonlinear image filtering. For ex-
ample, oil paint and cartoon effects are based on smoothed structure
information to interactively synthesize renderings with soft color
transitions. BeCasso empowers users to easily create aesthetic ren-
derings by implementing a two-fold strategy: First, it provides pa-
rameter presets that may serve as a starting point for a custom styl-
ization based on global parameter adjustments. Thereby, users can
obtain initial renditions that may be fine-tuned afterwards. Sec-
ond, it enables local style adjustments: using on-screen painting
metaphors, users are able to locally adjust different stylization fea-
tures, e.g., to vary the level of abstraction, pen, brush and stroke di-
rection or the contour lines. In this way, the app provides tools for
both higher-level interaction and low-level control [Isenberg 2016]
to serve the different needs of non-experts and digital artists.
Keywords: mobile, non-photorealistic rendering, image filtering,
stylization, interaction, GPU
Concepts: Computing methodologies Image manipulation;
Human-centered computing Mobile devices;
http://www.hpi3d.de |http://www.digitalmasterpieces.com
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not
made or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for components
of this work owned by others than ACM must be honored. Abstracting with
credit is permitted. To copy otherwise, or republish, to post on servers or to
redistribute to lists, requires prior specific permission and/or a fee. Request
permissions from permissions@acm.org. c
2017 ACM.
SA ’16, December 05-08, 2016, Macao
ISBN: 978-1-4503-4551-4/16/12
DOI: http://dx.doi.org/10.1145/2999508.2999518
Technical Approach
Image stylization enjoys a growing popularity on mobile devices to
foster casual creativity [Winnem¨
oller 2013]. With the limiting hard-
ware capabilities of mobile devices, an interactive processing of
high-resolution images becomes an increasingly challenging task,
in particular for artistic rendering that is based on multi-stage non-
linear filtering. One approach is to shift complex processing tasks
to dedicated servers and only use mobile devices for image display,
which however sacrifices interactive manipulations by users.
This work presents BeCasso, a mobile app that implements a GPU-
based, efficient image analysis and processing pipeline to realize
the objective of an interactive image processing on mobile devices:
(1) real-time color grading using lookup tables is employed to sim-
ulate rendering with reduced color palettes, (2) a multi-scale ap-
proach processes images on downsampled versions and performs
upsampling to achieve deliberate levels of abstraction [Semmo et al.
2016], (3) graph-based processing chains of multi-stage effects
are analyzed to dynamically trigger only invalidated stages, and
(4) algorithms for an efficient (re-)use of textures reduce the mem-
ory footprint while maintaining rendering performance. These en-
hancements significantly facilitate the implementation of interac-
tive tools to adjust filtering effects at run-time, such as toon, wa-
tercolor and oil paint (Figure 1). This is demonstrated by an on-
screen painting interface for per-pixel parameterization, e.g., to lo-
cally vary the color diffusion and level of abstraction.
Acknowledgments
This work was partly funded by the Federal Ministry of Education and Re-
search (BMBF), Germany, for the AVA project 01IS15041B and within the
InnoProfile Transfer research group “4DnD-Vis” (www.4dndvis.de).
References
ISENBERG, T. 2016. Interactive NPAR: What Type of Tools Should We
Create? In Proc. NPAR, 89–96.
SEMMO, A., TRAPP, M., D ¨
URSCHMID,T.,D
¨
OLLNER, J., AND PASE-
WALDT, S. 2016. Interactive Multi-scale Oil Paint Filtering on Mobile
Devices. In Proc. ACM SIGGRAPH Posters, 42:1–42:2.
WINNEM ¨
OLLER, H . 2013. NPR in the Wild. In Image and Video-Based
Artistic Stylisation. Springer, 353–374.
... However, increasing the style-representation capacity of a network generally represents a trade-off in quality, memory, and run-time performance versus single-style networks [17,35]. In mobile applications, single-style-per-network approaches have thus remained the most suitable and prevalent network architecture (e.g., refer to Prisma [31] and Becasso [32]). While most style transfer methods seek to achieve plausible results on a global level without requiring user interaction, several approaches enable control over perceptual elements of the output to a varying degree [18]. ...
Article
Full-text available
Fast style transfer methods have recently gained popularity in art-related applications as they make a generalized real-time stylization of images practicable. However, they are mostly limited to one-shot stylizations concerning the interactive adjustment of style elements. In particular, the expressive control over stroke sizes or stroke orientations remains an open challenge. To this end, we propose a novel stroke-adjustable fast style transfer network that enables simultaneous control over the stroke size and intensity, and allows a wider range of expressive editing than current approaches by utilizing the scale-variance of convolutional neural networks. Furthermore, we introduce a network-agnostic approach for style-element editing by applying reversible input transformations that can adjust strokes in the stylized output. At this, stroke orientations can be adjusted, and warping-based effects can be applied to stylistic elements, such as swirls or waves. To demonstrate the real-world applicability of our approach, we present StyleTune, a mobile app for interactive editing of neural style transfers at multiple levels of control. Our app allows stroke adjustments on a global and local level. It furthermore implements an on-device patch-based upsampling step that enables users to achieve results with high output fidelity and resolutions of more than 20 megapixels. Our approach allows users to art-direct their creations and achieve results that are not possible with current style transfer applications.
... While the approach of Johnson et al. [10] can represent a single style per network, follow-up works added the ability to represent multiple styles (Zhang et al. [11], Msg Net) or arbitrary styles (Huang et al. adaptive Instance Normalization (adaIN) [12]). However, representing more styles with one single network generally represents a tradeoff in quality, memory and run-time performance versus single-style networks [13], [14], and thus the single-styleper-network approaches have prevailed in mobile applications as the "gold standard" (e.g., refer to Prisma [15] and Becasso [16]). While the overall goal of most style transfer methods has been to achieve plausible global results without requiring user interaction, several methods allow to adjust perceptual factors of the output in varying degrees, and thus directly or indirectly control semiotic aspects known from artwork production [7]. ...
Conference Paper
Full-text available
We present StyleTune, a mobile app for interactive multi-level control of neural style transfers that facilitates creative adjustments of style elements and enables high output fidelity. In contrast to current mobile neural style transfer apps, StyleTune supports users to adjust both the size and orientation of style elements, such as brushstrokes and texture patches, on a global as well as local level. To this end, we propose a novel stroke-adaptive feed-forward style transfer network, that enables control over stroke size and intensity and allows a larger range of edits than current approaches. For additional level-of-control, we propose a network-agnostic method for stroke-orientation adjustment by utilizing the rotation-variance of Convolutional Neural Networks (CNNs). To achieve high output fidelity, we further add a patch-based style transfer method that enables users to obtain output resolutions of more than 20 Megapixel (Mpix). Our approach empowers users to create many novel results that are not possible with current mobile neural style transfer apps.
... While the approach of Johnson et al. [10] can represent a single style per network, follow-up works added the ability to represent multiple styles (Zhang et al. [11], Msg Net) or arbitrary styles (Huang et al. adaptive Instance Normalization (adaIN) [12]). However, representing more styles with one single network generally represents a tradeoff in quality, memory and run-time performance versus single-style networks [13], [14], and thus the single-styleper-network approaches have prevailed in mobile applications as the "gold standard" (e.g., refer to Prisma [15] and Becasso [16]). While the overall goal of most style transfer methods has been to achieve plausible global results without requiring user interaction, several methods allow to adjust perceptual factors of the output in varying degrees, and thus directly or indirectly control semiotic aspects known from artwork production [7]. ...
Preprint
Full-text available
We present StyleTune, a mobile app for interactive multi-level control of neural style transfers that facilitates creative adjustments of style elements and enables high output fidelity. In contrast to current mobile neural style transfer apps, StyleTune supports users to adjust both the size and orientation of style elements, such as brushstrokes and texture patches, on a global as well as local level. To this end, we propose a novel stroke-adaptive feed-forward style transfer network, that enables control over stroke size and intensity and allows a larger range of edits than current approaches. For additional level-of-control, we propose a network agnostic method for stroke-orientation adjustment by utilizing the rotation-variance of CNNs. To achieve high output fidelity, we further add a patch-based style transfer method that enables users to obtain output resolutions of more than 20 Megapixel. Our approach empowers users to create many novel results that are not possible with current mobile neural style transfer apps.
... With the continuous development of mobile graphics hardware, interactive high-quality image and video processing, such as based on nonlinear filtering for oil paint stylization [45,47] and a MapReduce approach [22], is becoming feasible and thus of particular interest for industrial and educational purposes [52], e. g., when used for implementing casual creativity applications. At this, popular applications such as BeCasso [38,25] and Pictory [44,26] typically employ a user-centric approach for assisted image stylization targeting mobile artists and users seeking casual creativity, thereby integrating user experience concepts for making image filters usable in their daily life [16]. This is technically achieved by parameterizing image filters at three levels of control, i. e., using presets, global parameter adjustments and complementary on-screen painting that operates within the filters' parameter spaces for local adjustments [39]. ...
Thesis
Full-text available
With the improvement of cameras and smartphones, more and more people can now take high-resolution pictures. Especially in the field of advertising and marketing, images with extremely high resolution are needed, e. g., for high quality print results. Also, in other fields such as art or medicine, images with several billion pixels are created. Due to their size, such gigapixel images cannot be processed or displayed similar to conventional images. Processing methods for such images have high performance requirements. Especially for mobile devices, which are even more limited in screen size and memory than computers, processing such images is hardly possible. In this thesis, a service-based approach for processing gigapixel images is presented to approach this problem. Cloud-based processing using different microservices enables a hardware-independent way to process gigapixel images. Therefore, the concept and implementation of such an integration into an existing service-based architecture is presented. To enable the exploration of gigapixel images, the integration of a gigapixel image viewer into a web application is presented. Furthermore, the design and implementation will be evaluated with regard to advantages, limitations, and runtime.
... is paper presents the ongoing development of BeCasso [Pasewaldt et al. 2016;], a productization of image stylization research into an interactive mobile iOS app. BeCasso transforms photos into artistic renditions that are inspired by real paintings and drawings techniques, such as oil and watercolor paintings. ...
Conference Paper
Full-text available
Photo filtering apps successfully deliver image-based stylization techniques to a broad audience, in particular in the ubiquitous domain (e.g., smartphones, tablet computers). Interacting with these inherently complex techniques has so far mostly been approached in two different ways: (1) by exposing many (technical) parameters to the user, resulting in a professional application that typically requires expert domain knowledge, or (2) by hiding the complexity via presets that only allows the application of filters but prevents creative expression thereon. In this work, we outline challenges of and present approaches for providing interactive image filtering on mobile devices, thereby focusing on how to make them usable for people in their daily life. This is discussed by the example of BeCasso, a user-centric app for assisted image stylization that targets two user groups: mobile artists and users seeking casual creativity. Through user research, qualitative and quantitative user studies, we identify and outline usability issues that showed to prevent both user groups from reaching their objectives when using the app. On the one hand, user-group-targeting has been improved by an optimized user experience design. On the other hand, multiple level of controls have been implemented to ease the interaction and hide the underlying complex technical parameters. Evaluations underline that the presented approach can increase the usability of complex image stylization techniques for mobile apps.
... More complex stylization effects are provided by neural style transfer apps [Gatys et al. 2016;Johnson et al. 2016] such as Prisma, which simulate artistic styles of famous artists and epochs, but typically do not provide explicit parameterizations of style components or phenomena. Conversely, this can be achieved by mask-based parameter painting apps such as BeCasso [Pasewaldt et al. 2016;Semmo et al. 2016a] and PaintCan [Benedetti et al. 2014], where image filtering techniques that simulate watercolor, oil paint, and cartoon styles can be adjusted with a high, medium and low level-of-control [Isenberg 2016]. The combination of neural style transfer with post processing via image filtering can be done with Pictory [Semmo et al. 2017b]. ...
Conference Paper
Full-text available
With the continuous advances of mobile graphics hardware, high-quality image stylization—e.g., based on image filtering, stroke-based rendering, and neural style transfer—is becoming feasible and increasingly used in casual creativity apps. The creative expression facilitated by these mobile apps, however, is typically limited with respect to the usage and application of pre-defined visual styles, which ultimately does not include their design and composition—an inherent requirement of prosumers. We present ProsumerFX, a GPU-based app that enables to interactively design parameterizable image stylization components on-device by reusing building blocks of image processing effects and pipelines. Furthermore, the presentation of the effects can be customized by modifying the icons, names, and order of parameters and presets. Thereby, the customized visual styles are defined as platform-independent effects and can be shared with other users via a web-based platform and database. Together with the presented mobile app, this system approach supports collaborative works for designing visual styles, including their rapid prototyping, A/B testing, publishing, and distribution. Thus, it satisfies the needs for creative expression of both professionals as well as the general public.
... Prisma uses neural style transfer [25,63] to create known artistic effects which are not parameterizable. BeCasso [50,62] uses GPU-based image filtering techniques to create watercolor, oil paint, and cartoon styles with a high, medium as well as low level of control [31]. The combination of neural style transfer with post processing via image filtering can be done with Pictory [67]. ...
Thesis
Full-text available
With the continuous advances of mobile graphics hardware, high-quality image stylization, e.g., based on image filtering, stroke-based rendering, and neural style transfer, is becoming feasible and increasingly used in casual creativity apps. Nowadays, users want to create and distribute their own works and become a prosumer, i.e., being both consumer and producer. However, the creativity facilitated by contemporary mobile apps, is typically limited with respect to the usage and application of pre-defined visual styles, that ultimately does not include their design and composition – an inherent requirement of prosumers. This thesis presents the concept and implementation of a GPU-based mobile application that enables to interactively design parameterizable image stylization effects on-device, by reusing building blocks of image processing effects and pipelines. The parameterization is supported by three levels of control: (1) convenience presets, (2) global parameters, and (3) local parameter adjustments using on-screen painting. Furthermore, the created visual styles are defined using a platform-independent document format and can be shared with other users via a web-based community platform. The presented app is evaluated with regard to variety and visual quality of the styles, run-time performance measures, memory consumption, and implementation complexity metrics to demonstrate the feasibility of the concept. The results show that the app supports the interactive combination of complex effects such as neural style transfer, watercolor filtering, oil paint filtering, and pencil hatching filtering to create unique high-quality effects. This approach supports collaborative works for designing visual styles, including their rapid prototyping, A/B testing, publishing, and distribution. Hence, it satisfies the needs for creative expression of both professionals and novice users, i.e., the general public.
Conference Paper
Full-text available
We present StyleTune, a mobile app for interactive style transfer enhancement that enables global and spatial control over stroke elements and can generate high fidelity outputs. The app uses adjustable neural style transfer (NST) networks to enable art-direction of stroke size and orientation in the output image. The implemented approach enables continuous and seamless edits through a unified stroke-size representation in the feature space of the style transfer network. StyleTune introduces a three-stage user interface, that enables users to first explore global stroke parametrizations for a chosen NST. They can then interactively locally retouch the stroke size and orientation using brush metaphors. Finally, high resolution outputs of 20 Megapixels and more can be obtained using a patch-based upsampling and local detail transfer approach, that transfers small-scale details such as paint-bristles and canvas structure. The app uses Apple’s CoreML and Metal APIs for efficient on-device processing.
Article
The mandala thangka, as a religious art in Tibetan Buddhism, is an invaluable cultural and artistic heritage. However, drawing a mandala pattern of thangka style is both time- and effort-consuming and requires mastery due to intricate details. Retaining and digitizing this heritage is an unresolved research challenge to date. In this paper, we propose a parametric approach to model and generate mandala thangka patterns to address this issue. Specifically, we construct parameterized models of three stylistic elements used in the interior mandalas of Nyingma school in Tibetan Buddhism according to their geometric features, namely the star, crescent, and lotus flower motifs. Varieties of interior mandala patterns are successfully generated using these parameterized motifs based on the hierarchical structures observed from hand-drawn mandalas. Moreover, we design a user interaction tool which can flexibly generate stylized mandala patterns with arbitrary shapes and colors. The experimental results show that our approach can efficiently generate beautifully-layered colorful traditional mandala patterns used in Buddhism and stylized mandala patterns used in modern art design, which significantly reduce the time and effort in manual production and, more importantly, contributes to the digitization of this great heritage.
Poster
Full-text available
This work presents an interactive mobile implementation of a filter that transforms images into an oil paint look. At this, a multi-scale approach that processes image pyramids is introduced that uses flow-based joint bilateral upsampling to achieve deliberate levels of abstraction at multiple scales and interactive frame rates. The approach facilitates the implementation of interactive tools that adjust the appearance of filtering effects at run-time, which is demonstrated by an on-screen painting interface for per-pixel parameterization that fosters the casual creativity of non-artists.
Chapter
During the early years of computer graphics, the results were arguably not as realistic as the intended goal set forth. However, it was not until sometime later that non-realism was accepted as a goal worthwhile pursuing. Since then, NPR has matured considerably and found uses in numerous applications, ranging from movies and television, production tools, and games, to novelty and casual creativity apps on mobile devices. This chapter presents examples from each of these categories within their historical and applied context.
NPR in the Wild. In Image and Video-Based Artistic Stylisation
  • H Winnemöller
WINNEMÖLLER, H. 2013. NPR in the Wild. In Image and Video-Based Artistic Stylisation. Springer, 353-374.
Interactive NPAR: What Type of Tools Should We Create?
  • T Isenberg
ISENBERG, T. 2016. Interactive NPAR: What Type of Tools Should We Create? In Proc. NPAR, 89-96.