Conference PaperPDF Available

Abstract and Figures

This work presents Pictory, a mobile app that empowers users to transform photos into artistic renditions by using a combination of neural style transfer with user-controlled state-of-the-art nonlinear image filtering. The combined approach features merits of both artistic rendering paradigms: deep convolutional neural networks can be used to transfer style characteristics at a global scale, while image filtering is able to simulate phenomena of artistic media at a local scale. Thereby, the proposed app implements an interactive two-stage process: first, style presets based on pre-trained feed-forward neural networks are applied using GPU-accelerated compute shaders to obtain initial results. Second, the intermediate output is stylized via oil paint, watercolor, or toon filtering to inject characteristics of traditional painting media such as pigment dispersion (watercolor) as well as soft color blendings (oil paint), and to filter artifacts such as fine-scale noise. Finally, on-screen painting facilitates pixel-precise creative control over the filtering stage, e. g., to vary the brush and color transfer, while joint bilateral upsampling enables outputs at full image resolution suited for printing on real canvas.
Content may be subject to copyright.
Pictory: Combining Neural Style Transfer and Image Filtering
Amir Semmo Matthias Trapp Jürgen Döllner
Hasso-Plattner-Institut,
Faculty of Digital Engineering,
University of Potsdam, Germany
{rst.last}@hpi.uni-potsdam.de
Mandy Klingbeil
Hasso-Plattner-Institut,
University of Potsdam, Germany
Digital Masterpieces GmbH, Germany
mandy.klingbeil@hpi.uni-potsdam.de
Content Image NST NST with Watercolor Filtering NST NST with Oil Paint Filtering
Style Image ‘Udnie’
Style Image ‘Mosaic’
Figure 1: Outputs of the mobile app Pictory that combines the results of a feed-forward NST [Johnson et al. 2016] with image
ltering to inject paint characteristics (here: watercolor, oil paint). Content image by Frank Köhntopp is in the public domain.
ABSTRACT
This work presents Pictory, a mobile app that empowers users to
transform photos into artistic renditions by using a combination of
neural style transfer with user-controlled state-of-the-art nonlinear
image ltering. The combined approach features merits of both
artistic rendering paradigms: deep convolutional neural networks
can be used to transfer style characteristics at a global scale, while
image ltering is able to simulate phenomena of artistic media
at a local scale. Thereby, the proposed app implements an inter-
active two-stage process: rst, style presets based on pre-trained
feed-forward neural networks are applied using GPU-accelerated
compute shaders to obtain initial results. Second, the intermediate
output is stylized via oil paint, watercolor, or toon ltering to in-
ject characteristics of traditional painting media such as pigment
dispersion (watercolor) as well as soft color blendings (oil paint),
and to lter artifacts such as ne-scale noise. Finally, on-screen
painting facilitates pixel-precise creative control over the ltering
stage, e. g., to vary the brush and color transfer, while joint bilat-
eral upsampling enables outputs at full image resolution suited for
printing on real canvas.
SIGGRAPH ’17 Appy Hour, Los Angeles, CA, USA
©
2017 Copyright held by the owner/author(s). This is the author’s version of the
work. It is posted here for your personal use. Not for redistribution. The denitive
Version of Record was published in Proceedings of SIGGRAPH ’17 Appy Hour, July 30 -
August 03, 2017, http://dx.doi.org/10.1145/3098900.3098906.
CCS CONCEPTS
Computing methodologies Non-photorealistic render-
ing;Image processing;
KEYWORDS
mobile, neural style transfer, image ltering, artistic rendering
ACM Reference format:
Amir Semmo, Matthias Trapp, Jürgen Döllner, and Mandy Klingbeil. 2017.
Pictory: Combining Neural Style Transfer and Image Filtering In Proceedings
of SIGGRAPH ’17 Appy Hour, Los Angeles, CA, USA, July 30 - August 03, 2017,
2 pages.
DOI: 10.1145/3098900.3098906
1 MOTIVATION
Image-based artistic rendering (IB-AR) enjoys a growing popular-
ity in mobile expressive rendering [Dev 2013; Winnemöller 2013]
to simulate the appeal of traditional artistic styles and media for
visual communication [Kyprianidis et al
.
2013; Rosin and Collo-
mosse 2013] such as oil paint, watercolor, and cartoon. Classical
IB-AR paradigms typically simulate their characteristics and phe-
nomena by a feature-level engineering approach, e. g., to locally
direct the smoothing and adjustment of image colors via ltering. A
more generalized approach has been introduced by the architecture
engineering approach of deep learning, which activates layers of
pre-trained deep convolutional neural networks (CNNs) to match
content and style statistics, and thus perform a neural style transfer
(NST) between arbitrary images [Gatys et al
.
2016]. While rst
applications demonstrate the practicability of NSTs by the example
of color and texture transfers as well as casual creativity apps (e.g.,
Prisma), local eects and phenomena of traditional artistic media
at high-delity and resolution are still hard to reproduce.
SIGGRAPH ’17 Appy Hour, July 30 - August 03, 2017, Los Angeles, CA, USA Semmo et al.
NST (FJBU as closeup) NST with Post-process Watercolor RenderingNST with Post-process Oil Paint Filtering
Content Image
Figure 2: Results produced for an input image with a resolution of
2
,
048
×
2
,
048
pixels. The low-resolution NST result (
512
×
512
pixels) is used with the high-resolution input for ow-based joint bilateral upsampling (FJBU). Afterward, post-process image
ltering is performed to locally inject paint characteristics. Content image by Redd Angelo is in the public domain.
We conjecture that NSTs may be used as one of multiple processing
stages and combined with the knowledge and algorithms of other
paradigms [Semmo et al
.
2017]. NSTs would thus operate as a rst
stage of image processing to introduce higher-level abstractions—to
be followed by low-level, established ltering techniques to simulate
drawing media and, e. g., their interplay with substrates (Figure 1).
2 TECHNICAL APPROACH
This work presents Pictory, a mobile app that combines NSTs with
image ltering. At this, the generative approach of Johnson et
al. [Johnson et al
.
2016] is combined with the image processing
framework of Semmo et al. [Semmo et al
.
2016] to implement in-
teractive ltering. Thereby, image abstraction at a global scale is
combined with local paint eects such as edge darkening, pigment
density variation, and wet-in-wet of watercolor [Bousseau et al
.
2006; Wang et al
.
2014], and smooth continuous oilpaint-like texture
eects via ow-based Gaussian ltering with Phong shading [Hertz-
mann 2002; Semmo et al
.
2016]. Figure 2 shows an output where the
abstract style of Pablo Picasso’s “La Muse” is used to generate an
eect of higher-level abstraction, before adding mentioned lters
to inject the respective low-level paint characteristics. Each of the
ltering eects can be locally parameterized by image masking,
e. g., over the color and texture transfer modality of the NST or the
lters’ parameters such as wetness, smoothness, and relief.
The mobile app was implemented using the OpenGL ES Shading
Language using compute shaders, and was deployed on Android.
To process images with full HD resolution, neural networks with re-
duced layers for the convolutional stages are applied in a tile-based
approach to optimize processing time and memory consumption.
In addition, ow-based joint bilateral upsampling [Kopf et al
.
2007;
Semmo et al
.
2016] of the low-resolution NST result is performed
with the high-resolution input to reduce visual noise and obtain
ne paint structures at the ltering stage (Figure 2). Using these op-
timizations, our app provides initial NST results between 2 seconds
(512
×
512 pixels) and 10 seconds (1024
×
1024 pixels), and enables
post-process ltering at interactive frame rates on a Google
Pixel
C with a NVIDIA®Maxwell 256 core GPU.
ACKNOWLEDGMENTS
We would like to thank Moritz Hilscher and Hendrik Tjabben for
their substantial contributions to the app prototype. This work was
funded by the Federal Ministry of Education and Research (BMBF),
Germany, for the AVA project 01IS15041B and within the InnoPro-
le Transfer research group “4DnD-Vis” (www.4dndvis.de).
REFERENCES
Adrien Bousseau, Matt Kaplan, Joëlle Thollot, and François X. Sillion. 2006. Interactive
Watercolor Rendering with Temporal Coherence and Abstraction. In Proc. NPAR.
ACM, New York, 141–149. doi: 10.1145/1124728. 1124751
Kapil Dev. 2013. Mobile Expressive Renderings: The State of the Art. IEEE Computer
Graphics and Applications 33, 3 (May/June 2013), 22–31. doi: 10. 1109/MCG.2013.20
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style Transfer
Using Convolutional Neural Networks. In Proc. CVPR. IEEE Computer Society, Los
Alamitos, 2414–2423. doi: 10.1109/CVPR. 2016.265
Aaron Hertzmann. 2002. Fast Paint Texture. In Proc. NPAR. ACM, New York, 91–96.
doi: 10.1145/508530. 508546
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual Losses for Real-Time
Style Transfer and Super-Resolution. In Proc. ECCV. Springer International, Cham,
Switzerland, 694–711. doi: 10.1007/978-3-319-46475-6_43
Johannes Kopf, Michael F. Cohen, Dani Lischinski, and Matt Uyttendaele. 2007. Joint
Bilateral Upsampling. ACM Transactions on Graphics 26, 3, Article 96 (July 2007).
doi: 10.1145/1276377. 1276497
Jan Eric Kyprianidis, John Collomosse, Tinghuai Wang, and Tobias Isenberg. 2013.
State of the “Art”: A Taxonomy of Artistic Stylization Techniques for Images and
Video. IEEE Transactions on Visualization and Computer Graphics 19, 5 (May 2013),
866–885. doi: 10.1109/TVCG.2012.160
Paul Rosin and John Collomosse (Eds.). 2013. Image and Video based Artistic Stylisation.
Computational Imaging and Vision, Vol. 42. Springer, London/Heidelberg. doi: 10.
1007/978-1-4471-4519-6
Amir Semmo, Tobias Dürschmid, Matthias Trapp, Mandy Klingbeil, Jürgen Döllner,
and Sebastian Pasewaldt. 2016. Interactive Image Filtering with Multiple Levels-of-
control on Mobile Devices. In Proc. MGIA. ACM, New York, Article 2, 8 pages. doi:
10.1145/2999508. 2999521
Amir Semmo, Tobias Isenberg, and Jürgen Döllner. 2017. Neural Style Transfer: A
Paradigm Shift for Image-based Artistic Rendering?. In Proc. NPAR. ACM, New
York. To appear.
Amir Semmo, Matthias Trapp, Tobias Dürschmid, Jürgen Döllner, and Sebastian Pase-
waldt. 2016. Interactive Multi-scale Oil Paint Filtering on Mobile Devices. In Proc.
ACM SIGGRAPH Posters. ACM, New York, 42:1–42:2. doi: 10. 1145/2945078.2945120
Miaoyi Wang, Bin Wang, Yun Fei, Kanglai Qian, Wenping Wang, Jiating Chen, and Jun-
Hai Yong. 2014. Towards Photo Watercolorization with Artistic Verisimilitude. IEEE
Transactions on Visualization and Computer Graphics 20, 10 (Feb. 2014), 1451–1460.
doi: 10.1109/TVCG. 2014.2303984
Holger Winnemöller. 2013. NPR in the Wild. In Image and Video based Artistic
Stylisation, Paul Rosin and John Collomosse (Eds.). Computational Imaging and
Vision, Vol. 42. Springer, Chapter 17, 353–374. doi: 10. 1007/978-1-4471-4519-6_17
... In recent studies, by introducing deep convolutional neural networks, the performance of computer vision task are dramatic improved. In the area of IBAR, the quality of synthesized images are also improved by using Neural Style Transfer (NST) methods [3][4][5][6] . WE are motivated to explore effective strategies to creating artistic images by using these NST methods. ...
... This phenomenon can be slightly solved by introducing new constrains, but the stroke texture of the synthesized image may still be distorted due to the neural style transfer process. A combined method are proposed by Klingbei and coworkers [6] to use classical method as post-process to reinforce the local effects of the Neural Style Transfer Stage. This method need direction field to generate the final results, Klingbei et al. generate the direction field from the results of the Neural Style Stage instead of the original direction field in content image. ...
... Semmo et al. [5] show that NST combined with Non-NST method may be an effective method. Klingbei and coworkers [6] are the first to explore the combination of NST method and classical IBAR method. But in their work, they only use the traditional NST method proposed by Gatys et al. [3] as their NST stage and a direction-aware filter based method as their post-process stage. ...
Article
Neural learning methods have been shown to be effective in style transfer. These methods, which are called NST, aim to synthesize a new image that retains the high-level structure of a content image while keeps the low-level features of a style image. However, these models using convolutional structures only extract local statistical features of style images and semantic features of content images. Since the ab- sence of low-level features in the content image, these methods would synthesize images that look un- natural and full of traces of machines. In this paper, we find that direction, that is, the orientation of each painting stroke, can capture the soul of image style preferably and thus generates much more natural and vivid stylizations. According to this observation, we propose a direction-aware neural style transfer with texture enhancement. There are four major innovations. First, we separate the style transfer method into two stage, namely, NST stage and texture enhancement stage. Second, for the NST stage, a novel direction field loss is proposed to steer the direction of strokes in the synthesized image. And to build this loss function, we propose novel direction field loss networks to generate and compare the direction fields of content image and synthesized image. By incorporating the direction field loss in neural style transfer, we obtain a new optimization objective. Through minimizing this objective, we can produce synthesized images that better follow the direction field of the content image. Third, our method provides a simple interaction mechanism to control the generated direction fields, and further control the texture direction in synthesized images. Fourth, with a texture enhancement module, our method can add vivid texture details to the synthesized image. Experiments show that our method outperforms state-of-the-art style transfer method.
... With the continuous advances in mobile hardware, many state-of-the-art image editing tools have been transferred to mobile platforms, raising new questions about how such applications should be designed (Isenberg, 2016). These tools include image recoloring, neural style transfer and filtering methods (Ryffel et al., 2017, Dürschmid et al., 2017, Reimann et al., 2019, Semmo et al., 2017. Ryffel et al. extended their soft segmentation approach to an augmented reality application, allowing users to virtually recolor a select number of paintings in a museum (Ryffel et al., 2017). ...
... Paintings were decomposed into layers in a preprocessing step which meant that at runtime, paintings could be recolored and displayed on the user's mobile device in real time. A recent neural style transfer approach proposed using a network with a reduced number of layers to reduce computation time, relying on upsampling to create high resolution results from the low resolution network output (Semmo et al., 2017). Other neural style transfer mobile applications consider the usability of the application, devising methods to increase the learnability of the application through user driven redesigns (Reimann et al., 2019) and allowing users to manipulate different style parameters and presets, and even share styles between users (Dürschmid et al., 2017). ...
Conference Paper
Full-text available
Multimedia software products can be used to create and edit various aspects of online media. Recently, the affordances of mobile devices and high-speed mobile data networks mean that these editing capabilities are more readily available for mobile devices enabling a broader consumer-base. However, the precise role of the user in creative practice is often neglected in favor of reporting faster, more streamlined device functionality. In this paper, we seek to identify high-level human-computer interaction issues concerning video recoloring interfaces that are driven by the needs of different user-types via a methodological and explorative process. By conducting a pilot study, we have captured both quantitative and qualitative responses that formatively explore the role of the user in video recoloring tasks carried out on mobile devices. This research presents a variety of user responses to a video recoloring application, identifying areas of future investigation for explorative practices in user interface design for video recoloring visualization. These findings present important information for researchers exploring the use of state-of-art video recoloring processes and contribute to dialogues surrounding the study of mobile technology in use.
... In recent years, deep convolutional neural networks have demonstrated dramatic improvements in performance for computer vision task. In the area of IBAR, Neural Style Transfer methods [1,5,19,20] have improved the transfer quality of Classical Image-based artistic rendering dramatically. ...
... Then they introduce a flow-guided anisotropic filtering for detecting highly coherent lines to produce a coherent line drawing effect. Amir et al. [18,20] propose a similar direction field extraction method based on adaptively smoothed structure tensor. They use direction field to generate paint texture that looks like directional painting strokes. ...
Conference Paper
Neural learning methods have been shown to be effective in style transfer. These methods, which are called NST, aim to synthesize a new image that retains the high-level structure of a content image while keeps the low-level features of a style image. However, these models using convolutional structures only extract local statistical features of style images and semantic features of content images. Since the absence of low-level features in the content image, these methods would synthesize images that look unnatural and full of traces of machines. In this paper, we find that direction, that is, the orientation of each painting stroke, can capture the soul of image style preferably and thus generates much more natural and vivid stylizations. According to this observation, we propose a Direction-aware Neural Style Transfer (DaNST) with two major innovations. First, a novel direction field loss is proposed to steer the direction of strokes in the synthesized image. And to build this loss function, we propose novel direction field loss networks to generate and compare the direction fields of content image and synthesized image. By incorporating the direction field loss in neural style transfer, we obtain a new optimization objective. Through minimizing this objective, we can produce synthesized images that better follow the direction field of the content image. Second, our method provides a simple interaction mechanism to control the generated direction fields, and further control the texture direction in synthesized images. Experiments show that our method outperforms state-of-the-art in most styles such as oil painting and mosaic.
... With the continuous development of mobile graphics hardware, interactive high-quality image and video processing, such as based on nonlinear filtering for oil paint stylization [45,47] and a MapReduce approach [22], is becoming feasible and thus of particular interest for industrial and educational purposes [52], e. g., when used for implementing casual creativity applications. At this, popular applications such as BeCasso [38,25] and Pictory [44,26] typically employ a user-centric approach for assisted image stylization targeting mobile artists and users seeking casual creativity, thereby integrating user experience concepts for making image filters usable in their daily life [16]. This is technically achieved by parameterizing image filters at three levels of control, i. e., using presets, global parameter adjustments and complementary on-screen painting that operates within the filters' parameter spaces for local adjustments [39]. ...
Thesis
Full-text available
With the improvement of cameras and smartphones, more and more people can now take high-resolution pictures. Especially in the field of advertising and marketing, images with extremely high resolution are needed, e. g., for high quality print results. Also, in other fields such as art or medicine, images with several billion pixels are created. Due to their size, such gigapixel images cannot be processed or displayed similar to conventional images. Processing methods for such images have high performance requirements. Especially for mobile devices, which are even more limited in screen size and memory than computers, processing such images is hardly possible. In this thesis, a service-based approach for processing gigapixel images is presented to approach this problem. Cloud-based processing using different microservices enables a hardware-independent way to process gigapixel images. Therefore, the concept and implementation of such an integration into an existing service-based architecture is presented. To enable the exploration of gigapixel images, the integration of a gigapixel image viewer into a web application is presented. Furthermore, the design and implementation will be evaluated with regard to advantages, limitations, and runtime.
... Under the help of these industrial applications [13], [99], [100], people can create their own art paintings and share their artwork with others on Twitter and Facebook, which is a new form of social communication. There are also some related application papers: [101] introduces an iOS app Pictory which combines style transfer techniques with image filtering; [102] further presents the technical implementation details of Pictory; [103] demonstrates the design of another GPU-based mobile app ProsumerFX. ...
Article
Full-text available
The seminal work of Gatys et al. demonstrated the power of Convolutional Neural Networks (CNNs) in creating artistic imagery by separating and recombining image content and style. This process of using CNNs to render a content image in different styles is referred to as Neural Style Transfer (NST). Since then, NST has become a trending topic both in academic literature and industrial applications. It is receiving increasing attention and a variety of approaches are proposed to either improve or extend the original NST algorithm. In this paper, we aim to provide a comprehensive overview of the current progress towards NST. We first propose a taxonomy of current algorithms in the field of NST. Then, we present several evaluation methods and compare different NST algorithms both qualitatively and quantitatively. The review concludes with a discussion of various applications of NST and open problems for future research. A list of papers discussed in this review, corresponding codes, pre-trained models and more comparison results are publicly available at: https://osf.io/f8tu4/.
... With the availability of public GPU-based frameworks for executing neural networks (e.g., CoreML), it is even feasible to apply mobile feed-forward NSTs in sub-seconds. Popular examples are Whisky16 and Pictory [36] to interactively apply styles to content images, and DeepStyleCam [40] to transfer multiple styles in real-time. ...
Article
Full-text available
Mobile expressive rendering gained increasing popularity among users seeking casual creativity by image stylization and supports the development of mobile artists as a new user group. In particular, neural style transfer has advanced as a core technology to emulate characteristics of manifold artistic styles. However, when it comes to creative expression, the technology still faces inherent limitations in providing low-level controls for localized image stylization. In this work, we first propose a problem characterization of interactive style transfer representing a trade-off between visual quality, run-time performance, and user control. We then present MaeSTrO, a mobile app for orchestration of neural style transfer techniques using iterative, multi-style generative and adaptive neural networks that can be locally controlled by on-screen painting metaphors. At this, we enhance state-of-the-art neural style transfer techniques by mask-based loss terms that can be interactively parameterized by a generalized user interface to facilitate a creative and localized editing process. We report on a usability study and an online survey that demonstrate the ability of our app to transfer styles at improved semantic plausibility.
... With the advent of publicly available GPUbased frameworks for execution of neural networks (e. g., CoreML), it is now even feasible to apply mobile feed-forward NSTs in sub-seconds. Besides Whisky16-using their own GPU processing framework-Pictory [32] was one of the first applications to interactively apply styles to content images. ...
Conference Paper
Full-text available
Mobile expressive rendering gained increasing popularity amongst users seeking casual creativity by image stylization and supports the development of mobile artists as a new user group. In particular, the neural style transfer has advanced as a core technology to emulate characteristics of manifold artistic styles and media without deep prior knowledge of photo processing or editing. However, when it comes to creative expression, the technology still faces inherent limitations in providing low-level controls for localized image stylization, e.g., with respect to image feature semantics or the user's ideas and interest. The goal of this work is to implement and enhance state-of-the-art neural style transfer techniques, providing a generalized user interface with interactive tools for local control that facilitate a creative editing process on mobile devices. At this, we first propose a problem characterization consisting of three goals that represent a trade-off between visual quality, run-time performance and ease of control. We then present MaeSTrO, a mobile app for orchestration of three neural style transfer techniques using iterative, multi-style generative and adaptive neural networks that can be locally controlled by on-screen painting metaphors to direct a semantics-based composition and perform location-based filtering. Based on first user tests, we conclude with insights, showing different levels of satisfaction for the implemented techniques and user interaction design, pointing out directions for future research.
... MaeSTrO implements three different neural network techniques, each providing a trade-off between usability and picture quality. Single-style feed-forward [Johnson et al. 2016] are currently used in the majority of techniques for mobile style transfer (e.g., [Semmo et al. 2017b]), since they enable nearly interactive performance, even on mobile devices. Once trained off-line, the feed-forward network-representing a single style-is globally applied to the whole input image. ...
Conference Paper
Full-text available
We present MaeSTrO, a mobile app for image stylization that empowers users to direct, edit and perform a neural style transfer with creative control. The app uses iterative style transfer, multi-style generative and adaptive networks to compute and apply flexible yet comprehensive style models of arbitrary images at run-time. Compared to other mobile applications, MaeSTrO introduces an interactive user interface that empowers users to orchestrate style transfers in a two-stage process for an individual visual expression: first, initial semantic segmentation of a style image can be complemented by on-screen painting to direct sub-styles in a spatially-aware manner. Second, semantic masks can be virtually drawn on top of a content image to adjust neural activations within local image regions, and thus direct the transfer of learned sub-styles. This way, the general feed-forward neural style transfer is evolved towards an interactive tool that is able to consider composition variables and mechanisms of general artwork production, such as color, size and location-based filtering. MaeSTrO additionally enables users to define new styles directly on a device and synthesize high-quality images based on prior segmentations via a servicebased implementation of compute-intensive iterative style transfer techniques.
... This paper presents ongoing developments in the design and implementation of Pictory, a mobile app that combines NSTs with other paradigms of image-based artistic rendering as proposed by Semmo et al. [2017a;2017b]. This combined approach enables Pictory to transform images into high-quality artistic renditions on a global and local scale. ...
Conference Paper
Full-text available
This work presents advances in the design and implementation of Pictory, an iOS app for artistic neural style transfer and interactive image editing using the CoreML and Metal APIs. Pictory combines the benefits of neural style transfer, e.g., high degree of abstraction on a global scale, with the interactivity of GPU-accelerated stateof-the-art image-based artistic rendering on a local scale. Thereby, the user is empowered to create high-resolution, abstracted renditions in a two-stage approach. First, a photo is transformed using a pre-trained convolutional neural network to obtain an intermediate stylized representation. Second, image-based artistic rendering techniques (e.g., watercolor, oil paint or toon filtering) are used to further stylize the image. Thereby, fine-scale texture noise—introduced by the style transfer—is filtered and interactive means are provided to individually adjust the stylization effects at run-time. Based on qualitative and quantitative user studies, Pictory has been redesigned and optimized to support casual users as well as mobile artists by providing effective, yet easy to understand, tools to facilitate image editing at multiple levels of control.
... Conversely, this can be achieved by mask-based parameter painting apps such as BeCasso [Pasewaldt et al. 2016;Semmo et al. 2016a] and PaintCan [Benedetti et al. 2014], where image filtering techniques that simulate watercolor, oil paint, and cartoon styles can be adjusted with a high, medium and low level-of-control [Isenberg 2016]. The combination of neural style transfer with post processing via image filtering can be done with Pictory [Semmo et al. 2017b]. Given these applications to reflect the state of the art in semiautomatic image and video stylization, however, mobile users are only able to consume effects. ...
Conference Paper
Full-text available
With the continuous advances of mobile graphics hardware, high-quality image stylization—e.g., based on image filtering, stroke-based rendering, and neural style transfer—is becoming feasible and increasingly used in casual creativity apps. The creative expression facilitated by these mobile apps, however, is typically limited with respect to the usage and application of pre-defined visual styles, which ultimately does not include their design and composition—an inherent requirement of prosumers. We present ProsumerFX, a GPU-based app that enables to interactively design parameterizable image stylization components on-device by reusing building blocks of image processing effects and pipelines. Furthermore, the presentation of the effects can be customized by modifying the icons, names, and order of parameters and presets. Thereby, the customized visual styles are defined as platform-independent effects and can be shared with other users via a web-based platform and database. Together with the presented mobile app, this system approach supports collaborative works for designing visual styles, including their rapid prototyping, A/B testing, publishing, and distribution. Thus, it satisfies the needs for creative expression of both professionals as well as the general public.
Conference Paper
Full-text available
In this meta paper we discuss image-based artistic rendering (IB-AR) based on neural style transfer (NST) and argue, while NST may represent a paradigm shift for IB-AR, that it also has to evolve as an interactive tool that considers the design aspects and mechanisms of artwork production. IB-AR received significant attention in the past decades for visual communication, covering a plethora of techniques to mimic the appeal of artistic media. Example-based rendering represents one the most promising paradigms in IB-AR to (semi-)automatically simulate artistic media with high fidelity, but so far has been limited because it relies on pre-defined image pairs for training or informs only low-level image features for texture transfers. Advancements in deep learning showed to alleviate these limitations by matching content and style statistics via activations of neural network layers, thus making a generalized style transfer practicable. We categorize style transfers within the taxonomy of IB-AR, then propose a semiotic structure to derive a technical research agenda for NSTs with respect to the grand challenges of NPAR. We finally discuss the potentials of NSTs, thereby identifying applications such as casual creativity and art production.
Conference Paper
Full-text available
With the continuous development of mobile graphics hardware, interactive high-quality image stylization based on nonlinear filtering is becoming feasible and increasingly used in casual creativity apps. However, these apps often only serve high-level controls to parameterize image filters and generally lack support for low-level (artistic) control, thus automating art creation rather than assisting it. This work presents a GPU-based framework that enables to parameterize image filters at three levels of control: (1) presets followed by (2) global parameter adjustments can be interactively refined by (3) complementary on-screen painting that operates within the filters' parameter spaces for local adjustments. The framework provides a modular XML-based effect scheme to effectively build complex image processing chains-using these interactive filters as building blocks-that can be efficiently processed on mobile devices. Thereby, global and local parameterizations are directed with higher-level algorithmic support to ease the interactive editing process, which is demonstrated by state-of-the-art stylization effects, such as oil paint filtering and watercolor rendering.
Poster
Full-text available
This work presents an interactive mobile implementation of a filter that transforms images into an oil paint look. At this, a multi-scale approach that processes image pyramids is introduced that uses flow-based joint bilateral upsampling to achieve deliberate levels of abstraction at multiple scales and interactive frame rates. The approach facilitates the implementation of interactive tools that adjust the appearance of filtering effects at run-time, which is demonstrated by an on-screen painting interface for per-pixel parameterization that fosters the casual creativity of non-artists.
Conference Paper
We consider image transformation problems, where an input image is transformed into an output image. Recent methods for such problems typically train feed-forward convolutional neural networks using a per-pixel loss between the output and ground-truth images. Parallel work has shown that high-quality images can be generated by defining and optimizing perceptual loss functions based on high-level features extracted from pretrained networks. We combine the benefits of both approaches, and propose the use of perceptual loss functions for training feed-forward networks for image transformation tasks. We show results on image style transfer, where a feed-forward network is trained to solve the optimization problem proposed by Gatys et al. in real-time. Compared to the optimization-based method, our network gives similar qualitative results but is three orders of magnitude faster. We also experiment with single-image super-resolution, where replacing a per-pixel loss with a perceptual loss gives visually pleasing results.
Chapter
During the early years of computer graphics, the results were arguably not as realistic as the intended goal set forth. However, it was not until sometime later that non-realism was accepted as a goal worthwhile pursuing. Since then, NPR has matured considerably and found uses in numerous applications, ranging from movies and television, production tools, and games, to novelty and casual creativity apps on mobile devices. This chapter presents examples from each of these categories within their historical and applied context.
Article
We consider image transformation problems, where an input image is transformed into an output image. Recent methods for such problems typically train feed-forward convolutional neural networks using a \emph{per-pixel} loss between the output and ground-truth images. Parallel work has shown that high-quality images can be generated by defining and optimizing \emph{perceptual} loss functions based on high-level features extracted from pretrained networks. We combine the benefits of both approaches, and propose the use of perceptual loss functions for training feed-forward networks for image transformation tasks. We show results on image style transfer, where a feed-forward network is trained to solve the optimization problem proposed by Gatys et al in real-time. Compared to the optimization-based method, our network gives similar qualitative results but is three orders of magnitude faster. We also experiment with single-image super-resolution, where replacing a per-pixel loss with a perceptual loss gives visually pleasing results .
Book
Non-photorealistic rendering (NPR) is a combination of computer graphics and computer vision that produces renderings in various artistic, expressive or stylized ways such as painting and drawing. This book focuses on image and video based NPR, where the input is a 2D photograph or a video rather than a 3D model. 2D NPR techniques have application in areas as diverse as consumer and professional digital photography and visual effects for TV and film production. The book covers the full range of the state of the art of NPR with every chapter authored by internationally renowned experts in the field, covering both classical and contemporary techniques. It will enable both graduate students in computer graphics, computer vision or image processing and professional developers alike to quickly become familiar with contemporary techniques, enabling them to apply 2D NPR algorithms in their own projects.
Article
We present a novel artistic-verisimilitude driven system for watercolor rendering of images and photos. Our system achieves realistic simulation of a set of important characteristics of watercolor paintings that have not been well implemented before. Specifically, we designed several image filters to achieve: 1) watercolor-specified color transferring; 2) saliency-based level-of-detail drawing; 3) hand tremor effect due to human neural noise; and 4) an artistically controlled wet-in-wet effect in the border regions of different wet pigments. A user study indicates that our method can produce watercolor results of artistic verisimilitude better than previous filter-based or physical-based methods. Furthermore, our algorithm is efficient and can easily be parallelized, making it suitable for interactive image watercolorization.
Article
Mobile applications are incorporating underlying platforms' pervasiveness in many innovative ways. Performance barriers due to resource constraints are slowly vanishing, and people are increasingly using mobile devices to perform many daily tasks they previously performed on desktop computers. Although a mobile platform's ability to handle graphics-related tasks requires further investigation, researchers have already made substantial progress. One particular related research area is nonphotorealistic rendering (NPR). NPR involves inherent abstraction, and mobile platforms offer relatively less computing power. So, a convergence of these areas can help deal with producing complex renderings on resource-constrained mobile platforms. This tutorial describes the state of NPR techniques for mobile devices, especially PDAs, tablets, and mobile phones, to motivate the development of efficient mobile NPR apps. In particular, the article addresses NPR advantages, challenges, and solutions. It also discusses mobile NPR visualizations, usability concerns, and future research directions.