Interactive Multi-scale Oil Paint Filtering on Mobile Devices *
Amir Semmo Matthias Trapp Tobias D¨
Hasso Plattner Institute, University of Potsdam, Germany∗
Digital Masterpieces GmbH∗
Figure 1: Results of the interactive multi-scale oil paint ﬁltering approach that processes image pyramids and uses ﬂow-based joint bilateral
upsampling (FJBU) with the input image (left). Scale factors: 100% / full resolution without FJBU (middle), 33% / with FJBU (right).
This work presents an interactive mobile implementation of a ﬁl-
ter that transforms images into an oil paint look. At this, a multi-
scale approach that processes image pyramids is introduced that
uses ﬂow-based joint bilateral upsampling to achieve deliberate lev-
els of abstraction at multiple scales and interactive frame rates. The
approach facilitates the implementation of interactive tools that ad-
just the appearance of ﬁltering effects at run-time, which is demon-
strated by an on-screen painting interface for per-pixel parameteri-
zation that fosters the casual creativity of non-artists.
Keywords: oil paint ﬁlter, ﬂow-based joint bilateral upsampling
Concepts: •Computing methodologies →Image manipulation;
1 Introduction and Motivation
Image stylization enjoys a growing popularity on mobile devices to
foster casual creativity [Winnem¨
oller 2013]. However, the imple-
mentation and provision of high-quality image effects for artistic
rendering is still faced by the inherent limitations of mobile graph-
ics hardware such as computing power and memory resources.
In particular with the continuous advancements of mobile cam-
era hardware, the interactive processing of high-resolution image
data becomes an increasingly challenging task. This especially con-
cerns image-based artistic rendering [Kyprianidis et al. 2013] that
requires several passes of (non-)linear ﬁltering. This work presents
answers to these challenges by the example of an interactive oil
paint ﬁlter. It demonstrates how complex nonlinear image ﬁlters
can be efﬁciently processed on mobile GPUs, while providing ﬁne-
grained controls for high-level and low-level run-time parameteri-
zation to support the visual expression of non-artists—a contempo-
rary ﬁeld of research of the NPR community [Isenberg 2016].
This is the authors’ version of the work. It is posted here for your personal
use. Not for redistribution. The deﬁnitive version will be published in Pro-
ceedings of the 43rd International Conference and Exhibition on Computer
Graphics & Interactive Techniques (SIGGRAPH ’16).
2016 Copyright held by the owner/author(s).
SIGGRAPH ’16, July 24-28, 2016, Anaheim, CA,
2 Technical Approach
The original oil paint ﬁlter requires wide kernels for Gaussian ﬁl-
tering (σ≈20) and leads to a high number of texture fetches to
achieve ﬁrm color blendings [Semmo et al. 2016]—a performance
limiting factor on mobile GPUs. Previous works typically employ
separated ﬁlter kernels to alleviate this problem, but do not ulti-
mately solve it for multi-stage and iterated nonlinear ﬁltering.
The proposed solution is based on a multi-scale approach that oper-
ates on image pyramids and uses joint bilateral upsampling [Kopf
et al. 2007] with the high-resolution input (Figure 1). At this, ﬂow-
based joint bilateral upsampling (FJBU) is proposed that uses the
smoothed structure—adapted to the main feature contours of the
ﬁltered low-resolution image—to produce a painterly look. The
FJBU uses a separable orientation-aligned implementation that ﬁl-
ters in the gradient direction and along the ﬂow curves induced
by the tangent ﬁeld. Together with real-time color grading us-
ing lookup tables, the enhancements enable interactive performance
when processing input images with full HD resolution, and thus al-
low interactive per-pixel parameterizations via on-screen painting.
The ﬁlter was implemented using the OpenGL ES Shading Lan-
guage and deployed on Android. For images with full HD resolu-
tion, it performs at 10 fps (scale factor 25%) and 6 fps (scale factor
50%) on a OnePlus Two with an Adreno 430 GPU.
Acknowledgments. This work was partly funded by the Federal
Ministry of Education and Research (BMBF), Germany, within the
InnoProﬁle Transfer research group “4DnD-Vis”.
ISENBERG, T. 2016. Interactive NPAR: What Type of Tools Should We Create? In
Proc. NPAR, The Eurographics Association, Goslar, Germany, 89–96.
KOPF, J ., COHE N, M . F., LISCHINSKI, D ., AND UY TT END AEL E, M . 2007. Joint
Bilateral Upsampling. ACM Trans. Graph. 26, 3.
KYPRIANIDIS, J. E. , CO LL OMO SS E, J. , WANG , T., AND ISENBERG, T. 2013. State
of the ’Art’: A Taxonomy of Artistic Stylization Techniques for Images and Video.
IEEE Trans. Vis. Comput. Graphics 19, 5, 866–885.
SEMM O, A ., LIMBERGER, D., K YPRIANIDIS, J. E ., AND D¨
OLL NE R, J. 2016. Image
Stylization by Interactive Oil Paint Filtering. Computers & Graphics 55, 157–171.
OLL ER , H . 2013. NPR in the Wild. In Image and Video-Based Artistic
Stylisation. Springer, 353–374.