Content uploaded by Matthias Trapp
Author content
All content in this area was uploaded by Matthias Trapp on Jan 07, 2019
Content may be subject to copyright.
Interactive GPU-Based Image Deformation
For Mobile Devices
Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner
Hasso Plattner Institute, University of Potsdam –16th September 2016 –CGVC2016
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 2
Outline
1. Introduction to image deformation
2. Previous work and deformation approaches
3. Challenges of interactive image deformation on mobile devices
4. Conceptual overview
5. Real-time implementation details
6. Performance results & discussion
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 3
Introduction to Image Deformation
Manipulation of an image
▪To correct distortions
▪For creative purposes, e.g.,
caricatures
Mobile devices
▪Built-in touch screen
▪Direct manipulation
➔Ideally suited for image
deformation
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 4
Challenges of Interactive Image Deformation
Maximizing visual quality
▪High-resolution proxy geometry required
➔High geometric complexity
➔High number of vertices to process
Mobile devices
▪Low computational power
▪Battery drain
➔Limits processable #vertices/time
Interactivity is a hard constraint
▪Limited time frame to process an operation
➔Limits geometric resolution
➔Limits visual quality
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 5
Previous Work: Deformation Approaches
Cage-based deformation [Meng et al. 2011]
▪Fitted control structure with handles
[Meng et al. 2011]
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 5
Previous Work: Deformation Approaches
Cage-based deformation [Meng et al. 2011]
▪Fitted control structure with handles
Mesh-based image deformation [Schaefer et al. 2006]
▪Image draped into displaced mesh
[Schaefer et al. 2006]
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 5
Previous Work: Deformation Approaches
Cage-based deformation [Meng et al. 2011]
▪Fitted control structure with handles
Mesh-based image deformation [Schaefer et al. 2006]
▪Image draped into displaced mesh
Camera textures [Spindler et al. 2006]
▪Texture-based representation of 3D deformations
[Spindler et al. 2006]
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 8
Mesh-Based Image Deformation
Mesh: proxy geometry textured with input image
Initial mesh
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 8
Mesh-Based Image Deformation
Mesh: proxy geometry textured with input image
1. Deform: modify vertex positions
1.
1
Initial mesh
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 8
Mesh-Based Image Deformation
Mesh: proxy geometry textured with input image
1. Deform: modify vertex positions
1.
1
Deformed mesh
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 8
Mesh-Based Image Deformation
Mesh: proxy geometry textured with input image
1. Deform: modify vertex positions
2. Apply: render textured mesh
1.
1
1.2
Deformed mesh
Input image
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 8
Mesh-Based Image Deformation
Mesh: proxy geometry textured with input image
1. Deform: modify vertex positions
2. Apply: render textured mesh
1.
1
1.2
Deformed mesh
Input image
Deformed image
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 13
Introducing the Intermediate Deformation Buffer
Apply more expensive than deform
Decouple geometric resolutions –Store deformation state separate from mesh
Increase maximum geometric resolution
➔Improve visual quality
Interactive editing
▪High-resolution deform
▪Low-resolution apply
High quality export
▪High-resolution deform
▪High-resolution apply
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 14
Introducing the Intermediate Deformation Buffer
Initial mesh
Input image
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 14
Introducing the Intermediate Deformation Buffer
Deformation buffer
Initial mesh
Input image
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 14
Introducing the Intermediate Deformation Buffer
Deformation buffer
1.
1
Initial mesh
Input image
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 14
Introducing the Intermediate Deformation Buffer
Modified
deformation buffer
1.
1
Initial mesh
Input image
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 14
Introducing the Intermediate Deformation Buffer
Modified
deformation buffer
1.
1
Initial mesh
Input image
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 14
Introducing the Intermediate Deformation Buffer
1.2a
Modified
deformation buffer
1.
1
Deformed mesh
Input image
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 14
Introducing the Intermediate Deformation Buffer
1.2a
Modified
deformation buffer
1.
11.2b
Deformed mesh Deformed image
Input image
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 14
Introducing the Intermediate Deformation Buffer
1.2a
Modified
deformation buffer
1.
11.2b
Deformed mesh Deformed image
Initial mesh
Input image
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 14
Introducing the Intermediate Deformation Buffer
1.2a
Modified
deformation buffer
1.
11.2b
Deformed mesh Deformed image
Initial mesh
Input image
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 14
Introducing the Intermediate Deformation Buffer
1.2a
12a
Modified
deformation buffer
1.
11.2b
Deformed mesh Deformed image
Deformed mesh
Input image
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 14
Introducing the Intermediate Deformation Buffer
1.2a
12a
Modified
deformation buffer
1.
11.2b
Deformed mesh Deformed image
12b
Deformed mesh
Input image
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 14
Introducing the Intermediate Deformation Buffer
1.2a
12a
Modified
deformation buffer
1.
11.2b
Deformed mesh Deformed image
12b
Deformed mesh Deformed image
Input image
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 14
Introducing the Intermediate Deformation Buffer
Interactive editing
1.2a
12a
Modified
deformation buffer
1.
11.2b
Deformed mesh Deformed image
12b
Deformed mesh Deformed image
Input image
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 14
Introducing the Intermediate Deformation Buffer
Interactive editing
High quality export
1.2a
12a
Modified
deformation buffer
1.
11.2b
Deformed mesh Deformed image
12b
Deformed mesh Deformed image
Input image
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 28
Deformation Buffer: Implementation
Deformation buffer
▪Encodes vertex displacement
▪Stored in video memory
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 28
Deformation Buffer: Implementation
Deformation buffer
▪Encodes vertex displacement
▪Stored in video memory
Pass 1: Deform
▪Compute shaders modify buffer in-place
▪Invocation per element computes new position
Deform
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 28
Deformation Buffer: Implementation
Deformation buffer
▪Encodes vertex displacement
▪Stored in video memory
Pass 1: Deform
▪Compute shaders modify buffer in-place
▪Invocation per element computes new position
Pass 2: Apply
▪Render textured mesh
▪Apply vertex displacement during rendering
Deform
Apply
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 31
Implementation Details: Compute Shader
Preferred variant: compute shader
▪Lower memory consumption
▪Greater flexibility
Shader invocation per buffer element
▪Operation parameters passed as uniform
variables
▪Deformation buffer bound as shader
storage buffer object
▪Directly modify buffer content
#version 310 es
precision mediump float;
layout(local_size_x = 64, local_size_y = 1) in;
uniform vec2 u_Origin;
uniform vec2 u_Direction;
uniform float u_Radius;
layout(std430) buffer DeformationBuffer {
vec2 positions[];
};
void main() {
// check invocation range
int index = int(gl_GlobalInvocationID.x);
if (index >= positions.length())
return;
// translate vertex
float dist = distance(positions[index], u_Origin);
if (dist < u_Radius) {
float impact = smoothstep(0.0, u_Radius, dist);
positions[index] += u_Direction * impact;
}
}
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 32
Implementation Details: Texture Storage Precision
Artifacts for 8 and 16 bit precision ➔32 bit precision required, however
▪Limited support for 32-bit textures on mobile devices
▪Read back only for 8 bit precision (required for undo & export)
➔Cannot use texture storage
8 bit storage precision 16 bit storage precision 32 bit storage precision
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 33
Deformation Application: Implementation
Multi-resolution mesh based on a single vertex buffer
▪Index buffers with different strides
Stride = 1 Stride = 2 Stride = 4
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 34
Template-Based Deformation Operations
Fat finger problem: difficult to achieve fine details with touch gestures [Siek et al. 2005]
➔Incorporate user-defined deformation templates created with standard drawing software
Explicit templates apply complete deformation
▪Replace deformation buffer content
Implicit templates define shapes
▪Encode operation impact factor
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 35
Discussion: Performance Evaluation
129 257 513 1025 2049
Apply 3,72 4,72 6,06 11,88 26,67
Deform 10,95 10,95 10,95 10,90 11,20
0
5
10
15
20
25
30
35
40
Runtime [ms]
Mesh Resolution
Constant buffer resolution
(2049x2049)
▪Constant “deform” runtime
Varying mesh resolution
▪Increasing “apply” runtime
Confirms assumption:
Two modes of operations for
maximum quality
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 36
Summary
Mesh-based image deformation with
▪Intermediate deformation buffer to store vertex displacement
▪Multi-resolution mesh for interactive editing and high-quality export
▪Deformation templates for fine details
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 37
References
MENG W., ZHANG X., DONG W., PAUL J.-C.: Multicage Image Deformation On GPU (2011)
SCHAEFER S., MCPHAIL T., WARREN J.: Image Deformation Using Moving Least Squares (2006)
SPINDLER M., BUBKE M., GERMER T., STROTHOTTE T.: Camera Textures (2006)
SIEK, K. A., ROGERS Y., CONNELLY K. H.: Fat Finger Worries: How Older and Younger Users Physically
Interact with PDAs (2005)
16th Sep 2016 Jan Ole Vollmer, Matthias Trapp, Jürgen Döllner –Interactive GPU-Based Image Deformation for Mobile Devices 38
Thank you!
Contact:
www.hpi3d.de
jan.vollmer@hpi.de
This work was partially funded by the Federal Ministry of Education and Research
(BMBF), Germany within the InnoProfile Transfer research groups "4DnD-Vis”.