Kenny Mitchell

Kenny Mitchell
Edinburgh Napier University · School of Computing

Professor, PhD, MSc, BSc (Hons)

About

88
Publications
14,691
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
1,023
Citations
Introduction
Professor Kenny Mitchell is a Technical Director of Rendering at Roblox Corporation and chair of Video Game Technology at Edinburgh Napier University providing practical technology solutions for video games, Disney theme parks, movie production, and immersive media. Over the past 20 years made video games with ground breaking high-end graphics Harry Potter, Star Wars, etc. He assists as CTO for 3Finery Ltd and Cobra Simulation Ltd. He is co-Editor-in-Chief of ACM Games Journal.
Additional affiliations
December 2020 - present
Roblox
Position
  • Managing Director
April 2017 - December 2020
Disney Research
Position
  • Senior Researcher
August 2013 - present
Edinburgh Napier University
Position
  • Professor
Description
  • http://games.soc.napier.ac.uk

Publications

Publications (88)
Article
Anamorphosis for 2D displays can provide viewer centric perspective viewing, enabling 3D appearance, eye contact and engagement, by adapting dynamically in real time to a single moving viewer’s viewpoint, but at the cost of distorted viewing for other viewers. We present a method for constructing non-linear projections as a combination of anamorphi...
Article
Full-text available
We present a practical neural computational approach for interactive design of Audio-Animatronic® facial performances. An offline quasi-static reference simulation, driven by a coupled mechanical assembly, accurately predicts hyperelastic skin deformations. To achieve interactive digital pose design, we train a shallow, fully connected neural netwo...
Conference Paper
Preparing datasets for use in the training of real-time face tracking algorithms for HMDs is costly. Manually annotated facial landmarks are accessible for regular photography datasets, but introspectively mounted cameras for VR face tracking have incompatible requirements with these existing datasets. Such requirements include operating ergonomica...
Chapter
Full-text available
JUNGLE is an interactive, visual platform for the collaborative manipulation and consumption of nonlinear transmedia stories. Intuitive visual interfaces encourage JUNGLE users to explore vast libraries of story worlds, expand existing stories, or conceive of entirely original story worlds. JUNGLE stories utilize multiple media forms including vide...
Article
Full-text available
We introduce Intermediated Reality (IR), a framework for intermediated communication enabling collaboration through remote possession of entities (e.g., toys) that come to life in mobile Mediated Reality (MR). As part of a two-way conversation, each person communicates through a toy figurine that is remotely located in front of the other participan...
Article
We propose, DeepPRT, a deep convolutional neural network to compactly encapsulate the radiance transfer of a freely deformable object for rasterization in real-time. With pre-computation of radiance transfer (PRT) we can store complex light interactions appropriate to the shape of a given object at each surface point for subsequent real-time render...
Article
Full-text available
Shadow-retargeting maps depict the appearance of real shadows to virtual shadows given corresponding deformation of scene geometry, such that appearance is seamlessly maintained. By performing virtual shadow reconstruction from unoccluded real-shadow samples observed in the camera frame, this method efficiently recovers deformed shadow appearance....
Preprint
Full-text available
We present a single-image 3D face synthesis technique that can handle challenging facial expressions while recovering fine geometric details. Our technique employs expression analysis for proxy face geometry generation and combines supervised and unsupervised learning for facial detail synthesis. On proxy generation, we conduct emotion prediction t...
Conference Paper
Full-text available
Dense 3D face reconstruction plays a fundamental role in visual media production involving digital actors. We improve upon high fidelity reconstruction from a single 2D photo with a reconstruction framework that is robust to large variations in expressions, poses and illumination. We provide a global optimization step improving the alignment of 3D...
Conference Paper
Interactive play can take very different forms, from playing with physical board games to fully digital video games. In recent years, new video game paradigms were introduced to connect real-world objects to virtual game characters. However, even these applications focus on a specific section of the Reality-Virtuality Continuum, where the visual em...
Article
Pre-calculated depth information is essential for efficient light field video rendering, due to the prohibitive cost of depth estimation from color when real-time performance is desired. Standard state-of-the-art video codecs fail to satisfy such performance requirements when the amount of data to be decoded becomes too large. In this paper, we pro...
Article
We present several mixed‐reality‐based remote collaboration settings by using consumer head‐mounted displays. We investigated how two people are able to work together in these settings. We found that the person in the AR system will be regarded as the “leader” (i.e., they provide a greater contribution to the collaboration), whereas no similar “lea...
Article
Image‐based lighting has allowed the creation of photo‐realistic computer‐generated content. However, it requires the accurate capture of the illumination conditions, a task neither easy nor intuitive, especially to the average digital photography enthusiast. This paper presents an approach to directly estimate an HDR light probe from a single LDR...
Article
We propose an end-to-end solution for presenting movie quality animated graphics to the user while still allowing the sense of presence afforded by free viewpoint head motion. By transforming offline rendered movie content into a novel immersive representation, we display the content in real-time according to the tracked head pose. For each frame,...
Conference Paper
Lightfield video, as a high-dimensional function, is very demanding in terms of storage. As such, lightfield video data, even in a compressed form, do not typically fit in GPU or main memory unless the capture area, resolution or duration is sufficiently small. Additionally, latency minimization--critical for viewer comfort in use-cases such as vir...
Conference Paper
Full-text available
We present immersive storytelling in VR enhanced with non-linear sequenced sound, touch and light. Our Deep Media [Rose 2012] aim is to allow for guests to physically enter rendered movies with novel non-linear storytelling capability. With the ability to change the outcome of the story through touch and physical movement, we enable the agency of g...
Article
Full-text available
We propose a novel pre-filtering method that reduces the noise introduced by depth-of-field and motion blur effects in geometric buffers (G-buffers) such as texture, normal and depth images. Our pre-filtering uses world positions and their variances to effectively remove high-frequency noise while carefully preserving high-frequency edges in the G-...
Conference Paper
Full-text available
We propose an end-to-end solution for presenting movie quality animated graphics to the user while still allowing the sense of presence afforded by free viewpoint head motion. By transforming offline rendered movie content into a novel immersive representation, we display the content in real-time according to the tracked head pose. For each frame,...
Article
We present a real-time multi-view facial capture system facilitated by synthetic training imagery. Our method is able to achieve high-quality markerless facial performance capture in real-time from multi-view helmet camera data, employing an actor specific regressor. The regressor training is tailored to specified actor appearance and we further co...
Conference Paper
We present a system for rapid acquisition of bespoke, animatable, full-body avatars including face texture and shape. A blendshape rig with a skeleton is used as a template for customization. Identity blendshapes are used to customize the body and face shape at the fitting stage, while animation blendshapes allow the face to be animated. The subjec...
Article
Over the past three decades, the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D) has showcased exceptional progress from academic and industrial research covering all aspects of interactive computer graphics.
Conference Paper
We propose a framework for real-time tracking of humans using sparse multi-modal sensor sets, including data obtained from optical markers and inertial measurement units. A small number of sensors leaves the performer unencumbered by not requiring dense coverage of the body. An inverse dynamics solver and physics-based body model are used, ensuring...
Article
We propose a new real-time temporal filtering and antialiasing (AA) method for rasterization graphics pipelines. Our method is based on Pixel History Linear Models (PHLM), a new concept for modeling the history of pixel shading values over time using linear models. Based on PHLM, our method can predict per-pixel variations of the shading function b...
Conference Paper
Real-time facial performance capture has recently been gaining popularity in virtual film production, driven by advances in machine learning, which allows for fast inference of facial geometry from video streams. These learning-based approaches are significantly influenced by the quality and amount of labelled training data. Tedious construction of...
Article
Full-text available
We present a method of adding sophisticated physical simulations to voxel-based games such as the hugely popular Minecraft (2012. http://minecraft.gamepedia.com/Liquid), thus providing a dynamic and realistic fluid simulation in a voxel environment. An assessment of existing simulators and voxel engines is investigated, and an efficient real-time m...
Conference Paper
Full-text available
With a recent rise in the availability of affordable head mounted gear sets, various sensory stimulations (e.g., visual, auditory and haptics) are integrated to provide seamlessly embodied virtual experience in areas such as education, entertainment, therapy and social interactions. Currently, there is an abundance of available toolkits and applica...
Conference Paper
Compelling virtual reality experiences require high quality imagery as well as head motion with six degrees of freedom. Most existing systems limit the motion of the viewer (prerecorded fixed position 360 video panoramas), or are limited in realism, e.g. video game quality graphics rendered in real-time on low powered devices. We propose a solution...
Conference Paper
Perceptually lossless foveated rendering methods exploit human perception by selectively rendering at different quality levels based on eye gaze (at a lower computational cost) while still maintaining the user's perception of a full quality render. We consider three foveated rendering methods and propose practical rules of thumb for each method to...
Article
In this paper, we propose a new adaptive rendering method to improve the performance of Monte Carlo ray tracing, by reducing noise contained in rendered images while preserving high-frequency edges. Our method locally approximates an image with polynomial functions and the optimal order of each polynomial function is estimated so that our reconstru...
Article
We address the problem of denoising Monte Carlo renderings by studying existing approaches and proposing a new algorithm that yields state-of-the-art performance on a wide range of scenes. We analyze existing approaches from a theoretical and empirical point of view, relating the strengths and limitations of their corresponding components with an e...
Conference Paper
We contribute a foveated rendering implementation in Unreal Engine 4 (UE4) and a straight-forward metric to allow calculation of rendered foveal region sizes to compensate for overall system latency and maintain perceptual losslessness. Our system demonstrates appreciable computational gains in large resolution real-time rendering scenarios.
Conference Paper
Parameterisation of models is typically generated for a single pose, the rest pose. When a model deforms, its parameterisation characteristics change, leading to distortions in the appearance of texture-mapped mesostructure. Such distortions are undesirable when the represented surface detail is heterogeneous in terms of elasticity (e.g. texture wi...
Article
In this paper we present a novel approach to author vegetation cover of large natural scenes. Unlike stochastic scatter-instancing tools for plant placement (such as multi-class blue noise generators), we use a simulation based on ecological processes to produce layouts of plant distributions. In contrast to previous work on ecosystem simulation, h...
Conference Paper
We propose a type of relationship descriptor based on carpet unrolling that computes the joint positions of a character based on the sum of relative vectors originating from a local coordinate system embedded on the surface of a carpet. Given a terrain that a character is to walk over, the carpet is unrolled over the surface of the terrain. The car...
Conference Paper
Augmented Reality (AR) holds unique and promising potential to bridge between real-world activities and digital experiences, allowing users to engage their imagination and boost their creativity. We propose the concept of Augmented Creativity as employing ar on modern mobile devices to enhance real-world creative activities, support education, and...
Article
Full-text available
We propose a new adaptive rendering algorithm that enhances the performance of Monte Carlo ray tracing by reducing the noise, i.e., variance, while preserving a variety of high-frequency edges in rendered images through a novel prediction based reconstruction. To achieve our goal, we iteratively build multiple, but sparse linear models. Each linear...
Conference Paper
We present efficient rendering of opaque, sparse, voxel environments with data amplified in local graphics memory with streamout from a geomery shader to a cached vertex buffer pool. We show that our Poxel rendering primitive aligns with optimized rasterization hardware and so results in high visual quality over ray casting methods. Lossless run le...
Conference Paper
We present a low-cost solution for yaw drift in head-mounted display systems that performs better than current commercial solutions and provides a wide capture area for pose tracking. Our method applies an extended Kalman filter to combine marker tracking data from an overhead camera with onboard head-mounted display accelerometer readings. To achi...
Conference Paper
Authoring virtual terrains can be a challenging task. Procedural and stochastic methods for automated terrain generation produce plausible results but lack intuitive control of the terrain features, while data driven methods offer more creative control at the cost of a limited feature set, higher storage requirements and blending artefacts. Moreove...
Article
We investigate the influence of motion effects in the domain of mobile Augmented Reality (AR) games on user experience and task performance. The work focuses on evaluating responses to a selection of synthesized camera oriented reality mixing techniques for AR, such as motion blur, defocus blur, latency and lighting responsiveness. In our cross sec...
Patent
Full-text available
An augmented reality (AR) audio system for augmenting environment or ambient sound with sounds from a virtual speaker or sound source positioned at a location in the space surrounding an AR participant. The sound from the virtual speaker may be triggered by an action of the listener and/or by the location or relative orientation of the listener. Th...
Article
We present a theoretical analysis of error of combinations of Monte Carlo estimators used in image synthesis. Importance sampling and multiple importance sampling are popular variance-reduction strategies. Unfortunately, neither strategy improves the rate of convergence of Monte Carlo integration. Jittered sampling (a type of stratified sampling),...
Patent
Full-text available
Techniques are disclosed for performing image space reprojection iteratively. An insignificant parallax threshold depth is computed for a source image. Portions of the image having depth values greater than the insignificant parallax threshold depth may be shifted uniformly to produce corresponding portions of the reprojection (target) image. An it...
Conference Paper
Full-text available
The ubiquity of mobile devices with powerful processors and integrated video cameras is re-opening the discussion on practical augmented reality (AR). Despite this technological convergence, several issues prevent reliable and immersive AR on these platforms. We address one such problem, the shading of virtual objects and determination of lighting...
Conference Paper
Motion blur effects are commonly used in racing games [Sousa 2008; Vlachos 2008; Ritchie et al. 2010] to add a sense of realism as well as to minimize artifacts due to strobing and temporal aliasing [Glassner 1999]. Typically, motion blur computations are expensive, and for real-time applications, trade-offs are made between the quality of the effe...
Conference Paper
Animation of models often introduces distortions to their parameterisation, as these are typically optimised for a single frame. The net effect is that under deformation, the mapped features, i.e. UV texture maps, bump maps or displacement maps, may appear to stretch or scale in an undesirable way. Ideally, what we would like is for the appearance...
Article
At each shade point, the spherical visibility function encodes occlusion from surrounding geometry, in all directions. Computing this function is difficult and point-sampling approaches, such as ray-tracing or hardware shadow mapping, are traditionally used to efficiently approximate it. We propose a semi-analytic solution to the problem where the...
Conference Paper
We investigate the advantages of a stereo, multi-spectral acquisition system for material classification in ground-level landscape images. Our novel system allows us to acquire high-resolution, multi-spectral stereo pairs using commodity photographic equipment. Given additional spectral information we obtain better classification of vegetation clas...
Conference Paper
Augmented reality applications on hand-held devices suffer from the limited available processing power. While methods to detect the location of artificially textured markers within the scene are commonly used, geometric properties of three-dimensional objects are rarely exploited for object tracking. In order to track such geometry efficiently on m...
Article
Animated image sequences often exhibit a large amount of inter-frame coherence which standard rendering algorithms and pipelines are ill-equipped to exploit, limiting their efficiency. To address this inefficiency we transfer rendering results across frames using a novel image warping algorithm based on fixed point iteration. We analyze the behavio...
Conference Paper
Many rendering algorithms willingly sacrifice accuracy, favoring plausible shading with high-performance. Modular Radiance Transfer (MRT) models coarse-scale, distant indirect lighting effects in scene geometry that scales from high-end GPUs to low-end mobile platforms. MRT eliminates scene-dependent precomputation by storing compact transport on s...
Conference Paper
This paper presents a controller for camera convergence and interaxial separation that specifically addresses challenges in interactive stereoscopic applications like games. In such applications, unpredictable viewer- or object-motion often compromises stereopsis due to excessive binocular disparities. We derive constraints on the camera separation...
Article
Full-text available
This paper presents a controller for camera convergence and interaxial separation that specifically addresses challenges in interactive stereoscopic applications like games. In such applications, unpredictable viewer- or object-motion often compromises stereopsis due to excessive binocular disparities. We derive constraints on the camera separation...
Article
Many rendering algorithms willingly sacrifice accuracy, favoring plausible shading with high-performance. Modular Radiance Transfer (MRT) models coarse-scale, distant indirect lighting effects in scene geometry that scales from high-end GPUs to low-end mobile platforms. MRT eliminates scene-dependent precomputation by storing compact transport on s...
Conference Paper
Many rendering algorithms willingly sacrifice accuracy, favoring plausible shading with high-performance. Modular Radiance Transfer (MRT) models coarse-scale, distant indirect lighting effects in scene geometry that scales from high-end GPUs to low-end mobile platforms. MRT eliminates scene-dependent precomputation by storing compact transport on s...
Conference Paper
Full-text available
Integrating animated virtual objects with their surroundings for high-quality augmented reality requires both geometric and radio-metric consistency. We focus on the latter of these problems and present an approach that captures and factorizes external lighting in a manner that allows for realistic relighting of both animated and static virtual obj...
Article
This article presents a flexible, extendable system called Tracktivity that can capture gameplay metrics in any type of leaderboard-based video game. This system incorporates novel visualizations, including a dynamic competition balancing (DCB) measure and driver progression, to enhance the resulting gameplay experience. This article also presents...
Conference Paper
Surround Haptics is a new tactile technology that uses a low-resolution grid of inexpensive vibrating actuators to generate high-resolution, continuous, moving tactile strokes on human skin [1]. The user would not feel the discrete tactile pulses and buzzes that are so common today, but rather a smooth tactile motion, akin to what we feel when some...
Conference Paper
Full-text available
Real-time rendering of indirect lighting significantly enhances the sense of realism in video games. Unfortunately, previously including such effects often required time consuming scene dependent precomputation and heavy runtime computations unsuitable for low-end devices, such as mobile phones or game consoles. Modular Radiance Transfer (MRT) [Loo...
Conference Paper