Ken Perlin

New York University | NYU · Department of Computer Science

This pilot study aimed to identify postural strategies in response to sensory perturbations (visual, auditory, somatosensory) in adults with and without sensory loss. We tested people with unilateral peripheral vestibular hypofunction (N = 12, mean age 62 range 23-78), or with Unilateral Sensorineural Hearing Loss (USNHL, N = 9, 48, 22-82), or heal...

When creating 3D content, highly specialized skills are generally needed to design and generate models of objects and other assets by hand. We address this problem through high-quality 3D asset retrieval from multi-modal inputs, including 2D sketches, images and text. We use CLIP as it provides a bridge to higher-level latent features. We use these...

How do we design effective immersive VR experiences? Looking Inside Cells is a set of collaborative immersive virtual reality science learning simulations that were designed by applying best practices for learning experience design, taking advantage of the unique affordances of VR for learning. In this paper we describe design features of immersive...

Exploring large virtual environments, such as cities, is a central task in several domains, such as gaming and urban planning. VR systems can greatly help this task by providing an immersive experience; however, a common issue with viewing and navigating a city in the traditional sense is that users can either obtain a local or a global view, but n...

Video conferences play a vital role in our daily lives. However, many nonverbal cues are missing, including gaze and spatial information. We introduce LookAtChat, a web-based video conferencing system, which empowers remote users to identify gaze awareness and spatial relationships in small-group conversations. Leveraging real-time eye-tracking tec...

Writing or sketching on whiteboards is an essential part of collaborative discussions in business meetings, reading groups, design sessions, and interviews. However, prior work in collaborative virtual reality (VR) systems has rarely explored the design space of multi-user layouts and interaction modes with virtual whiteboards. In this paper, we pr...

We developed a novel assessment platform with untethered virtual reality, 3-dimensional sounds, and pressure sensing floor mat to help assess the walking balance and negotiation of obstacles given diverse sensory load and/or cognitive load. The platform provides an immersive 3D city-like scene with anticipated/unanticipated virtual obstacles. Parti...

Despite various collaborative software that supports expressing ideas, people still largely prefer physical notebooks or whiteboards. The reason is that they provide free-form expressions, co-presence of all participants and easy collaboration. However, when working with remote participants, people often choose the convenience of video conferencing...

Virtual Reality (VR) enables users to collaborate while exploring scenarios not realizable in the physical world. We propose CollabVR, a distributed multi-user collaboration environment, to explore how digital content improves expression and understanding of ideas among groups. To achieve this, we designed and examined three possible configurations...

Immersive co-located theatre aims to bring the social aspects of traditional cinematic and theatrical experience into Virtual Reality (VR). Within these VR environments, participants can see and hear each other, while their virtual seating location corresponds to their actual position in the physical space. These elements create a realistic sense o...

The virtual reality ecosystem has gained momentum in the gaming, entertainment, and enterprise markets, but is hampered by limitations in concurrent user count, throughput, and accessibility to mass audiences. Based on our analysis of the current state of the virtual reality ecosystem and relevant aspects of traditional media, we propose a set of d...

As synthetic imagery is used more frequently in training deep models, it is important to understand how different synthesis techniques impact the performance of such models. In this work, we perform a thorough evaluation of the effectiveness of several different synthesis techniques and their impact on the complexity of classifier domain adaptation...

CAVE is a shared narrative six degrees of freedom (6DoF) virtual reality experience. In 3.5 days, 1,927 people attended its premiere at SIGGRAPH 2018. Thirty participants at a time each saw and heard the same narrative from their own individual location in the room, as they would when attending live theater. CAVE set out to disruptively change how...

Mary and the Monster: Chapter One is a shared AR experience that brings to life "Frankenstein" through the eyes of a young Mary Shelley. Using Parallux technology and the Magic Leap One, audiences are immersed in the author's world as she conjures the most famous gothic tale of all time.

CAVE is a shared narrative six degrees of freedom (6DoF) virtual reality experience. In 3.5 days, 1,927 people attended its premiere at SIGGRAPH 2018. Thirty participants at a time each saw and heard the same narrative from their own individual location in the room, as they would when attending live theater. CAVE set out to disruptively change how...

Everyday mobile usage of AR and VR Head-Mounted Displays (HMDs) is becoming a feasible consumer reality. The current research agenda for HMDs has a strong focus on technological impediments (e.g. latency, field of view, locomotion, tracking, input) as well as perceptual aspect (e.g. distance compression, vergence-accomodation ). However, this ignor...

Mixed Reality (MR) enables users to explore scenarios not realizable in the physical world. This allows users to communicate with the help of digital content. We investigate how different configurations of participants and content affect communication in a shared immersive environment. We designed and implemented side-by-side, mirrored face-to-face...

Virtual Reality enables users to explore content whose physics are only limited by our creativity. Such limitless environments provide us with many opportunities to explore innovative ways to support productivity and collaboration. We present Spacetime, a scene editing tool built from the ground up to explore the novel interaction techniques that e...

Following their introduction in the 1960s, head-mounted VR systems mainly focused on visual and aural senses. In order to enhance immersion in the virtual world, researchers have since pursued the addition of movement and haptics through motion platforms, exoskeletons, and other hand-held devices. From a proliferation of low-cost devices that can s...

Real-time simulation of fluid and smoke is a long standing problem in computer graphics. Despite significant progress in the literature, state-of-the-art approaches require large compute resources, making real-time applications impractical. In this work, we propose a data-driven approach that leverages the approximation power of deep-learning metho...

We present a novel method for real-time continuous pose recovery of markerless complex articulable objects from a single depth image. Our method consists of the following stages: a randomized decision forest classifier for image segmentation, a robust method for labeled dataset generation, a convolutional network for dense feature extraction, and f...

Paper submitted for presentation at the Annual Meeting of the American Educational Research Association (AERA), New Orleans, LA (April, 2011). Results are presented from an experimental investigation of the effect of mode of play on motivational and educational outcomes. A computer-based arithmetic game was adapted to allow for solo, competitive, o...

A computer-based geometry game was adapted to allow for play using a conceptual rule or an arithmetic problem-solving mechanic. Participants (n = 91) from an urban middle school were randomly assigned to experimental conditions. Results suggest that play in the number condition was more situationally interesting than play in the rule condition. Par...

What will the interface between people and computers look like in five years? In ten years? In twenty five years? Will we still have screens? Keyboards? Will we all be seeing Princess Leia in a beam of light? Based on current trends and inspired guesswork, we will go together on a tour of the future.

In this paper, we will discuss the approach for assessment of learning and related learner variables taken by the Games for Learning Institute (G4LI). We will first describe game mechanics in general, and then introduce the concepts of learning mechanics and assessment mechanics and describe criteria for their design and requirements of how they ca...

Procedural noise functions are widely used in computer graphics, from off-line rendering in movie production to interactive video games. The ability to add complex and intricate details at low memory and authoring cost is one of its main attractions. This survey is motivated by the inherent importance of noise in graphics, the widespread use of noi...

Humans express their emotions in many ways, in particular through face, eye, and body motion. So creators of virtual humans strive to convincingly depict emotional movements using a variety of methods.This course focuses on the use of realistic human body motion to generate emotional expressiveness. Topics include: applications and research relatin...

Recently, there has been great interest in multi-touch interfaces. Such devices have taken the form of camera-based systems such as Microsoft Surface [de los Reyes et al. 2007] and Perceptive Pixel's FTIR Display [Han 2005] as well as hand-held devices using capacitive sensors such as the Apple iPhone [Jobs et al. 2008]. However, optical systems ar...

Recently, there has been great interest in multi-touch interfaces. Such devices have taken the form of camera-based systems such as Microsoft Surface [de los Reyes et al. 2007] and Perceptive Pixel's FTIR Display [Han 2005] as well as hand-held devices using capacitive sensors such as the Apple iPhone [Jobs et al. 2008]. However, optical systems ar...

Recently, there has been great interest in multi-touch interfaces. These have taken the form of optical systems such as Microsoft Surface and Perceptive Pixel's FTIR display as well as hand-held devices using capacitive sensors such as the Apple iPhone. However, optical systems are inherently bulky while capacitive systems are only practical in sma...

Young people interact with games, animations, and simulations all of the time. But few of them are able to create interactive media. The obstacle: traditional programming languages are too difficult to learn and understand. This panel brings together a group of researchers, developers, and educators who are aiming to democratize the activity of pro...

Multi-touch input has been an active area of research for over two decades but has always suffered from the absence of an easily available high quality touch input device. For this reason, exciting user interfaces developed in the lab have appeared on CNN, but not on everyone's desk, computer screens, table-tops, walls and floors. What has been nee...

This paper reports assessment results of a four-year interdisciplinary research project with the goal to design and test a web-based software environment for real-time, applied programming for underrepresented students' early literacy (RAPUNSEL). The study presented here assessed the impact of the resulting computer game environment called Peeps, w...

Multi-touch input has been an active area of research for over two decades but has always suffered from the absence of an easily available high quality touch input device. For this reason, exciting user interfaces developed in the lab have appeared on CNN, but not on everyone's desk, computer screens, table-tops, walls and floors. What has been nee...

Using Procedural Animation techniques, Autonomous Digital Actors can be created that can take blocking and emotive direction to act out scenes without the use of linear or motion captured animation. We describe the high level architecture of our research that enables (1) interfaces that “coach” the movement styles of digital actors, and (2) the inc...

What is the future of interactive entertainment? Can we tap deeper emotions? Can we go beyond game-like experiences to create powerful interactive literary narratives? Are these even meaningful questions? In this paper I will address one possible way to make these questions meaningful, and one possible path to their answer.

By use of a dance game, and after much input and advice from thirteen year old design consultants, we teach Java programming in a way that will interest middle school girls – a critical age group for addressing gender inequity in programming.

The planar manipulator display uses physical objects for human-computer interaction. This system addresses the poor match between standard computer interfaces and human spatial awareness. The desire to take advantage of these abilities was our primary motivation in developing the planar manipulator display (PMD). The PMD uses movable physical objec...

When we think of ”acting” in computer games, we tend to use a lower standard than the way we think of acting in live action films or theatre. Why is acting in computer games so bad? This is an important question because we will only be able to develop compute games into a more psychologically mature narrative medium when we can imbue them with acto...

We describe a new technique for measuring the bidirectional texture function (BTF) of a surface that requires no mechanical movement, can measure surfaces under arbitrary lighting conditions, and can be made small, portable and inexpensive. The enabling innovation is the use of a tapered kaleidoscope, which allows a camera to view the same surface...

It has long been a goal of the animated agent community to bring embodied agent technologies out of the research labs and into the hands of authors and designers who may put them to practical use. Improv is a set of tools and techniques developed at NYU which make it possible to create applications involving animated agents that behave, interact an...

We are developing software tools for authoring real-time applications involving virtual actors. The actors have mood, presence and personality. They follow a script, using body language and gesture to convey an interactive story that has been scripted beforehand by an author Scripts can contain random elements, so the same story is never told twice...

We present methods for synthesizing 3D shape features on subdivision surfaces using multi-scale procedural techniques. Multi-scale synthesis is a powerful approach for creating surfaces with different levels of detail. Our methods can also blend multiple example multi-resolution surfaces, including procedurally defined surfaces as well as captured...

Two deficiencies in the original Noise algorithm are corrected: second order interpolation discontinuity and unoptimal gradient computation. With these defects corrected, Noise both looks better and runs faster. The latter change also makes it easier to define a uniform mathematical reference standard.

We propose a model of random fractal continuous-derivable procedural noise like Perlin textures, that is animated on a way similar to fluid motion, which a purely stochastic model cannot reproduce. This is done by introducing correlated rotations (despite the noise at a given time is not), and an advection of small scales by large scales.

The NYU Media Research Laboratory has developed a single-person, non-invasive, active autostereoscopic display with no mechanically moving parts that provides a realistic stereoscopic image over a large continuous viewing area and range of distance [Perlin]. We believe this to be the first such display in existence. The display uses eye tracking to...

We believe that navigation in information spaces is best supported by tapping into our natural spatial and geographic ways of thinking. To this end, we are developing a new computer interface model called Pad. The ongoing Pad project uses a spatial metaphor for computer interface design. It provides an intuitive base for the support of such applica...

Improv is a system for the creation of real-time behavior-based animated actors. There have been several recent efforts to build network distributed autonomous agents. But in general these efforts do not focus on the author's view. To create rich interactive worlds inhabited by believable animated actors, authors need the proper tools. Improv provi...

Virtual Worlds can seem to come to life when they have virtual characters in them. On the other hand, if these characters are not “believable” they can actually damage the sense of presence and immersion for a human participant. This problem is not purely one of engineering or of artistic judgement, but rather it calls for a combination of both. Ho...

We present a display device which solves a long-standing problem: to give a true stereoscopic view of simulated objects, without artifacts, to a single unencumbered observer, while allowing the observer to freely change position and head rotation.Based on a novel combination of temporal and spatial multiplexing, this technique will enable artifact-...

We present new methods for painterly video processing. Based on our earlier still image processing technique, we "paint over" successive frames of animation, applying paint only in regions where the source video is changing. Image regions with minimal changes, such as due to video noise, are also left alone, using a simple difference masking techni...

We present a set of very low bandwidth techniques for navigating remote environments. In a typical setup using our system, a virtual environment resides on a server machine, and one or more users explore the environment from client machines. Each client uses previous views of the environment to predict the next view, using the known camera motion a...

Nested User Interface Components combine the concepts of Zooming User Interfaces (ZUIs) with recursive nesting of active graphical user interface widgets. The resulting system of recursively nesting interface components has a number of desirable properties. The level of detail of the view of any widget component and its children, as well as the res...

Scheduling, routing, and layout tasks are examples of hard operations-research problems that have broad application in industry. Typical algorithms for these problems combine some form of gradient descent to find local minima with some strategy for escaping nonoptimal local minima and traversing the search space. Our idea is to divide these two sub...

This paper describes an unbounded resolution paint system. The image is represented using a bi-orthogonal wavelet basis. The wavelet framework makes it possible to implement the necessary multiscale image operations in an efficient manner and without artifacts. This approach has also many other advantages, including conciseness of the representatio...

We present actively procedural multiresolution paint textures. Texture elements may be linearly combined to create complex composite textures that continue to refine themselves when viewed at successively greater magnification. Actively procedural textures constitute a powerful drawing tool that can be used in a multiresolution paint system. They p...

We describe Pad++, a zoomable graphical sketchpad that we are exploring as an alternative to traditional window and icon-based interfaces. We discuss the motivation for Pad++, describe the implementation and present prototype applications. In addition, we introduce an informational physics strategy for interface design and briefly contrast it with...

This paper begins with an overview of the Improv animation and authoring system, followed by the music and audio features, and concluding with a description of recent demonstrations and installations created with Improv

Building on principles from prior work on procedural texture synthesis (K. Perlin, 1985), we are able to create remarkably lifelike, responsively animated characters in real time. Rhythmic and stochastic noise functions are used to define time varying parameters that drive computer generated puppets. Because we are conveying just the “texture” of m...

We believe that navigation in information spaces is best supported by tapping into our natural spatial and geographic ways of thinking. To this end, we are developing a new computer interface model called Pad. The ongoing Pad project uses a spatial metaphor for computer interface design. It provides an intuitive base for the support of such applica...

Every year we spend a greater proportion of our daily lives, at home, in business, in education, and in entertainment, in front of a computer screen. As networking becomes commonplace, this increasingly includes our activities with other people. We want to aim at a richer interaction. How can technology become a seamless extension of our real world...

We model phenomena intermediate between shape and texture by using space-filling applicative functions to modulate density. The model is essentially an extension of procedural solid texture synthesis, but evaluated throughout a volumetric region instead of only at surfaces.We have been able to obtain visually realistic representations of such shape...

We have developed two interactive document visualiza- tions in Pad++, an environment for exploring zooming techniques for interfaces. The first is a page-based ellipti- cal view that shows an entire document simultaneously and provides multiple navigation mechanisms. We have used it to develop a tutorial document. The second tech- nique employs an...

This paper reports on the assessment of a three-year research project funded by the National Science Foundation to design a successful web-based software environment for real-time, applied programming for underrepresented students' early literacy (RAPUNSEL). The goal was to develop an engaging way to teach computer programming to middle school girl...