Ken Perlin

Ken Perlin
New York University | NYU · Department of Computer Science

About

78
Publications
26,365
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
4,149
Citations
Citations since 2017
29 Research Items
1418 Citations
2017201820192020202120222023050100150200250
2017201820192020202120222023050100150200250
2017201820192020202120222023050100150200250
2017201820192020202120222023050100150200250

Publications

Publications (78)
Article
Full-text available
This pilot study aimed to identify postural strategies in response to sensory perturbations (visual, auditory, somatosensory) in adults with and without sensory loss. We tested people with unilateral peripheral vestibular hypofunction (N = 12, mean age 62 range 23-78), or with Unilateral Sensorineural Hearing Loss (USNHL, N = 9, 48, 22-82), or heal...
Preprint
Full-text available
When creating 3D content, highly specialized skills are generally needed to design and generate models of objects and other assets by hand. We address this problem through high-quality 3D asset retrieval from multi-modal inputs, including 2D sketches, images and text. We use CLIP as it provides a bridge to higher-level latent features. We use these...
Chapter
How do we design effective immersive VR experiences? Looking Inside Cells is a set of collaborative immersive virtual reality science learning simulations that were designed by applying best practices for learning experience design, taking advantage of the unique affordances of VR for learning. In this paper we describe design features of immersive...
Preprint
Full-text available
Exploring large virtual environments, such as cities, is a central task in several domains, such as gaming and urban planning. VR systems can greatly help this task by providing an immersive experience; however, a common issue with viewing and navigating a city in the traditional sense is that users can either obtain a local or a global view, but n...
Preprint
Full-text available
Video conferences play a vital role in our daily lives. However, many nonverbal cues are missing, including gaze and spatial information. We introduce LookAtChat, a web-based video conferencing system, which empowers remote users to identify gaze awareness and spatial relationships in small-group conversations. Leveraging real-time eye-tracking tec...
Conference Paper
Full-text available
Writing or sketching on whiteboards is an essential part of collaborative discussions in business meetings, reading groups, design sessions, and interviews. However, prior work in collaborative virtual reality (VR) systems has rarely explored the design space of multi-user layouts and interaction modes with virtual whiteboards. In this paper, we pr...
Preprint
Full-text available
We developed a novel assessment platform with untethered virtual reality, 3-dimensional sounds, and pressure sensing floor mat to help assess the walking balance and negotiation of obstacles given diverse sensory load and/or cognitive load. The platform provides an immersive 3D city-like scene with anticipated/unanticipated virtual obstacles. Parti...
Preprint
Despite various collaborative software that supports expressing ideas, people still largely prefer physical notebooks or whiteboards. The reason is that they provide free-form expressions, co-presence of all participants and easy collaboration. However, when working with remote participants, people often choose the convenience of video conferencing...
Preprint
Full-text available
Virtual Reality (VR) enables users to collaborate while exploring scenarios not realizable in the physical world. We propose CollabVR, a distributed multi-user collaboration environment, to explore how digital content improves expression and understanding of ideas among groups. To achieve this, we designed and examined three possible configurations...
Conference Paper
Full-text available
Immersive co-located theatre aims to bring the social aspects of traditional cinematic and theatrical experience into Virtual Reality (VR). Within these VR environments, participants can see and hear each other, while their virtual seating location corresponds to their actual position in the physical space. These elements create a realistic sense o...
Conference Paper
The virtual reality ecosystem has gained momentum in the gaming, entertainment, and enterprise markets, but is hampered by limitations in concurrent user count, throughput, and accessibility to mass audiences. Based on our analysis of the current state of the virtual reality ecosystem and relevant aspects of traditional media, we propose a set of d...
Preprint
Full-text available
As synthetic imagery is used more frequently in training deep models, it is important to understand how different synthesis techniques impact the performance of such models. In this work, we perform a thorough evaluation of the effectiveness of several different synthesis techniques and their impact on the complexity of classifier domain adaptation...
Article
CAVE is a shared narrative six degrees of freedom (6DoF) virtual reality experience. In 3.5 days, 1,927 people attended its premiere at SIGGRAPH 2018. Thirty participants at a time each saw and heard the same narrative from their own individual location in the room, as they would when attending live theater. CAVE set out to disruptively change how...
Conference Paper
Mary and the Monster: Chapter One is a shared AR experience that brings to life "Frankenstein" through the eyes of a young Mary Shelley. Using Parallux technology and the Magic Leap One, audiences are immersed in the author's world as she conjures the most famous gothic tale of all time.
Conference Paper
CAVE is a shared narrative six degrees of freedom (6DoF) virtual reality experience. In 3.5 days, 1,927 people attended its premiere at SIGGRAPH 2018. Thirty participants at a time each saw and heard the same narrative from their own individual location in the room, as they would when attending live theater. CAVE set out to disruptively change how...
Conference Paper
Full-text available
Everyday mobile usage of AR and VR Head-Mounted Displays (HMDs) is becoming a feasible consumer reality. The current research agenda for HMDs has a strong focus on technological impediments (e.g. latency, field of view, locomotion, tracking, input) as well as perceptual aspect (e.g. distance compression, vergence-accomodation ). However, this ignor...
Conference Paper
Full-text available
Mixed Reality (MR) enables users to explore scenarios not realizable in the physical world. This allows users to communicate with the help of digital content. We investigate how different configurations of participants and content affect communication in a shared immersive environment. We designed and implemented side-by-side, mirrored face-to-face...
Conference Paper
Full-text available
Virtual Reality enables users to explore content whose physics are only limited by our creativity. Such limitless environments provide us with many opportunities to explore innovative ways to support productivity and collaboration. We present Spacetime, a scene editing tool built from the ground up to explore the novel interaction techniques that e...
Conference Paper
Following their introduction in the 1960s, head-mounted VR systems mainly focused on visual and aural senses. In order to enhance immersion in the virtual world, researchers have since pursued the addition of movement and haptics through motion platforms, exoskeletons, and other hand-held devices. From a proliferation of low-cost devices that can s...
Article
Full-text available
Real-time simulation of fluid and smoke is a long standing problem in computer graphics. Despite significant progress in the literature, state-of-the-art approaches require large compute resources, making real-time applications impractical. In this work, we propose a data-driven approach that leverages the approximation power of deep-learning metho...
Conference Paper
Full-text available
We present a novel method for real-time continuous pose recovery of markerless complex articulable objects from a single depth image. Our method consists of the following stages: a randomized decision forest classifier for image segmentation, a robust method for labeled dataset generation, a convolutional network for dense feature extraction, and f...
Article
Full-text available
Paper submitted for presentation at the Annual Meeting of the American Educational Research Association (AERA), New Orleans, LA (April, 2011). Results are presented from an experimental investigation of the effect of mode of play on motivational and educational outcomes. A computer-based arithmetic game was adapted to allow for solo, competitive, o...
Conference Paper
A computer-based geometry game was adapted to allow for play using a conceptual rule or an arithmetic problem-solving mechanic. Participants (n = 91) from an urban middle school were randomly assigned to experimental conditions. Results suggest that play in the number condition was more situationally interesting than play in the rule condition. Par...
Conference Paper
What will the interface between people and computers look like in five years? In ten years? In twenty five years? Will we still have screens? Keyboards? Will we all be seeing Princess Leia in a beam of light? Based on current trends and inspired guesswork, we will go together on a tour of the future.
Technical Report
Full-text available
In this paper, we will discuss the approach for assessment of learning and related learner variables taken by the Games for Learning Institute (G4LI). We will first describe game mechanics in general, and then introduce the concepts of learning mechanics and assessment mechanics and describe criteria for their design and requirements of how they ca...
Article
Procedural noise functions are widely used in computer graphics, from off-line rendering in movie production to interactive video games. The ability to add complex and intricate details at low memory and authoring cost is one of its main attractions. This survey is motivated by the inherent importance of noise in graphics, the widespread use of noi...
Article
Humans express their emotions in many ways, in particular through face, eye, and body motion. So creators of virtual humans strive to convincingly depict emotional movements using a variety of methods.This course focuses on the use of realistic human body motion to generate emotional expressiveness. Topics include: applications and research relatin...
Article
Recently, there has been great interest in multi-touch interfaces. Such devices have taken the form of camera-based systems such as Microsoft Surface [de los Reyes et al. 2007] and Perceptive Pixel's FTIR Display [Han 2005] as well as hand-held devices using capacitive sensors such as the Apple iPhone [Jobs et al. 2008]. However, optical systems ar...
Article
Recently, there has been great interest in multi-touch interfaces. Such devices have taken the form of camera-based systems such as Microsoft Surface [de los Reyes et al. 2007] and Perceptive Pixel's FTIR Display [Han 2005] as well as hand-held devices using capacitive sensors such as the Apple iPhone [Jobs et al. 2008]. However, optical systems ar...
Conference Paper
Recently, there has been great interest in multi-touch interfaces. These have taken the form of optical systems such as Microsoft Surface and Perceptive Pixel's FTIR display as well as hand-held devices using capacitive sensors such as the Apple iPhone. However, optical systems are inherently bulky while capacitive systems are only practical in sma...
Conference Paper
Young people interact with games, animations, and simulations all of the time. But few of them are able to create interactive media. The obstacle: traditional programming languages are too difficult to learn and understand. This panel brings together a group of researchers, developers, and educators who are aiming to democratize the activity of pro...
Conference Paper
Multi-touch input has been an active area of research for over two decades but has always suffered from the absence of an easily available high quality touch input device. For this reason, exciting user interfaces developed in the lab have appeared on CNN, but not on everyone's desk, computer screens, table-tops, walls and floors. What has been nee...
Article
Full-text available
This paper reports assessment results of a four-year interdisciplinary research project with the goal to design and test a web-based software environment for real-time, applied programming for underrepresented students' early literacy (RAPUNSEL). The study presented here assessed the impact of the resulting computer game environment called Peeps, w...
Conference Paper
Multi-touch input has been an active area of research for over two decades but has always suffered from the absence of an easily available high quality touch input device. For this reason, exciting user interfaces developed in the lab have appeared on CNN, but not on everyone's desk, computer screens, table-tops, walls and floors. What has been nee...
Conference Paper
Using Procedural Animation techniques, Autonomous Digital Actors can be created that can take blocking and emotive direction to act out scenes without the use of linear or motion captured animation. We describe the high level architecture of our research that enables (1) interfaces that “coach” the movement styles of digital actors, and (2) the inc...
Conference Paper
What is the future of interactive entertainment? Can we tap deeper emotions? Can we go beyond game-like experiences to create powerful interactive literary narratives? Are these even meaningful questions? In this paper I will address one possible way to make these questions meaningful, and one possible path to their answer.
Conference Paper
By use of a dance game, and after much input and advice from thirteen year old design consultants, we teach Java programming in a way that will interest middle school girls – a critical age group for addressing gender inequity in programming.
Article
Full-text available
The planar manipulator display uses physical objects for human-computer interaction. This system addresses the poor match between standard computer interfaces and human spatial awareness. The desire to take advantage of these abilities was our primary motivation in developing the planar manipulator display (PMD). The PMD uses movable physical objec...
Conference Paper
When we think of ”acting” in computer games, we tend to use a lower standard than the way we think of acting in live action films or theatre. Why is acting in computer games so bad? This is an important question because we will only be able to develop compute games into a more psychologically mature narrative medium when we can imbue them with acto...
Article
We describe a new technique for measuring the bidirectional texture function (BTF) of a surface that requires no mechanical movement, can measure surfaces under arbitrary lighting conditions, and can be made small, portable and inexpensive. The enabling innovation is the use of a tapered kaleidoscope, which allows a camera to view the same surface...
Article
Full-text available
It has long been a goal of the animated agent community to bring embodied agent technologies out of the research labs and into the hands of authors and designers who may put them to practical use. Improv is a set of tools and techniques developed at NYU which make it possible to create applications involving animated agents that behave, interact an...
Article
We are developing software tools for authoring real-time applications involving virtual actors. The actors have mood, presence and personality. They follow a script, using body language and gesture to convey an interactive story that has been scripted beforehand by an author Scripts can contain random elements, so the same story is never told twice...
Article
We present methods for synthesizing 3D shape features on subdivision surfaces using multi-scale procedural techniques. Multi-scale synthesis is a powerful approach for creating surfaces with different levels of detail. Our methods can also blend multiple example multi-resolution surfaces, including procedurally defined surfaces as well as captured...
Article
Two deficiencies in the original Noise algorithm are corrected: second order interpolation discontinuity and unoptimal gradient computation. With these defects corrected, Noise both looks better and runs faster. The latter change also makes it easier to define a uniform mathematical reference standard.
Article
We propose a model of random fractal continuous-derivable procedural noise like Perlin textures, that is animated on a way similar to fluid motion, which a purely stochastic model cannot reproduce. This is done by introducing correlated rotations (despite the noise at a given time is not), and an advection of small scales by large scales.
Article
Full-text available
The NYU Media Research Laboratory has developed a single-person, non-invasive, active autostereoscopic display with no mechanically moving parts that provides a realistic stereoscopic image over a large continuous viewing area and range of distance [Perlin]. We believe this to be the first such display in existence. The display uses eye tracking to...
Article
We believe that navigation in information spaces is best supported by tapping into our natural spatial and geographic ways of thinking. To this end, we are developing a new computer interface model called Pad. The ongoing Pad project uses a spatial metaphor for computer interface design. It provides an intuitive base for the support of such applica...
Article
Full-text available
Improv is a system for the creation of real-time behavior-based animated actors. There have been several recent efforts to build network distributed autonomous agents. But in general these efforts do not focus on the author's view. To create rich interactive worlds inhabited by believable animated actors, authors need the proper tools. Improv provi...
Conference Paper
Virtual Worlds can seem to come to life when they have virtual characters in them. On the other hand, if these characters are not “believable” they can actually damage the sense of presence and immersion for a human participant. This problem is not purely one of engineering or of artistic judgement, but rather it calls for a combination of both. Ho...
Conference Paper
Full-text available
We present a display device which solves a long-standing problem: to give a true stereoscopic view of simulated objects, without artifacts, to a single unencumbered observer, while allowing the observer to freely change position and head rotation.Based on a novel combination of temporal and spatial multiplexing, this technique will enable artifact-...
Article
We present new methods for painterly video processing. Based on our earlier still image processing technique, we "paint over" successive frames of animation, applying paint only in regions where the source video is changing. Image regions with minimal changes, such as due to video noise, are also left alone, using a simple difference masking techni...
Article
We present a set of very low bandwidth techniques for navigating remote environments. In a typical setup using our system, a virtual environment resides on a server machine, and one or more users explore the environment from client machines. Each client uses previous views of the environment to predict the next view, using the known camera motion a...
Conference Paper
Nested User Interface Components combine the concepts of Zooming User Interfaces (ZUIs) with recursive nesting of active graphical user interface widgets. The resulting system of recursively nesting interface components has a number of desirable properties. The level of detail of the view of any widget component and its children, as well as the res...
Conference Paper
Scheduling, routing, and layout tasks are examples of hard operations-research problems that have broad application in industry. Typical algorithms for these problems combine some form of gradient descent to find local minima with some strategy for escaping nonoptimal local minima and traversing the search space. Our idea is to divide these two sub...
Article
This paper describes an unbounded resolution paint system. The image is represented using a bi-orthogonal wavelet basis. The wavelet framework makes it possible to implement the necessary multiscale image operations in an efficient manner and without artifacts. This approach has also many other advantages, including conciseness of the representatio...
Article
We present actively procedural multiresolution paint textures. Texture elements may be linearly combined to create complex composite textures that continue to refine themselves when viewed at successively greater magnification. Actively procedural textures constitute a powerful drawing tool that can be used in a multiresolution paint system. They p...
Article
We describe Pad++, a zoomable graphical sketchpad that we are exploring as an alternative to traditional window and icon-based interfaces. We discuss the motivation for Pad++, describe the implementation and present prototype applications. In addition, we introduce an informational physics strategy for interface design and briefly contrast it with...
Article
Full-text available
This paper begins with an overview of the Improv animation and authoring system, followed by the music and audio features, and concluding with a description of recent demonstrations and installations created with Improv
Article
Building on principles from prior work on procedural texture synthesis (K. Perlin, 1985), we are able to create remarkably lifelike, responsively animated characters in real time. Rhythmic and stochastic noise functions are used to define time varying parameters that drive computer generated puppets. Because we are conveying just the “texture” of m...
Article
We believe that navigation in information spaces is best supported by tapping into our natural spatial and geographic ways of thinking. To this end, we are developing a new computer interface model called Pad. The ongoing Pad project uses a spatial metaphor for computer interface design. It provides an intuitive base for the support of such applica...
Conference Paper
Every year we spend a greater proportion of our daily lives, at home, in business, in education, and in entertainment, in front of a computer screen. As networking becomes commonplace, this increasingly includes our activities with other people. We want to aim at a richer interaction. How can technology become a seamless extension of our real world...
Conference Paper
We model phenomena intermediate between shape and texture by using space-filling applicative functions to modulate density. The model is essentially an extension of procedural solid texture synthesis, but evaluated throughout a volumetric region instead of only at surfaces.We have been able to obtain visually realistic representations of such shape...
Article
Full-text available
We have developed two interactive document visualiza- tions in Pad++, an environment for exploring zooming techniques for interfaces. The first is a page-based ellipti- cal view that shows an entire document simultaneously and provides multiple navigation mechanisms. We have used it to develop a tutorial document. The second tech- nique employs an...
Article
Full-text available
This paper reports on the assessment of a three-year research project funded by the National Science Foundation to design a successful web-based software environment for real-time, applied programming for underrepresented students' early literacy (RAPUNSEL). The goal was to develop an engaging way to teach computer programming to middle school girl...

Network

Cited By

Projects

Project (1)
Archived project
Explore the use of procedural 3D animation in the creation of real-time, interactive digital actors and environments