Shunyu Yao’s scientific contributions

We aim to obtain an interpretable, expressive and disentangled scene representation that contains comprehensive structural and textural information for each object. Previous representations learned by neural networks are often uninterpretable, limited to a single object, or lack 3D knowledge. In this work, we address the above issues by integrating 3D modeling into a deep generative model. We adopt a differentiable shape renderer to decode geometrical object attributes into a shape, and a neural generator to decode learned latent codes to texture. The encoder is therefore forced to perform an inverse graphics task and transform a scene image into a structured representation with 3D attributes of objects and learned texture latent codes. The representation supports reconstruction and a variety of 3D-aware scene manipulation applications. The disentanglement of structure and texture in our representation allows us to rotate and move objects freely while maintaining consistent texture, as well as changing the object appearance without affecting their structures. We systematically evaluate our representation and demonstrate that our editing scheme is superior to 2D counterparts.

We aim to obtain an interpretable, expressive, and disentangled scene representation that contains comprehensive structural and textural information for each object. Previous scene representations learned by neural networks are often uninterpretable, limited to a single object, or lacking 3D knowledge. In this work, we propose 3D scene de-rendering networks (3D-SDN) to address the above issues by integrating disentangled representations for semantics, geometry, and appearance into a deep generative model. Our scene encoder performs inverse graphics, translating a scene into a structured object-wise representation. Our decoder has two components: a differentiable shape renderer and a neural texture generator. The disentanglement of semantics, geometry, and appearance supports 3D-aware scene manipulation, e.g., rotating and moving objects freely while keeping the consistent shape and texture, and changing the object appearance without affecting its shape. Experiments demonstrate that our editing scheme based on 3D-SDN is superior to its 2D counterpart. ©2018 Poster presentation at the 32nd annual Conference on Neural Information Processing Systems (NIPS 2018), December 3-5, 2018, Montréal, Québec.