Harnessing Procedurally Generated 3D VFX: HCCC from within VR

Preprints and early-stage research may not have been peer reviewed yet.
To read the file of this research, you can request a copy directly from the author.


This paper summarizes the development of a novel application that addresses creativity across multiple domains including music, games, visual arts, entertainment , and programming. Through a participatory iterative design and evaluation research methodology this creative application pursues new depths of mixed reality (XR) human-machine interaction (HCI). The nature of the co-creative framework emphasizes the machine's role as a central agent in virtual world-building, and whose creative and artistic decisions are separate from, but of a collaboratory nature to, a Human actor. By harnessing powerful procedural 3D animation and visual effects tools designed and used as industry standards by digital artists to create the highest-quality cinematic results, complex world-making from within Virtual Reality (VR) is made possible. Presented here is a unique system that combines Artificial Intelligence (AI), VR, and complex content generation that utilizes Web-and Cloud-based frameworks to integrate Real-time 3D rendering with procedural modelling and dynamic simulation. Collaborative creativity (CC) is therefore made accessible to both multiple Human agents through tele-presence and to Artificial agents-creatively responding to their constantly evolving virtual world.

No file available

Request Full-text Paper PDF

To read the file of this research,
you can request a copy directly from the author.

ResearchGate has not been able to resolve any citations for this publication.
Conference Paper
Full-text available
The 3D modeling methods and approach presented in this paper attempt to bring the richness and spontaneity of human kinesthetic interaction in the physical world to the process of shaping digital form, by exploring playfully creative interaction techniques that augment gestural movement. The principal contribution of our research is a novel dynamics-driven approach for immersive freeform modeling, which extends our physical reach and supports new forms of expression. In this paper we examine three augmentations of freehand 3D interaction that are inspired by the dynamics of physical phenomena. These are experienced via immersive augmented reality to intensify the virtual physicality and heighten the sense of creative empowerment.
Full-text available
The Journal of Aesthetic Education 40.1 (2006) 1-21 What are the humanities, and how should they be cultivated? With respect to this crucial question, opinions differ as to how widely the humanities should be construed and pursued. Initially connoting the study of Greek and Roman classics, the concept now more generally covers arts and letters, history, and philosophy. But does it also include the social sciences, which are often distinguished from the humanities and grouped as a separate academic division with greater pretensions to scientific status? And should our pursuit of humanistic study be concentrated on the traditional methods and topics of high culture that give the humanities an authoritative aura of established nobility, or should it extend to new and funkier forms of interdisciplinary research such as popular culture or race and gender studies? Despite such questions and controversy, it is clear (even from etymology) that the meaning of the humanities essentially relates to our human condition and our efforts to perfect our humanity and its expression. But what, then, does it mean to be human? I cannot pretend here to adequately answer such a complex and difficult question. I will, however, argue that because the body is an essential and valuable dimension of our humanity, it should be recognized as a crucial topic of humanistic study and experiential learning. Though the truth of this thesis should be obvious, it goes sharply against the grain of our traditional understanding of the humanities. One striking example of such antisomatic bias is the very term that German speakers use to designate the humanities, "Geisteswissenschaften," whose literal English translation would be "spiritual (or mental) sciences," as contrasted to the natural sciences (Naturwissenschaften) that treat physical life, with which, of course, the body is clearly linked. Hence, given the pervasive physical/spiritual opposition, the body is essentially omitted or marginalized in our conception of humanistic studies. We humanist intellectuals generally take the body for granted because we are so passionately interested in the life of the mind and the creative arts that express our human spirit. But the body is not only an essential dimension of our humanity, it is also the basic instrument of all human performance, our tool of tools, a necessity for all our perception, action, and even thought. Just as skilled builders need expert knowledge of their tools, so we need better somatic knowledge to improve our understanding and performance in the arts and human sciences and to advance our mastery in the highest art of all—that of perfecting our humanity and living better lives. We need to think more carefully through the body in order to cultivate ourselves and edify our students because true humanity is not a mere genetic given but an educational achievement in which body, mind, and culture must be thoroughly integrated. To pursue this project of somatic inquiry, I have been working on an interdisciplinary field called somaesthetics, whose disciplinary connections extend also beyond the humanities to the biological, cognitive, and health sciences, which I see as valuable allies for humanistic research. Somaesthetics, roughly defined, concerns the body as a locus of sensory-aesthetic appreciation (aisthesis) and creative self-fashioning. As an ameliorative discipline of both theory and practice, it aims to enrich not only our abstract, discursive knowledge of the body but also our lived somatic experience and performance; it seeks to enhance the meaning, understanding, efficacy, and beauty of our movements and of the environments to which our movements contribute and from which they also draw their energies and significance. Somaesthetics, therefore, involves a wide range of knowledge forms and disciplines that structure such somatic care or can improve it. Recognizing that body, mind, and culture are deeply codependent, somaesthetics comprises an interdisciplinary research program to integrate their study. Mental life relies on somatic experience and cannot be wholly separated from bodily processes, even if it cannot be wholly reduced to them. We think and feel with our bodies, especially with the body parts that constitute the brain and nervous system. Our bodies are likewise affected by mental life, as when certain thoughts bring a blush to the cheek and change our...
Full-text available
This paper establishes a link between the challenge of solv-ing highly ambitious problems in machine learning and the goal of reproducing the dynamics of open-ended evolution in artificial life. A major problem with the objective function in machine learning is that through deception it may actu-ally prevent the objective from being reached. In a similar way, selection in evolution may sometimes act to discourage increasing complexity. This paper proposes a single idea that both overcomes the obstacle of deception and suggests a sim-ple new approach to open-ended evolution: Instead of either explicitly seeking an objective or modeling a domain to cap-ture the open-endedness of natural evolution, the idea is to simply search for novelty. Even in an objective-based prob-lem, such novelty search ignores the objective and searches for behavioral novelty. Yet because many points in the search space collapse to the same point in behavior space, it turns out that the search for novelty is computationally feasible. Furthermore, because there are only so many simple behav-iors, the search for novelty leads to increasing complexity. In fact, on the way up the ladder of complexity, the search is likely to encounter at least one solution. In this way, by de-coupling the idea of open-ended search from only artificial life worlds, the raw search for novelty can be applied to real world problems. Counterintuitively, in the deceptive maze navigation task in this paper, novelty search significantly out-performs objective-based search, suggesting a surprising new approach to machine learning.
Conference Paper
Full-text available
In this research we observe the situated, embodied and playful interaction that participants engage in with open-ended interactive artworks. The larger project from which this work derives [28] contributes a methodological model for the evaluation of open- ended interactive artwork that treats each work individually and recognises the importance of the artist intent and the traditions from which the work derives. In this paper, we describe this evolving methodology for evaluating and understanding participation via three case studies of open-ended interactive art installations. This analysis builds an understanding of open-ended free-play non-narrative environments and the affordances these environments enable for participants.
The use of the senses of vision and audition as interactive means has dominated the field of Human-Computer Interaction (HCI) for decades, even though nature has provided us with many more senses for perceiving and interacting with the world around us. That said, it has become attractive for HCI researchers and designers to harness touch, taste, and smell in interactive tasks and experience design. In this paper, we present research and design insights gained throughout an interdisciplinary collaboration on a six-week multisensory display – Tate Sensorium – exhibited at the Tate Britain art gallery in London, UK. This is a unique and first time case study on how to design art experiences whilst considering all the senses (i.e., vision, sound, touch, smell, and taste), in particular touch, which we exploited by capitalizing on a novel haptic technology, namely, mid-air haptics. We first describe the overall set up of Tate Sensorium and then move on to describing in detail the design process of the mid-air haptic feedback and its integration with sound for the Full Stop painting by John Latham (1961). This was the first time that mid-air haptic technology was used in a museum context over a prolonged period of time and integrated with sound to enhance the experience of visual art. As part of an interdisciplinary team of curators, sensory designers, sound artists, we selected a total of three variations of the mid-air haptic experience (i.e., haptic patterns), which were alternated at dedicated times throughout the six-week exhibition. We collected questionnaire-based feedback from 2500 visitors and conducted 50 interviews to gain quantitative and qualitative insights on visitors’ experiences and emotional reactions. Whilst the questionnaire results are generally very positive with only a small variation of the visitors’ arousal ratings across the three tactile experiences designed for the Full Stop painting, the interview data shed light on the differences in the visitors’ subjective experiences. Our findings suggest multisensory designers and art curators can ensure a balance between surprising experiences versus the possibility of free exploration for visitors. In addition, participants expressed that experiencing art with the combination of mid-air haptic and sound was immersive and provided an up-lifting experience of touching without touch. We are convinced that the insights gained from this large-scale and real-world field exploration of multisensory experience design exploiting a new and emerging technology provide a solid starting point for the HCI community, creative industries, and art curators to think beyond conventional art experiences. Specifically, our work demonstrates how novel mid-air technology can make art more emotionally engaging and stimulating, especially abstract art that is often open to interpretation.
I summarise and attempt to clarify some concepts presented in and arising from Margaret Boden’s (1990) descriptive hierarchy of creativity, by beginning to formalise the ideas she proposes. The aim is to move towards a model which allows detailed comparison, and hence better understanding, of systems which exhibit behaviour which would be called “creative” in humans. The work paves the way for the description of naturalistic, multi-agent creative AI systems, which create in a societal context.I demonstrate some simple reasoning about creative behaviour based on the new framework, to show how it might be useful for the analysis and study of creative systems. In particular, I identify some crucial properties of creative systems, in terms of the framework components, some of which may usefully be proven a priori of a given system.I suggest that Boden’s descriptive framework, once elaborated in detail, is more uniform and more powerful than it first appears.
Express yourself! augmenting reality with graffiti
  • S Kuehnert
Kuehnert, S. 2015. Express yourself! augmenting reality with graffiti.
Dawn of the New Everything: Encounters with Reality and Virtual Reality
  • J Lanier
Lanier, J. 2017. Dawn of the New Everything: Encounters with Reality and Virtual Reality.Henry Holt and Company.
  • J Mccormack
McCormack, J. et al. 2018. Multisensory Immersive Analytics. Immersive Analytics, LNCS 11190. 57-94.
WinWin, Paul Petro Contemporary Art
  • N Tenhaff
Tenhaff, N. 2013. WinWin, Paul Petro Contemporary Art, Toronto, Ontario, Canada.
Push/Pull: an excitable sculpture, live demo with EEG sonification. LIVELab
  • N Tenhaff
Tenhaff, N. 2018. Push/Pull: an excitable sculpture, live demo with EEG sonification. LIVELab, McMaster University, Hamilton, Ontario, Canada.
Mind the Gap: An attempt to bridge computational and neuroscientific approaches to study creativity. Frontiers in Human Neuroscience. 8. Omitted for Blind Review
  • G Wiggins
  • J Bhattacharya
Wiggins, G. and Bhattacharya, J. 2014. Mind the Gap: An attempt to bridge computational and neuroscientific approaches to study creativity. Frontiers in Human Neuroscience. 8. Omitted for Blind Review