Perttu Hämäläinen's research while affiliated with Aalto University and other places

Publications (71)

Preprint
Full-text available
Cutscenes form an integral part of many video games, but their creation is costly, time-consuming, and requires skills that many game developers lack. While AI has been leveraged to semi-automate cutscene production, the results typically lack the internal consistency and uniformity in style that is characteristic of professional human directors. W...
Chapter
OpenSim is a widely used biomechanics simulator with several anatomically accurate human musculo-skeletal models. While OpenSim provides useful tools to analyse human movement, it is not fast enough to be routinely used for emerging research directions, e.g., learning and simulating motor control through deep neural networks and Reinforcement Learn...
Article
Full-text available
Virtual reality provides a powerful way to visualize the three-dimensional, atomic-level structures of molecules and materials. We present new virtual reality software for molecular modeling and for testing the use of virtual reality in organic chemistry education. The open-source software, named VRChem, was developed primarily for building, visual...
Article
This paper presents a novel approach to automated playtesting for the prediction of human player behavior and experience. We have previously demonstrated that Deep Reinforcement Learning (DRL) game-playing agents can predict both game difficulty and player engagement, operationalized as average pass and churn rates. We improve this approach by enha...
Preprint
This paper presents a novel approach to automated playtesting for the prediction of human player behavior and experience. It has previously been demonstrated that Deep Reinforcement Learning (DRL) game-playing agents can predict both game difficulty and player engagement, operationalized as average pass and churn rates. We improve this approach by...
Preprint
Full-text available
We propose a novel method for exploring the dynamics of physically based animated characters, and learning a task-agnostic action space that makes movement optimization easier. Like several previous papers, we parameterize actions as target states, and learn a short-horizon goal-conditioned low-level control policy that drives the agent's state tow...
Preprint
We propose a novel simulation model that is able to predict the per-level churn and pass rates of Angry Birds Dream Blast, a popular mobile free-to-play game. Our primary contribution is to combine AI gameplay using Deep Reinforcement Learning (DRL) with a simulation of how the player population evolves over the levels. The AI players predict level...
Preprint
We propose Deep Residual Mixture Models (DRMMs) which share the many desirable properties of Gaussian Mixture Models (GMMs), but with a crucial benefit: The modeling capacity of a DRMM can grow exponentially with depth, while the number of model parameters only grows quadratically. DRMMs allow for extremely flexible conditional sampling, as the con...
Preprint
Full-text available
OpenSim is a widely used biomechanics simulator with several anatomically accurate human musculo-skeletal models. While OpenSim provides useful tools to analyse human movement, it is not fast enough to be routinely used for emerging research directions, e.g., learning and simulating motor control through deep neural networks and Reinforcement Learn...
Conference Paper
Full-text available
Animation and machine learning research have shown great advancements in the past decade, leading to robust and powerful methods for learning complex physically-based animations. However, learning can take hours or days, especially if no reference movement data is available. In this paper, we propose and evaluate a novel combination of techniques f...
Conference Paper
Gameplay is often an emotionally charged activity, in particular when streaming in front of a live audience. From a games user research perspective, it would be beneficial to automatically detect and recognize players' and streamers' emotional expression, as this data can be used for identifying gameplay highlights, computing emotion metrics or to...
Conference Paper
Insufficient physical activity motivation is a major public health problem. Exergames-games requiring physical exertion-can be designed to support motivation. For example, granting superhuman movement abilities to players has been shown to support one's feeling of competence, an innate human need and a core intrinsic motivation factor posited by se...
Thesis
Full-text available
User interface design for Virtual Reality and other embodied interaction contexts has to carefully consider ergonomics. A common problem is that mid-air interaction may cause excessive arm fatigue, known as the “Gorilla arm” effect. To predict and prevent such problems at a low cost, this thesis investigates user testing of mid-air interaction with...
Preprint
A large body of animation research focuses on optimization of movement control, either as action sequences or policy parameters. However, as closed-form expressions of the objective functions are often not available, our understanding of the optimization problems is limited. Building on recent work on analyzing neural network training, we contribut...
Preprint
Full-text available
Animation and machine learning research have shown great advancements in the past decade, leading to robust and powerful methods for learning complex physically-based animations. However, learning can take hours or days, especially if no reference movement data is available. In this paper, we propose and evaluate a novel combination of techniques f...
Conference Paper
The general trend in exercise interventions, including those based on exergames, is to see high initial enthusiasm but significantly declining adherence. Social play is considered a core tenet of the design of exercise interventions help foster motivation to play. To determine whether social play aids in adherence to exergames, we analyzed data fro...
Preprint
Full-text available
Proximal Policy Optimization (PPO) is a highly popular model-free reinforcement learning (RL) approach. However, we observe that in a continuous action space, PPO can prematurely shrink the exploration variance, which leads to slow progress and may make the algorithm prone to getting stuck in local optima. Drawing inspiration from CMA-ES, a black-b...
Conference Paper
Empowerment of movement through superhuman strength and flexibility is a staple of action video game design. However, relatively little work has been done on the same in the context of Virtual Reality and exergames, especially outside the most obvious parameters such as jumping height and locomotion speed. We contribute a controlled experiment (N=3...
Conference Paper
We present an exploratory study of analyzing and visualizing player facial expressions from video with deep neural networks. We contribute a novel data processing and visualization technique we call Affect Gradients, which provides descriptive statistics of the expressive responses to game events, such as player death or collecting a power-up. As a...
Conference Paper
Drawing, sketching and other forms of visual communication are useful tools for communicating our thoughts especially in a multidisciplinary setting, yet instruction in drawing is often limited by the amount of students one instructor can guide at once. Aalto Drawing presents an accessible, low-threshold opportunity for any student to learn to draw...
Conference Paper
Emergent behaviour occurs when simple systems result in complex behaviour. In games, this allows for more meth- ods of interaction, and lets the player explore in a way they want. It is challenging to design for emergence, as it typ- ically arises unexpectedly . With "Plusminus", we devel- oped a method of augmenting physics to promote emergent gam...
Preprint
We propose the concept of intelligent middle-level game control, which lies on a continuum of control abstraction levels between the following two dual opposites: 1) high-level control that translates player's simple commands into complex actions (such as pressing Space key for jumping), and 2) low-level control which simulates real-life complexiti...
Conference Paper
This paper presents a review of intrinsic motivation in player modeling, with a focus on simulation-based game testing. Modern AI agents can learn to win many games; from a game testing perspective, a remaining research problem is how to model the aspects of human player behavior not explained by purely rational and goal-driven decision making. A m...
Article
We present a simple way to learn a transformation that maps samples of one distribution to the samples of another distribution. Our algorithm comprises an iteration of 1) drawing samples from some simple distribution and transforming them using a neural network, 2) determining pairwise correspondences between the transformed samples and training da...
Conference Paper
We introduce a novel approach for improving performance and motivation in sports, which we term computer-aided imagery (CAI). In many sports, mental preparation for performance involves imagery, the cognitive skill of rehearsing the task in one's mind. Imagery is however a difficult cognitive skill, which is why we propose the CAI approach, i.e., u...
Conference Paper
Computers can now simulate simple game physics systems hundreds of times faster than real-time, which enables real-time prediction and visualization of the effects of player actions. Predictive simulation is traditionally used as part of planning and game AI algorithms; we argue that it presents untapped potential for game mechanics and interfaces....
Article
Full-text available
This paper addresses the problem of offline path and movement planning for wall climbing humanoid agents. We focus on simulating bouldering, i.e. climbing short routes with diverse moves, although we also demonstrate our system on a longer wall. Our approach combines a graph-based highlevel path planner with low-level sampling-based optimization of...
Conference Paper
Efficient learning of 3D character control still remains an open problem despite of the remarkable recent advances in the field. We propose a new algorithm that combines planning by a samplingbased model-predictive controller and learning from the planned control, which is very noisy. We combine two methods of learning: 1) immediate but imprecise n...
Conference Paper
Virtual camera design is an important but tricky part of creating virtual reality experiences; interaction can feel awkward if the camera is not placed exactly at the user's eyes, but on the other hand a 3rd person perspective (3PP) can provide a better view of the environment and/or the avatar. To inform camera design, we contribute the first stud...
Conference Paper
We present the design and evaluation of the Augmented Climbing Wall (ACW). The system combines computer vision and interactive projected graphics for motivating and instructing indoor wall climbing. We have installed the system in a commercial climbing center, where it has been successfully used by hundreds of climbers, including both children and...
Conference Paper
Full-text available
This paper presents a novel algorithm for real-time path-planning in a dynamic environment such as a computer game. We utilize a real-time sampling approach based on the Rapidly Exploring Random Tree (RRT) algorithm that has enjoyed wide success in robotics. More specifically, our algorithm is based on the RRT* and informed RRT* variants. We contri...
Conference Paper
This paper seeks to expand the understanding of gravity as a powerful but underexplored design resource for movement-based games and play. We examine how gravity has been utilized and manipulated in digital, physical, and mixed reality games and sports, considering five central and gravity-related facets of user experience: realism, affect, challen...
Article
We present a novel, general-purpose Model-Predictive Control (MPC) algorithm that we call Control Particle Belief Propagation (C-PBP). C-PBP combines multimodal, gradient-free sampling and a Markov Random Field factorization to effectively perform simultaneous path finding and smoothing in high-dimensional spaces. We demonstrate the method in onlin...
Article
Motion Games, known as exergames or active video games, have become mainstream, due to technologies such as Microsoft Kinect, PlayStation Move, and Nintendo Wii. More research is needed on how to design and implement motion games that are fun and effective as exercise and motor-learning environments. Researchers have investigated how practicing rea...
Article
In outdoor rock climbing, routes are commonly marked in topos, which are typically implemented as annotated photographs or hand-drawn pictures. 3D technologies, such as structure from motion (SfM), enable the creation of detailed 3D models. These technologies are now within the reach of almost everybody, and they are becoming increasingly simple to...
Article
We present a Model-Predictive Control (MPC) system for online synthesis of interactive and physically valid character motion. Our system enables a complex (36-DOF) 3D human character model to balance in a given pose, dodge projectiles, and improvise a get up strategy if forced to lose balance, all in a dynamic and unpredictable environment. Such co...
Article
The video showcase describes our efforts in developing a novel augmented climbing wall illustrated. Our system combines projected graphics on an artificial climbing wall and body tracking using computer vision technology. The system is intended for accelerating motor skill learning or to make monotonous parts of the training fun by adding relevant...
Article
This paper describes our efforts in developing a novel augmented climbing wall. Our system combines projected graphics on an artificial climbing wall and body tracking using computer vision technology. The system is intended for accelerating motor skill learning or to make monotonous parts of the training fun by adding relevant goals and encouragin...
Article
Video games can empower their players beyond reality, giving them extraordinary abilities. We investigate a novel class of games that provide empowerment in both the real and the virtual world, in this case using a trampoline as part of the human-computer interface. We studied whether novice trampoline jumpers can learn trampolining skills while pl...
Conference Paper
Full-text available
In this paper, we investigate the use of volumetric shadows for enhancing three-dimensional perception and action in third-person motion games. They offer an alternative to previously studied cues and visual guides. Our preliminary survey revealed that from the games that require Kinect, 37% rely primarily on a third-person view and 9% on a first-p...
Conference Paper
This work in progress -paper describes our efforts in developing trampoline training games using computer vision technology. The study is part of a project about developing digitally augmented exercise environments for faster, safer and more engaging sports training. We describe four initial prototypes and the feedback obtained from testing them bo...
Conference Paper
Full-text available
We investigate consciously controlled pupil size as an input modality. Pupil size is affected by various processes, e.g., physical activation, strong emotional experiences and cognitive effort. Our hypothesis is that given continuous feedback, users can learn to control pupil size via physical and psychological self-regulation. We test it by measur...
Article
This thesis explores the possibilities of real-time audiovisual signal processing in the context of human-computer interfaces. Novel embodied interaction applications are presented, including two computer games controlled by body movements and voice, two computer games played by singing, software and an installation for creating clay animation, and...
Article
Full-text available
This paper describes a novel importance sampling method with applications in multimodal optimi-zation. Based on initial results, the method seems suitable for real-time computer vision, and en-ables an efficient frame-by-frame global search with no initialization step. The method is based on importance sampling with adaptive subdivision, developed...
Conference Paper
This paper presents Kick Ass Kung-Fu, a martial arts game installation where the player fights virtual enemies with kicks and punches as well as acrobatic moves such as cartwheels. Using real-time image processing and computer vision, the video image of the user is embedded inside 3D graphics. Compared to previous work, our system uses a profile vi...
Article
The role of children in helping to define and develop the control gestures for computer game, QuiQui's Giant Bounce, through traditional usability tests and Wizard of Oz (WOz) is discussed. The game has three unique characteristics such as use of 2D computer-animated dragon, adding of voice as second input modality, and use of Windows computer equi...
Conference Paper
Full-text available
This paper presents a prototype implementation of a physically interactive fitness game called Shadow Boxer. The game is controlled by body movements and it utilizes a web camera to detect the player's movements. A user study was conducted to study the playability and the effectiveness of the game as an exercising method. The results showed that pl...
Article
This paper describes the use of the Wizard of Oz (WOz) method in the design of computer vision based action games controlled with body movements. A WOz study was carried out with 34 children of ages 7 to 9 in order to find out the most intuitive movements for game controls and to evaluate the relationship between avatar and player actions. Our stud...
Article
This paper reports the results of an experimental study of human latency tolerance for gestural sound control without tactile feedback. 16 subjects played a Theremin that was routed through an adjustable delay buffer. It was found that when comparing to a reference with no latency, the just noticeable difference (JND) is between 20 and 30 milliseco...
Conference Paper
This paper studies gesture and speech controlled video for sports training. The goal is to combine the benefits of recording your performance with video equipment and training with a mirror. For example, a delayed camera view projected on a screen can be used to repeatedly perform and evaluate a spin kick, a move that is difficult to practice with...
Conference Paper
This paper describes Animaatiokone, an installation for experimenting and learning about stop-motion animation. Located in a movie theater, it allows people to create clay animation while waiting for a movie. Collaboration between users is supported, for example, by sharing of clay actors. The installation's user interface allows even beginners to...
Article
This paper presents a novel approach to usability evaluation with children called peer tutoring. Peer tutoring means that children teach other children to use the software that is evaluated. The basic philosophy behind this is to view software as a part of child's play, so that the teaching process is analogous to explaining the rules of a game suc...
Article
In this paper, we present two game tasks (flying game and spider game) in QuiQui's Giant Bounce, a computer game based on computer vision and hearing technology. The game consists of several game tasks with different themes of movement. The tasks presented here have been designed and tested with children [2] and experts in the fields of children's...
Conference Paper
In this paper, we present two game tasks (flying game and spider game) in QuiQui's Giant Bounce, a computer game based on computer vision and hearing technology. The game consists of several game tasks with different themes of movement. The tasks presented here have been designed and tested with children [2] and experts in the fields of children's...
Conference Paper
This paper describes the design of a perceptual user interface for controlling a flying cartoon-animated dragon in QuiQui’s Giant Bounce, a physically and vocally interactive computer game for 4 to 9 years old children. The dragon mimics the user’s movements and breathes fire when the user shouts. The game works on a PC computer equipped with pract...
Article
Although voice has been used as an input modality in various user interfaces, there are no reports of using pitch of the user's voice for real-time control of computer games. This paper ex-plores pitch-based control for novel games for musical education. Mapping pitch to the position of a game character provides visual feedback that helps you to le...
Article
This paper studies the reduction of nonlinearity of digital peak lim-iters used for maximizing signal levels. The goal is to control the time-varying gain smoothly enough to avoid frequency artifacts in the output signal. Smoother gain control is traditionally obtained by lowpass Þltering the gain or the signal envelope. However, sim-ple Þltering c...
Article
The paper reports results from an experimental study quantifying how latency affects the playing accuracy of two continuous sound instruments. 11 subjects played a conventional Theremin and a virtual reality Theremin. Both instruments provided the user only audio feedback. The subjects performed two tasks under different instrument latencies. They...
Article
My thesis deals with multimodal human-computer interfaces using cameras and microphones as input devices. The video and audio material can either be exploited in new ways or the computer can perceive the user through computer vision and hearing technology. The goal of the research is to develop new educational interaction methods that develop the u...

Citations

... Other studies have incorporated reinforcement learning and evolutionary methods to create AI agents that can automatically play games [6,25,55]. These AI agents can be further employed to perform automated game testing sessions [4,16,17,46,58]. Our work is different from those listed above, as we focus on assisting game developers by providing an approach to efficiently search large repositories of gameplay videos to find bug instances. ...
... The muscle also has low-pass filter characteristics, making the control problem hard for classical approaches-in addition to the typical redundancy of having more muscles than DoF. In all our experiments, we use the MuJoCo internal muscle model, which approximates these characteristics and has been used for other muscle-based studies [17,11,23,24]. One limitation consists in the non-elasticity of the tendon. ...
... To open up the promising field of HCI in esports and bring together researchers and industry professionals from different disciplines, an interdisciplinary consortium including Simone Kriglstein, Anna Lisa Martin-Niedecken, Laia Turmo Vidal, Madison Klarkowski, Katja Rogers, Selen Turkay, Magy Seif El-Nasr, Elena Márquez Segura, Anders Drachen, and Perttu Hämäläinen organized and established a special interest group at CHI 2021 [4]. ...
... A great deal of effort has gone into comprehending and explaining the contents of games, as well as how they relate to one another. It's difficult to give a definite description to a process or structure that is always changing (Ang and Zaphiris, 2008;Avedon and Sutton-Smith, 2015;Cardona-Rivera, Zagal, and Debus, 2020;Costikyan, 2002;Juul, 2001;Kevin Maroney, 2011;Myers and Mertz, 1998;Paddick, 1979;Roohi et al., 2020;Tavinor, 2008;Tekinbaş and Zimmerman, 2003;Whitton, 2014). Because of the game's format, it's an activity that appeals to a variety of audiences at various times. ...
... Technologies of multiple perspectives integration have been extensively studied to improve user's navigation [29], collaboration [34], and manipulation performance [23]. World-in-miniature (WIM), which can integrate the 1PP and 3PP simultaneously into a single interface with considerable interaction performance [14], is one of the most sound integration technologies. However, there was no study found in applying WIM in the VR-aided assembly assessment. ...
... There is evidence that humans use such strategy across domains, such as in causal reasoning [20] or perception [31]. In human-computer interaction, researchers have leveraged RL to automate the sequence of user actions in a KLM framework [49] or to predict fatigue in volumetric movements [11]. It was also used to explain search behavior in user interfaces [83] or menus [13] and as a model for multitasking [41]. ...
... Babadi et al. [BNH19] later leveraged Rajamäki and Hämäläinen's approach to train a DRL network that directly predicts the next action, without looking ahead, as most models in this section do. The Monte Carlo Tree Search approach is used to quickly produce adequate reference motions. ...
... Thus, Zwift is not comparable with exergames which add additional game elements to an existing activity, e.g. [40]. Consequently, we argue that Zwift cannot be classified as an exergame, but rather as a virtual sports experience. ...
... Word and emote embedding vectors are averaged for the chat content model used for input features. For the visual content model, we extract features for each frame using the Visual Geometry Group network(VGGnet) [46], which has been widely used in the recent literature as in [47,48], with 16 layers that were pre-trained on ImageNet and averaged the features for logistic regression input. ...
... This decline in interest was accompanied by a decline in usage of approximately 40%. A study of exergame play by 70 children over 6 weeks showed that adherence to the program was predicted by the children's sense of social belonging (Kaos et al., 2019), implying that to be successful, exergames should foster social interaction. ...