Suranga Nanayakkara

National University of Singapore | NUS · Department of Information Systems & Analytics

Productive Failure (PF) is a learning approach where students initially tackle novel problems targeting concepts they have not yet learned, followed by a consolidation phase where these concepts are taught. Recent application in STEM disciplines suggests that PF can help learners develop more robust conceptual knowledge. However, empirical validati...

Affect and cognitive load influence many user behaviors. In this paper, we propose Motion as Emotion, a novel method that utilizes fine differences in hand motion to recognise affect and cognitive load in virtual reality (VR). We conducted a study with 22 participants who used common free-hand gesture interactions to carry out tasks of varying diff...

Sound morphing is the process of gradually and smoothly transforming one sound into another to generate novel and perceptually hybrid sounds that simultaneously resemble both. Recently, diffusion-based text-to-audio models have produced high-quality sounds using text prompts. However, granularly controlling the semantics of the sound, which is nece...

Emotion-Cause analysis has attracted the attention of researchers in recent years. However, most existing datasets are limited in size and number of emotion categories. They often focus on extracting parts of the document that contain the emotion cause and fail to provide more abstractive, generalizable root cause. To bridge this gap, we introduce...

Spatial awareness, particularly awareness of distant environmental scenes known as vista-space, is crucial and contributes to the cognitive and aesthetic needs of People with Visual Impairments (PVI). In this work, through a formative study with PVIs, we establish the need for vista-space awareness amongst people with visual impairments, and the po...

People with a fear of being in water rarely engage in water activities and hence miss out on the associated health benefits. Prior research suggested virtual exposure to treat fears. However, when it comes to a fear of being in water, virtual water might not capture water's immersive qualities, while real water can pose safety risks. We propose ext...

Controllable generation in StyleGANs is usually achieved by training the model using labeled data. For audio textures, however, there is currently a lack of large semantically labeled datasets. Therefore, to control generation, we develop a method for semantic control over an unconditionally trained StyleGAN in the absence of such labeled datasets....

Hearing loss is a global challenge, especially pronounced in developing countries, posing challenges in daily life, from learning, social interactions to safety. Augmented Reality (AR) holds great promise for bridging the gap in access to spoken words and environmental sounds. However, designing effective, affordable, and localized speech recogniti...

Generative models for synthesizing audio textures explicitly encode controllability by conditioning the model with labelled data. While datasets for audio textures can be easily recorded in-the-wild, semantically labeling them is expensive, time-consuming, and prone to errors due to human annotator subjectivity. Thus, to control generation, there i...

Researchers have used machine learning approaches to identify motion sickness in VR experience. These approaches demand an accurately-labeled, real-world, and diverse dataset for high accuracy and generalizability. As a starting point to address this need, we introduce `VR.net', a dataset offering approximately 12-hour gameplay videos from ten real...

For the DCASE 2023 Task 7 (Track B), Foley Sound Synthesis, we submit two systems, (1) a StyleGAN conditioned on the class ID, and (2) an ensemble of StyleGANs each trained unconditionally on each class separately. We quantitatively find that both systems out-perform the task 7 baseline models in terms of FAD Scores. Given the high inter-class and...

In this paper, we propose a data-driven approach to train a Generative Adversarial Network (GAN) conditioned on "soft-labels" distilled from the penultimate layer of an audio classifier trained on a target set of audio texture classes. We demonstrate that interpolation between such conditions or control vectors provides smooth morphing between the...

Technology integration in educational settings has led to the development of novel sensor-based tools that enable students to measure and interact with their environment. Although reports from using such tools can be positive, evaluations are often conducted under controlled conditions and short timeframes. There is a need for longitudinal data col...

Retrieval Augment Generation (RAG) is a recent advancement in Open-Domain Question Answering (ODQA). RAG has only been trained and explored with a Wikipedia-based external knowledge base and is not optimized for use in other specialized domains such as healthcare and news. In this paper, we evaluate the impact of joint training of the retriever and...

Corneal Surface Reflections, or reflections on our eye-surface, have been shown as a valid and more socially acceptable source of information for passive lifelogging applications by prior work. However, automatic analysis of corneal surface reflections from a single RGB camera to support passive lifelogging is not extensively investigated in prior...

This research aimed to investigate how children with autism interacted with rich audio and visual augmented reality (AR) tabletop games. Based on in-depth needs analysis facilitated through autism centers in China, we designed and developed MagicBLOCKS, a series of tabletop AR interactive games for children with autism. We conducted a four-week fie...

Retrieval Augment Generation (RAG) is a recent advancement in Open-Domain Question Answering (ODQA). RAG has only been trained and explored with a Wikipedia-based external knowledge base and is not optimized for use in other specialized domains such as healthcare and news. In this paper, we evaluate the impact of joint training of the retriever and...

Emotional Self-Awareness (ESA) plays a vital role in physical and mental well-being. Recent advancements in artificial intelligence technologies have shown promising emotion recognition results, opening new opportunities to build systems to support ESA. However, little research has been done to understand users' perspectives on artificial-intellige...

Micro-learning [1], cognitive training, and even memory training [2] are a few examples of applications that have been made easier-to-use and more available with mobile, wearable, and ubiquitous technologies. Many of them are also presented through conversational interfaces. Current systems for learning and training focus on providing prompts via n...

Our memories and past experiences contribute to guiding our perception and action of future affective experiences. Virtual Reality (VR) experiences are more vividly memorized and recalled than non-VR ones, but there is little research on how to detect this recall in VR. We investigate the feasibility of recognizing autobiographical memory (AM) reca...

Tickling is a type of sensation that is associated with laughter, smiling, or other similar reactions. Psychology research has shown that tickling and laughter can significantly relieve stress. Although several tickling arti-facts have been suggested in prior work, limited knowledge is available if those artifacts could evoke laughter. In this arti...

We present ANISMA, a software and hardware toolkit to prototype on-skin haptic devices that generate skin deformation stimuli like pressure, stretch, and motion using shape-memory alloys (SMAs). Our toolkit embeds expert knowledge that makes SMA spring actuators more accessible to human–computer interaction (HCI) researchers. Using our software too...

Emotions are multimodal processes that play a crucial role in our everyday lives. Recognizing emotions is becoming more critical in a wide range of application domains such as healthcare, education, human-computer interaction, Virtual Reality, intelligent agents, entertainment, and more. Facial macro-expressions or intense facial expressions are th...

Open-source hardware such as RISC-V has been gaining substantial momentum. Recently, they have begun to embrace Google’s Android operating system to leverage its software ecosystem. Despite the encouraging progress, a challenging issue arises: a majority of Android applications are written in native languages and need to be recompiled to target new...

Emotions are complicated psycho-physiological processes that are related to numerous external and internal changes in the body. They play an essential role in human-human interaction and can be important for human-machine interfaces. Automatically recognizing emotions in conversation could be applied in many application domains like health-care, ed...

Extracting emotions from physiological signals has become popular over the past decade. Recent advancements in wearable smart devices have enabled capturing physiological signals continuously and unobtrusively. However, signal readings from different smart wearables are lossy due to user activities, making it difficult to develop robust models for...

Human-Computer Integration (HInt) is an emerging paradigm in the human-computer interaction (HCI) field. Its goal is to integrate the human body and the computational machine. Because HInt is not an isolated area of research, the authors draw upon discussions from related perspectives, including cybernetics, augmentation, cyborgs, and wearables. Wh...

With voice user interfaces (VUIs) becoming ubiquitous and speech synthesis technology maturing, it is possible to synthesise voices to resemble our friends and relatives (which we will collectively call ‘kin’) and use them on VUIs. However, designing such interfaces and investigating how the familiarity of kin voices affect user perceptions remain...

As social distancing is becoming the new normal, technology holds the potential to bridge this societal gap through novel interaction modalities that allow multiple users to collaborate and create content together. We present Jammify, an interactive multi-sensory system that focuses on providing a unique digital art-jamming experience with a visual...

In this paper, we illustrate how to fine-tune the entire Retrieval Augment Generation (RAG) architecture in an end-to-end manner. We highlighted the main engineering challenges that needed to be addressed to achieve this objective. We also compare how end-to-end RAG architecture outperforms the original RAG architecture for the task of question ans...

In pandemic times, public facilities, such as gymnasiums and other sports facilities are closed. Hence, keeping oneself physically fit becomes particularly difficult as well as getting help from a fitness trainer or instructor is almost impossible. We propose an alternative solution, GymSoles++, a virtual trainer supporting the user with their exer...

Leg length discrepancy (LLD) is common and typically burdens the spine and hip causing a variety of health issues including back pain and headache. Common orthopaedic solutions target correcting gait, typically by shoe lifts. Moreover, functional LLD can be temporary as it occurs after an injury or even after sitting in a twisted posture. Often, it...

After Deep Learning (DL) regained popularity recently, the Artificial Intelligence (AI) or Machine Learning (ML) field is undergoing rapid growth concerning research and real-world application development. Deep Learning has generated complexities in algorithms, and researchers and users have raised concerns regarding the usability and adoptability...

Augmented foot interfaces have been studied since the beginning of wearable computers. The worlds’ first wearable computer was an instrumented shoe that consisted of a toe operated switch with a wireless module. Since then, academic research and commercial products on augmented foot interfaces are booming with novel interfaces every year. This pape...

People with visual impairments (PVI) experience simple tasks, such as grocery shopping, to be an essential difficulty. Although the recent emergence of AI-technology has been dramatically improving visual recognition capabilities, the application to the daily life of PVI is still complex and erroneous. For example, image recognition engines require...

Prospective memory lapses, which involve forgetting to perform intended actions, affect independent living in older adults. Although memory training using smartphone applications could address them, users are sometimes unaware of available times for training or forget about it, presenting a need for proactive prompts. Existing applications mostly p...

Multimodal emotion recognition from speech is an important area in affective computing. Fusing multiple data modalities and learning representations with limited amounts of labeled data is a challenging task. In this paper, we explore the use of modality-specific "BERT-like" pretrained Self Supervised Learning (SSL) architectures to represent both...

It has been shown that combining data from multiple sources, such as observations, self-reports, and performance with physiological markers offers better insights into cognitive-affective states during the learning process. Through a study with 12 kindergartener