Jesse Hoey

Jesse Hoey
University of Waterloo | UWaterloo · David R. Cheriton School of Computer Science

PhD

About

214
Publications
52,622
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
7,643
Citations
Introduction
Dr. Jesse Hoey is an associate professor at the University of Waterloo, and an adjunct scientist at the Toronto Rehabilitation Institute in Toronto. His research focuses on decision-theoretic planning in large scale real-world uncertain domains. He also works on affective intelligence based on socio-psychological models. He applies these ideas primarily in the development of systems that help persons with a cognitive disability (e.g. Alzheimer's disease).
Additional affiliations
January 2010 - present
Toronto Rehabilitation Institute
Position
  • Adjunct Scientist
September 2014 - June 2015
July 2013 - present
University of Waterloo
Position
  • Professor (Associate)

Publications

Publications (214)
Article
Full-text available
Self-supervised pretraining has been observed to be effective at improving feature representations for transfer learning, leveraging large amounts of unlabelled data. This review summarizes recent research into its usage in X-ray, computed tomography, magnetic resonance, and ultrasound imaging, concentrating on studies that compare self-supervised...
Preprint
Self-supervised pretraining has been observed to be effective at improving feature representations for transfer learning, leveraging large amounts of unlabelled data. This review summarizes recent research into its usage in X-ray, computed tomography, magnetic resonance, and ultrasound imaging, concentrating on studies that compare self-supervised...
Preprint
In this study, we investigated whether self-supervised pretraining could produce a neural network feature extractor applicable to multiple classification tasks in B-mode lung ultrasound analysis. When fine-tuning on three lung ultrasound tasks, pretrained models resulted in an improvement of the average across-task area under the receiver operating...
Preprint
Self-supervised pretraining has been observed to improve performance in supervised learning tasks in medical imaging. This study investigates the utility of self-supervised pretraining prior to conducting supervised fine-tuning for the downstream task of lung sliding classification in M-mode lung ultrasound images. We propose a novel pairwise relat...
Chapter
Full-text available
Humanity faces multiple existential risks in the coming decades due to technological advances in AI, and the possibility of unintended behaviors emerging from such systems. We believe that better outcomes may be possible by rigorously exploring frameworks for intelligent (goal-oriented) behavior inspired by computational neuroscience. Here, we expl...
Article
Full-text available
In this study, we investigated whether self-supervised pretraining could produce a neural network feature extractor applicable to multiple classification tasks in B-mode lung ultrasound analysis. When fine-tuning on three lung ultrasound tasks, pretrained models resulted in an improvement of the average across-task area under the receiver operating...
Article
Background: Persons living with dementia and their care partners place a high value on aging in place and maintaining independence. Socially assistive robots - embodied characters or pets that provide companionship and aid through social interaction - are a promising tool to support these goals. There is a growing commercial market for these devic...
Preprint
Full-text available
Humanity faces multiple existential risks in the coming decades due to technological advances in AI, and the possibility of unintended behaviors emerging from such systems. We believe that better outcomes may be possible by rigorously exploring frameworks for intelligent (goal-oriented) behavior inspired by computational neuroscience. Here, we expl...
Preprint
We present our preliminary work on a multi-agent system involving the complex human phenomena of identity and dynamic teams. We outline our ongoing experimentation into understanding how these factors can eliminate some of the naive assumptions of current multi-agent approaches. These include a lack of complex heterogeneity between agents and uncha...
Article
Full-text available
Introduction: Socially assistive robots are devices designed to aid users through social interaction and companionship. Social robotics promise to support cognitive health and aging in place for older adults with and without dementia, as well as their care partners. However, while new and more advanced social robots are entering the commercial mar...
Article
Full-text available
Bayesian affect control theory is a model of affect-driven social interaction under conditions of uncertainty. In this paper, we investigate how the operationalization of uncertainty in the model can be related to the disruption of social orders—societal pressures to adapt to ongoing environmental and technological change. First, we study the theor...
Article
Social research highlights the stability of cultural beliefs, broadly arguing that population-level changes are uncommon and mostly explained by cohort replacement rather than individual-level change. We find evidence suggesting that cultural change may also occur rapidly in response to an economically and socially transformative period. Using data...
Article
Full-text available
Abstract Introduction: In this paper we study the support needed by professional caregivers of those with dementia, and present a first step towards development of VIPCare, a novel application with the goal of assisting new caregivers at care-centres in interacting with residents with dementia. Methods: A mixed-methods study including 2 questionnai...
Preprint
Full-text available
One's ability to learn a generative model of the world without supervision depends on the extent to which one can construct abstract knowledge representations that generalize across experiences. To this end, capturing an accurate statistical structure from observational data provides useful inductive biases that can be transferred to novel environm...
Preprint
Theoretical and Empirical Modeling of Identity and Sentiments in Collaborative Groups (THEMIS.COG) was an interdisciplinary research collaboration of computer scientists and social scientists from the University of Waterloo (Canada), Potsdam University of Applied Sciences (Germany), and Dartmouth College (USA). This white paper summarizes the resul...
Article
Multiagent Resource Allocation (MARA) distributes a set of resources among a set of intelligent agents in order to respect the preferences of the agents and to maximize some measure of global utility, which may include minimizing total costs or maximizing total return. We are interested in MARA solutions that provide optimal or close-to-optimal all...
Preprint
Full-text available
This paper presents the design of a cooperative multi-player betting game, Trust-ya, as a model of some elements of status processes in human groups. The game is designed to elicit status-driven leader-follower behaviours as a means to observe and influence social hierarchy. It involves a Bach/Stravinsky game of deference in a group, in which peopl...
Article
Intelligent assistive robots can enhance the quality of life of people with dementia and their caregivers. They can increase the independence of older adults, reduce tensions between a person with dementia and their caregiver, and increase social engagement. This article provides a review of assistive robots designed for and evaluated by persons wi...
Article
In this paper, we analyze the performance of an agent developed according to a well-accepted appraisal theory of human emotion with respect to how it modulates play in the context of a social dilemma. We ask if the agent will be capable of generating interactions that are considered to be more human than machine-like. We conducted an experiment wit...
Article
Full-text available
This article introduces the somatic transform that operationalizes the relation between affect and cognition at the psychological level of analysis by capitalizing on the relation between the cognitive-denotative and affective-connotative meaning of concepts as measured with semantic differential rating scales. Following discussion of levels of ana...
Chapter
We review Affect Control Theory (ACT) as a promising basis for equipping computational agents in social simulations with a sense of sociality. ACT is a computational theory that integrates sociological insights about the symbolic construction of the social order with psychological knowledge about cognitive-affective mechanisms. After explaining the...
Preprint
Full-text available
Machine learning has the potential to aid in mitigating the human effects of climate change. Previous applications of machine learning to tackle the human effects in climate change include approaches like informing individuals of their carbon footprint and strategies to reduce it. For these methods to be the most effective they must consider releva...
Preprint
In this paper, we develop a taxonomy of relevant models and proxy data for emotional expression and outline how the combinations and permutations of these models and data impact artificial intelligence (AI) systems deploying them. We should not take computer scientists at their word that the paradigms for human emotions they have developed internal...
Preprint
I argue that the management of uncertainty by agents in a social world is foundational to the formation of social structures and to the definition of culture. I present a deep Bayesian model for this management of uncertainty in intelligent systems, and I argue for its applicability to cultural sociology. As social systems grow more heterogeneous,...
Article
For conversational AI and virtual assistants to communicate with humans in a realistic way, they must exhibit human characteristics such as expression of emotion and personality. Current attempts toward constructing human-like dialogue agents have presented significant difficulties. We propose Human Level Attributes (HLAs) based on tropes as the ba...
Preprint
Full-text available
State-of-the-art neural dialogue systems excel at syntactic and semantic modelling of language, but often have a hard time establishing emotional alignment with the human interactant during a conversation. In this work, we bring Affect Control Theory (ACT), a socio-mathematical model of emotions for human-human interactions, to the neural dialogue...
Article
The self has long been construed as a rational, cognitive construct; the cognitive decline of dementia has therefore been largely viewed as the loss of self. Through qualitative interviews, we find that persons with dementia strive to maintain a coherent self despite their increasing disability. Using the theories of affect control theory (ACT) and...
Preprint
Many complex real-world problems, such as climate change mitigation, are intertwined with human social factors. Climate change mitigation, a social dilemma made difficult by the inherent complexities of human behavior, has an impact at a global scale. We propose applying multi-agent reinforcement learning (MARL) in this setting to develop intellige...
Article
In this paper, we examine the influence of personality traits of developers on the pull request evaluation process in GitHub. We first replicate Tsay et al. ’s work that examined the influence of social factors (e.g., ‘social distance’) and technical factors (e.g., test file inclusion) for evaluating contributions, and then extend it with persona...
Article
Computational analyses of data pertaining to human emotional expression have a surprisingly long history and an increasingly critical role in social machine learning (ML) and artificial intelligence (AI) applications. Contemporary, quotidian, narrow AI/ML technologies are most frequently used by social media platforms for modeling and predicting hu...
Preprint
Full-text available
For conversational AI and virtual assistants to communicate with humans in a realistic way, they must exhibit human characteristics such as expression of emotion and personality. Current attempts toward constructing human-like dialogue agents have presented significant difficulties. We propose Human Level Attributes (HLAs) based on tropes as the ba...
Preprint
Full-text available
In this paper, we present a novel method for measurably adjusting the semantics of text while preserving its sentiment and fluency, a task we call semantic text exchange. This is useful for text data augmentation and the semantic correction of text generated by chatbots and virtual assistants. We introduce a pipeline called SMERTI that combines ent...
Preprint
The presence of artificial agents in human social networks is growing. From chatbots to robots, human experience in the developed world is moving towards a socio-technical system in which agents can be technological or biological, with increasingly blurred distinctions between. Given that emotion is a key element of human interaction, enabling arti...
Article
Full-text available
Background: Technology has multiple potential applications to dementia from diagnosis and assessment to care delivery and supporting ageing in place. Objectives: To summarise key areas of technology development in dementia and identify future directions and implications. Method: Members of the US Alzheimer's Association Technology Professional...
Chapter
Full-text available
Alzheimer’s disease is characterized by the gradual loss of memory, ultimately progressing to forgetting who one is and has been. From a symbolic interactionist perspective, the progression of this disease raises the question of what happens to the “self” as part of an interactive social process with others. Our exploratory study of elders with mil...
Conference Paper
Full-text available
We analyzed the performance of an agent based on an appraisal theory of human emotion with respect to how it modulates play in a social dilemma game. An experiment with 117 participants showed how the agent was rated on dimensions of Human-Uniqueness (HU), separating humans from animals, and Human-Nature (HN), separating humans from machines. We sh...
Preprint
In this paper, we analyze the performance of an agent developed according to a well-accepted appraisal theory of human emotion with respect to how it modulates play in the context of a social dilemma. We ask if the agent will be capable of generating interactions that are considered to be more human than machine-like. We conduct an experiment with...
Chapter
Full-text available
Despite advances in gait analysis tools, including optical motion capture and wireless electrophysiology, our understanding of human mobility is largely limited to controlled conditions in a clinic and/or laboratory. In order to examine human mobility under natural conditions, or the ‘wild’, this paper presents a novel markerless model to obtain ga...
Conference Paper
The computational modeling of groups requires models that connect micro-level with macro-level processes and outcomes. Recent research in computational social science has started from simple models of human behaviour, and attempted to link to social structures. However, these models make simplifying assumptions about human understanding of culture...
Article
Recent advances in artificial intelligence and computer science can be used by social scientists in their study of groups and teams. Here, we explain how developments in machine learning and simulations with artificially intelligent agents can help group and team scholars to overcome two major problems they face when studying group dynamics. First,...
Article
Introduction: Technology interventions are showing promise to assist persons with dementia and their carers. However, low adoption rates for these technologies and ethical considerations have impeded the realization of their full potential. Methods: Building on recent evidence and an iterative framework development process, we propose the concep...
Article
Full-text available
Cognitive function is an important end point of treatments in dementia clinical trials. Measuring cognitive function by standardized tests, however, is biased toward highly constrained environments (such as hospitals) in selected samples. Patient-powered real-world evidence using information and communication technology devices, including environme...
Article
Current theories of occupational status conceptualize it as either a function of cultural esteem or the symbolic aspect of the class structure. Based on Weber’s definition of status as rooted in either cultural or class conditions, we argue that a consistent operationalization of occupational status must account for both of these dimensions. Using...
Chapter
Existing neural conversational models process natural language primarily on a lexico-syntactic level, thereby ignoring one of the most crucial components of human-to-human dialogue: its affective content. We take a step in this direction by proposing three novel ways to incorporate affective/emotional aspects into long short term memory (LSTM) enco...
Article
The adoption and effectiveness of cognitive assistive technologies hinge on harnessing the dynamics of human emotion. The authors discuss seminal advances in the integration of emotions in assistive technologies for dementia and propose Bayesian Affect Control Theory (BayesACT), a quantitative social-psychological theory, to model behavior and emot...
Conference Paper
Full-text available
Research in automatic affect recognition has come a long way. This paper describes the fifth Emotion Recognition in the Wild (EmotiW) challenge 2017. EmotiW aims at providing a common benchmarking platform for researchers working on different aspects of affective computing. This year there are two sub-challenges: a) Audio-video emotion recognition...
Conference Paper
This paper proposes a goal recognition and planning algorithm, HTN-GRP-PO, to enable intelligent assistant agents to recognize older adults’ goals and reason about desired further steps. It will be used in a larger system aimed to help older adults with cognitive impairments to accomplish activities of daily living independently. The algorithm addr...
Conference Paper
Smart homes have long been proposed as a viable mechanism to promote independent living for older adults in the home environment. Despite tremendous progress on the technology front, there has been limited uptake by end-users. A critical barrier to the adoption of smart home technology by older adults is the lack of engagement of end-users in the d...
Conference Paper
Pervasive intelligent assistive technologies promise to alleviate some of the increasing burden of care for persons with age-related cognitive disabilities, such as Alzheimer's disease. However, despite tremendous progress, many attempts to develop and implement real world applications have failed to become widely adopted. In this talk, I will argu...
Article
Full-text available
Existing neural conversational models process natural language primarily on a lexico-syntactic level, thereby ignoring one of the most crucial components of human-to-human dialogue: its affective content. We take a step in this direction by proposing three novel ways to incorporate affective/emotional aspects into long short term memory (LSTM) enco...
Book
Notions of identity and of the self have long been studied in social psychology and sociology as key guiding elements of social interaction and coordination. In the AI of the future, these notions will also play a role in producing natural, socially appropriate artificially intelligent agents that encompass subtle and complex human social and affec...
Article
In this paper, we propose an extension to graph-based sentiment lexicon induction methods by incorporating distributed and semantic word representations in building the similarity graph to expand a three-dimensional sentiment lexicon. We also implemented and evaluated the label propagation using four different word representations and similarity me...
Article
Affect Control Theory (ACT) is a powerful and general sociological model of human affective interaction. ACT provides an empirically derived mathematical model of culturally shared sentiments as heuristic guides for human decision making. BayesACT, a variant on classical ACT, combines affective reasoning with cognitive (denotative or logical) reaso...
Article
Full-text available
Identification of falls while performing normal activities of daily living (ADL) is important to ensure safety and well-being of an individual. However, falling is a short term activity that occurs rarely and infrequently. This poses a challenge to traditional classification algorithms, because there may be very little training data for falls (or n...
Article
Full-text available
A fall is an abnormal activity that occurs rarely; however, missing to identify falls can have serious health and safety implications on an individual. Due to the rarity of occurrence of falls, there may be insufficient or no training data available for them. Therefore, standard supervised machine learning methods may not be directly applied to han...
Article
Full-text available
Our overall aim is to develop an emotionally intelligent cognitive assistant (ICA) to help older adults with Alzheimer's disease (AD) to complete activities of daily living more independently. For improved adoption, such a system should take into account how individuals feel about who they are. This paper investigates different affective identities...
Conference Paper
Full-text available
This paper discusses the baseline for the Emotion Recognition in the Wild (EmotiW) 2016 challenge. Continuing on the theme of automatic affect recognition `in the wild', the EmotiW challenge 2016 consists of two sub-challenges: an audio-video based emotion and a new group-based emotion recognition sub-challenges. The audio-video based sub-challenge...
Conference Paper
This paper proposes a computer vision based pipeline for inferring the perceived personality of users from their Twitter profile images. We humans make impressions on a daily basis during communication. The perception of personality of a person gives information about the person’s behaviour and is an important attribute in developing rapport. The p...
Article
Drawing on Bayesian probability theory, we propose a generalization of affect control theory (BayesACT) that better accounts for the dynamic fluctuation of identity meanings for self and other during interactions, elucidates how people infer and adjust meanings through social experience, and shows how stable patterns of interaction can emerge from...
Conference Paper
Symbolic interactionist principles of sociology are based on the idea that human action is guided by culturally shared symbolic representations of identities, behaviours, situations and emotions. Shared linguistic, paralinguistic, or kinesic elements allow humans to coordinate action by enacting identities in social situations. Structures of identi...
Preprint
Full-text available
A fall is an abnormal activity that occurs rarely; however, missing to identify falls can have serious health and safety implications on an individual. Due to the rarity of occurrence of falls, there may be insufficient or no training data available for them. Therefore, standard supervised machine learning methods may not be directly applied to han...
Conference Paper
Full-text available
Automated systems to report falls have long been sought. However, it is very difficult to train classifiers for falls as these are rare events that are difficult to gather training data for. Further, the costs associated with false alarms and misses are not very well known or understood. In this paper, we present a decision-theoretic framework to f...
Article
Full-text available
Introduction: Information and communication technology (ICT) is potentially mature enough to empower outdoor and social activities in dementia. However, actual ICT-based devices have limited functionality and impact, mainly limited to safety. What is an ideal operational framework to enhance this field to support outdoor and social activities? Me...
Article
This paper describes a novel method for building affectively intelligent human-interactive agents. The method is based on a key sociological insight that has been developed and extensively verified over the last twenty years, but has yet to make an impact in artificial intelligence. The insight is that resource bounded humans will, by default, act...
Conference Paper
Full-text available
As sensing and actuation technologies grow more widespread, smart home infrastructures will become both feasible and flexible in supporting multiple applications. The development of these " smart home technologies " have been investigated by diverse fields spanning technical, sociological, and health-oriented disciplines, attempting to meet varying...
Conference Paper
Full-text available
As sensing and actuation technologies grow more widespread, smart home infrastructures will become both feasible and flexible in supporting multiple applications. The development of these "smart home technologies" have been investigated by diverse fields spanning technical, sociological, and health-oriented disciplines, attempting to meet varying u...
Article
Notions of identity and of the self have long been studied in social psychology and sociology as key guiding elements of social interaction and coordination. In the AI of the future, these notions will also play a role in producing natural, socially appropriate artificially intelligent agents that encompass subtle and complex human social and affec...
Article
Full-text available
Affect Control Theory (ACT) is a mathematical model that makes accurate predictions about human behaviour across a wide range of settings. The predictions, which are derived from statistics about human actions and identities in real and laboratory environments, are shared prescriptive and affective behaviours that are believed to lead to solutions...
Technical Report
Full-text available
Affect Control Theory (ACT) is a mathematically well-defined model that makes accurate predictions about the affective content of human action. The affective predictions, which are derived from statistics about human actions and identities in real and laboratory environments, are shared normative behaviours that are believed to lead to solutions to...