John Francis Canny

John Francis Canny
University of California, Berkeley | UCB · Department of Electrical Engineering and Computer Sciences

About

290
Publications
154,902
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
44,063
Citations
Citations since 2016
18 Research Items
14324 Citations
201620172018201920202021202205001,0001,5002,000
201620172018201920202021202205001,0001,5002,000
201620172018201920202021202205001,0001,5002,000
201620172018201920202021202205001,0001,5002,000

Publications

Publications (290)
Preprint
While there have been significant gains in the field of automated video description, the generalization performance of automated description models to novel domains remains a major barrier to using these systems in the real world. Most visual description methods are known to capture and exploit patterns in the training data leading to evaluation me...
Preprint
The design process of user interfaces (UIs) often begins with articulating high-level design goals. Translating these high-level design goals into concrete design mock-ups, however, requires extensive effort and UI design expertise. To facilitate this process for app designers and developers, we introduce three deep-learning techniques to create lo...
Chapter
Automatic video captioning aims to train models to generate text descriptions for all segments in a video, however, the most effective approaches require large amounts of manual annotation which is slow and expensive. Active learning is a promising way to efficiently build a training set for video captioning tasks while reducing the need to manuall...
Conference Paper
Sketching is an effective communication medium that augments and enhances what can be communicated in text. We introduce Sketchforme, the first neural-network-based system that can generate complex sketches based on text descriptions specified by users. Sketchforme's key contribution is to factor complex sketch rendering into layout and rendering s...
Conference Paper
Sketches and real-world user interface examples are frequently used in multiple stages of the user interface design process. Unfortunately, finding relevant user interface examples, especially in large-scale datasets, is a highly challenging task because user interfaces have aesthetic and functional properties that are only indirectly reflected by...
Preprint
Sketching and natural languages are effective communication media for interactive applications. We introduce Sketchforme, the first neural-network-based system that can generate sketches based on text descriptions specified by users. Sketchforme is capable of gaining high-level and low-level understanding of multi-object sketched scenes without bei...
Article
Modern datasets and models are notoriously difficult to explore and analyze due to their inherent high dimensionality and massive numbers of samples. Existing visualization methods which employ dimensionality reduction to two or three dimensions are often inefficient and/or ineffective for these datasets. This paper introduces t-SNE-CUDA, a GPU-acc...
Preprint
The internal states of most deep neural networks are difficult to interpret, which makes diagnosis and debugging during training challenging. Activation maximization methods are widely used, but lead to multiple optima and are hard to interpret (appear noise-like) for complex neurons. Image-based methods use maximally-activating image regions which...
Conference Paper
Full-text available
We present a novel Metropolis-Hastings method for large datasets that uses small expected-size mini-batches of data. Previous work on reducing the cost of Metropolis-Hastings tests yields only constant factor reductions versus using the full dataset for each sample. Here we present a method that can be tuned to provide arbitrarily small batch sizes...
Article
Full-text available
Surgical debridement is the process of removing dead or damaged tissue to allow the remaining parts to heal. Automating this procedure could reduce surgical fatigue and facilitate teleoperation, but doing so is challenging for Robotic Surgical Assistants (RSAs) such as the da Vinci Research Kit (dVRK) due to inherent non-linearities in cable-driven...
Conference Paper
Machine learning is growing in importance in industry, sciences, and many other fields. In many and perhaps most of these applications, users need to trade off competing goals. Machine learning, however, has evolved around the optimization of a single, usually narrowly-defined criterion. In most cases, an expert makes (or should be making) trade-of...
Conference Paper
Full-text available
We introduce Inquire, a tool designed to enable qualitative exploration of utterances in social media and large-scale texts. As opposed to keyword search, Inquire allows the effective use of sentences as queries to quickly explore millions of documents to retrieve semantically-similar sentences. We apply Inquire to LiveJournal.com (LJ) database, wh...
Conference Paper
Full-text available
This demo presents an instance of Inquire, a tool designed to support qualitative researchers in the early stages of research. The tool enables the search over millions of users' records to extract early insights to aid in the formulation of research strategies. The tool presents the work described in the Inquire paper by Paredes, et. al. [12] in t...
Article
Full-text available
We present a novel Metropolis-Hastings method for large datasets that uses small expected-size minibatches of data. Previous work on reducing the cost of Metropolis-Hastings tests yield variable data consumed per sample, with only constant factor reductions versus using the full dataset for each sample. Here we present a method that can be tuned to...
Conference Paper
Full-text available
We fuse science and design thinking to create a novel, IoT interactive urban lights system focused on increasing positive affect among pedestrians. Our contributions are three-fold. First, the design, construction, and evaluation of an efficient interactive lighting system focused on well-being, as opposed to systems focused on utility or landscapi...
Article
A fundamental task in machine learning and related fields is to perform inference on Bayesian networks. Since exact inference takes exponential time in general, a variety of approximate methods are used. Gibbs sampling is one of the most accurate approaches and provides unbiased samples from the posterior but it has historically been too expensive...
Conference Paper
Full-text available
Little is known about the affective expressivity of multisensory stimuli in wearable devices. While the theory of emotion has referenced single stimulus and multisensory experiments, it does not go further to explain the potential effects of sensorial stimuli when utilized in combination. In this paper, we present an analysis of the combinations of...
Conference Paper
Gibbs sampling is a workhorse for Bayesian inference but has several limitations when used for parameter estimation, and is often much slower than non-sampling inference methods. SAME (State Augmentation for Marginal Estimation) [15, 8] is an approach to MAP parameter estimation which gives improved parameter estimates over direct Gibbs sampling. S...
Conference Paper
Heart rate monitoring is widely used in clinical care, fitness training, and stress management. However, tracking individuals' heart rates faces two major challenges, namely equipment availability and user motivation. In this paper, we present a novel technique, LivePulse Games (LPG), to measure users' heart rates in real time by having them play g...
Chapter
Full-text available
In this chapter we investigate practical technologies for security and privacy in data analysis at large scale. We motivate our approach by discussing the challenges and opportunities in light of current and emerging analysis paradigms on large data sets. In particular, we present a framework for privacy-preserving distributed data analysis that is...
Article
Full-text available
Gibbs sampling is a workhorse for Bayesian inference but has several limitations when used for parameter estimation, and is often much slower than non-sampling inference methods. SAME (State Augmentation for Marginal Estimation) \cite{Doucet99,Doucet02} is an approach to MAP parameter estimation which gives improved parameter estimates over direct...
Conference Paper
Allreduce is a basic building block for parallel computing. Our target here is 'Big Data' processing on commodity clusters (mostly sparse power-law data). Allreduce can be used to synchronize models, to maintain distributed datasets, and to perform operations on distributed data such as sparse matrix multiply. We first review a key constraint on cl...
Article
Full-text available
Ensemble methods using the same underlying algorithm trained on different subsets of observations have recently received increased attention as practical prediction tools for massive datasets. We propose Subsemble: a general subset ensemble prediction method, which can be used for small, moderate, or large datasets. Subsemble partitions the full da...
Article
Full-text available
*Honorable Mention for Best Paper Award Stress causes and exacerbates many physiological and mental health problems. Routine and unobtrusive monitoring of stress would enable a variety of treatments, from break-taking to calming exercises. It may also be a valuable tool for assessing effects (frustration, difficulty) of using interfaces or applica...
Article
Early literacy is critical to child development, and determines a child's later educational and life opportunities. Moreover, preschool children are incessantly inquisitive, and will readily engage in question answering and asking activities if given the opportunity. We argue here that question asking/answering technologies can play a major role in...
Article
Heart rate monitoring is widely used in clinical care, fitness training, and stress management. However, tracking individuals' heart rate faces two major challenges, namely equipment availability and user motivation. In this paper, we present a novel technique, LivePulse Games (LPG), to measure users' heart rate in real time by having them play cas...
Conference Paper
Search advertising shows trends of vertical extension. Vertical ads typically offer better Return of Investment (ROI) to advertisers as a result of better user engagement. However, campaign and bids in vertical ads are not set at the keyword level. As a result, the matching between user query and ads suffers low recall rate and the match quality is...
Conference Paper
Full-text available
This paper presents "Deus Ex" a system which uses Behavioral Therapy (CBT). The expected gains are cinematographic fun and increase efficacy by providing a balance of interactivity and narrative. Through highly pervasive media such as mobile phones and digital video discs (DVD), machinima out to populations currently lacking access to this type of...
Conference Paper
Coding style is important to teach to beginning programmers, so that bad habits don't become permanent. This is often done manually at the University level because automated Python static analyzers cannot accurately grade based on a given rubric. However, even manual analysis of coding style encounters problems, as we have seen quite a bit of incon...
Article
Full-text available
Lack of proper English pronunciations is a major problem for immigrant population in developed countries like U.S. This poses various problems, including a barrier to entry into mainstream society. This paper presents a research study that explores the use of speech technologies merged with activity-based and arcade-based games to do pronunciation...
Article
This special issue aims to explore critical elements of the overall design, user experience, and resulting solutions related to using pervasive computing technologies to inform our understanding of the dynamics of ourselves and our ecosystem, community, and urban landscapes. This issue not only explores pervasive technologies but also reviews pract...
Article
Full-text available
Many large datasets exhibit power-law statistics: The web graph, social networks, text data, click through data etc. Their adjacency graphs are termed natural graphs, and are known to be difficult to partition. As a consequence most distributed algorithms on these graphs are communication intensive. Many algorithms on natural graphs involve an Allr...
Conference Paper
Full-text available
This poster presents a theoretical framework for the use of interactive machinima (machine + cinema) as an adaptable means to deliver Cognitive Behavioral Therapy (CBT) to large audiences. The expected gains are to improve engagement and adherence through interactive cinematographic. Theoretical foundations are aggregated in a machinima agent that...
Conference Paper
The preschool ”literacy gap” is one of the most difficult challenges for education in the US. Children in the lowest SES (Socio-Economic Status) quartile have less than half the working vocabulary of those in the top quartile at age 3. On the other hand, preschool children are incessantly inquisitive, and will readily engage in question answering a...
Conference Paper
This paper describes the BID Data Suite, a collection of hardware, software and design patterns that enable fast, large-scale data mining at very low cost. By co-designing all of these elements we achieve single-machine performance levels that equal or exceed reported cluster implementations for common benchmark problems. A key design criterion is...
Article
Full-text available
The goal of this special issue is to contribute to the advancement of ubiquitous information societies, where computers and humans are part of the same ecosystem. One crucial property of entities living in the same ecosystem is that they mutually influence and affect each other's behavior in a variety of ways. This special issue, organized as a fol...
Conference Paper
Full-text available
This paper presents a list of principles that can be used to conceptualize games for health behavior change. These principles are derived from lessons learned after teaching two design-centered courses on Gaming and Narrative Technologies for Health Behavior Change. Course sessions were designed to create many rapid prototypes on specific topics co...
Conference Paper
Full-text available
This paper describes our vision on what should be the research around sensing and adaptive interventions to make affective computing and stress management technology pervasive and unobtrusive. With the use of common computer peripherals and mobile computing devices as affect sensors, personalized and adaptive intervention technologies can be develo...
Conference Paper
Parents are well aware that pre-school children are incessantly inquisitive, and the high ratio of questions to statements suggests that questions are a primary method utilized by children for language acquisition, cognitive development, and formulating knowledge structures. Question-asking is furthermore a comfortable medium for a child to stay en...
Conference Paper
Understanding and facilitating real-life social interaction is a high-impact goal for Ubicomp research. Microphone arrays offer the unique capability to provide continuous, calm capture of verbal interaction in large physical spaces, such as homes and (especially open-plan) offices. Most microphone array work has focused on arrays of custom sensors...
Conference Paper
The human voice encodes a wealth of information about emotion, mood, stress, and mental state. With mobile phones (one of the mostly used modules in body area networks) this information is potentially available to a host of applications and can enable richer, more appropriate, and more satisfying human-computer interaction. In this paper we describ...
Conference Paper
We describe an innovative and scalable recommendation system successfully deployed at eBay. To build recommenders for long-tail marketplaces requires projection of volatile items into a persistent space of latent products. We first present a generative clustering model for collections of unstructured, heterogeneous, and ephemeral item data, under t...
Conference Paper
Full-text available
This paper investigates the usefulness of segmental phonemedynamics for classification of speaking styles. We modeled transition details based on the phoneme sequences emitted by a speech recognizer, using data obtained from a recording of 39 depressed patients with 7 different speaking styles- normal, pressured, slurred, stuttered, flat, slow and...
Conference Paper
Mental illness is one of the most undertreated health problems worldwide. Previous work has shown that there are remarkably strong cues to mental illness in short samples of the voice. These cues are evident in severe forms of illness, but it would be most valuable to make earlier diagnoses from a richer feature set. Furthermore there is an abstrac...
Article
Full-text available
In this paper, we describe our experiences and thoughts on building speech applications on mobile devices for developing countries. We describe three models of use for automatic speech recognition (ASR) systems on mobile devices that are currently used – embedded speech recognition, speech recognition in the cloud, and distributed speech recognitio...
Conference Paper
Rural health workers in India do not always have the training, credibility or motivation to effectively convince clients to adopt healthy practices. To help build their efficacy, we provided them with messages on mobile phones to present to clients. We present a study which compared three presentations of persuasive health messages by health worker...
Article
Behavioral targeting (BT) leverages historical user behavior to select the ads most relevant to users to display. The state-of-the-art of BT derives a linear Poisson regression model from fine-grained user behavioral data and predicts click-through rate (CTR) from user history. We designed and implemented a highly scalable and efficient solution to...
Conference Paper
Full-text available
In this paper we introduce a framework for privacy-preserving distributed computation that is practical for many real-world applications. The framework is called Peers for Privacy (P4P) and features a novel heterogeneous architecture and a number of efficient tools for performing private computation and ensuring security at large scale. It maintain...
Article
Today's youth are shaping the frontier of digital media in general, and mobile technology in particular. This special issue features applications with a youth focus, studies of how youth are appropriating pervasive technology, and a glimpse of how our lives will change when "pervasive" technology finally lives up to its name.
Article
Behavioral targeting (BT) leverages historical user behavior to select the ads most relevant to users to display. The state-of-the-art of BT derives a linear Poisson regression model from fine-grained user behavioral data and predicts click-through rate (CTR) from user history. We designed and implemented a highly scalable and efficient solution to...
Conference Paper
Full-text available
The advancement of precision micropower amplifiers, microcontrollers, and MEMs devices have allowed for a paradigm shift from traditionally large and costly health monitoring equipment only found in hospitals or care centers to smaller, wireless, low powered portable devices that can provide continuous monitoring for a number of applications. Along...
Conference Paper
Full-text available
Cellphones have the potential to improve education for the millions of underprivileged users in the developing world. However, mobile learning in developing countries remains under-studied. In this paper, we argue that cellphones are a perfect vehicle for making educational opportunities accessible to rural children in places and times that are mor...
Conference Paper
Full-text available
In many developing countries such as India and China, low educational levels often hinder economic empowerment. In this paper, we argue that mobile learning games can play an important role in the Chinese literacy acquisition process. We report on the unique challenges in the learning Chinese language, especially its logographic writing system. Bas...
Conference Paper
Full-text available
Researchers have long been interested in the potential of ICTs to enable positive change in developing regions com- munities. In these environments,ICT interventionsoften fail because political, social and cultural forces work against the changes ICTs entail. We argue that familiar uses of ICTs for information services in these contexts are less po...
Conference Paper
Full-text available
Dictionary-based disambiguation (DBD) is a very popular solution for text entry on mobile phone keypads but suffers from two problems: 1. the resolution of encoding collision (two or more words sharing the same numeric key sequence) and 2. entering out-of-vocabulary (OOV) words. In this paper, we present SHRIMP, a system and method that addresses t...
Article
Full-text available
In this paper we present ethno-mining, a mixed methods approach drawing on techniques from ethnography and data mining. Ethno-mining is characterized by tight, iterative loops that integrate both the results and the processes of ethnographic and data mining techniques to interpret data. Ethno-mining provides two key benefits. First, it makes use of...
Conference Paper
Full-text available
We developed and tested the Berkeley Tricorder, a health monitoring device capable of measuring a subject's ECG, EMG, Blood Oxygenation, Respiration (via Bioimpedance), and motion--almost equivalent to the feature set of a hospital bedside patient monitor. Our focus has been a highly integrated design incorporating the radio and all associated circ...
Article
Full-text available
A major problem in current privacy-preserving data-mining research is the lack of practical mechanisms to deal with malicious users who may submit bogus data to bias the computation. In this paper we explore private computation built on vector addition and its applications in privacy-preserving data mining. We show that such a paradigm not only sup...
Conference Paper
Full-text available
Literacy is one of the great challenges in the developing world. But universal education is an unattainable dream for those children who lack access to quality educational resources such as well-prepared teachers and schools. Worse, many of them do not attend school regularly due to their need to work for the family in the agricultural fields or ho...
Conference Paper
Full-text available
Video conferencing attempts to convey subtle cues of face-to-face interaction (F2F), but it is generally believed to be less effective than F2F. We argue that careful design based on an understanding of non-verbal communication can mitigate these differences. In this paper, we study the effects of video image framing in one-on-one meetings on empat...
Conference Paper
Full-text available
Low educational levels hinder economic empowerment in developing countries. We make the case that educational games can impact children in the developing world. We report on exploratory studies with three communities in North and South India to show some problems with digital games that fail to match rural children's understanding of games, to high...