Lydia Chilton

• Columbia University

Each type of creative or communicative work is underpinned by an implicit structure. People learn these structures from examples - a process known in cognitive science as schema induction. However, inducing schemas is challenging, as structural patterns are often obscured by surface-level variation. We present Schemex, an interactive visual workflo...

Multi-agent large language model simulations have the potential to model complex human behaviors and interactions. If the mechanics are set up properly, unanticipated and valuable social dynamics can surface. However, it is challenging to consistently enforce simulation mechanics while still allowing for notable and emergent dynamics. We present Ag...

Expertise is often built by learning from examples. This process, known as schema induction, helps us identify patterns from examples. Despite its importance, schema induction remains a challenging cognitive task. Recent advances in generative AI reasoning capabilities offer new opportunities to support schema induction through human-AI collaborati...

While organizations continue to invest in AI tools like M365 Copilot, little is known about how individual employees engage with these technologies once deployed. This study examines M365 Copilot adoption behaviors among a group of 10 experienced users across many industries in the United States. Findings reveal a strong preference for informal lea...

Human prosocial cooperation is essential for our collective health, education, and welfare. However, designing social systems to maintain or incentivize prosocial behavior is challenging because people can act selfishly to maximize personal gain. This complex and unpredictable aspect of human behavior makes it difficult for policymakers to foresee...

Humor is a social binding agent. It is an act of creativity that can provoke emotional reactions on a broad range of topics. Humor has long been thought to be "too human" for AI to generate. However, humans are complex, and humor requires our complex set of skills: cognitive reasoning, social understanding, a broad base of knowledge, creative think...

As AI becomes more capable, it is unclear how human creativity will remain essential in jobs that incorporate AI. We conducted a 14-week study of a student newsroom using an AI tool to convert web articles into social media videos. Most treated the tool as a creative springboard, yet still had to edit many AI outputs. The tool enabled the team to p...

Science communication increases public interest in science by educating, engaging, and encouraging everyday people to participate in the sciences. But traditional science communication is often too formal and inaccessible for general audiences. However, there is a growing trend on social media to make it more approachable using three techniques: re...

Everyone aspires to achieve personal goals. However, getting started is often complex and daunting, especially for large projects. AI has the potential to create plans and help jumpstart progress, but it often lacks sufficient personal context to be useful. We introduce JumpStarter, a system that uses AI-powered context curation to create action pl...

We evaluate recent Large Language Models (LLMs) on the challenging task of summarizing short stories, which can be lengthy, and include nuanced subtext or scrambled timelines. Importantly, we work directly with authors to ensure that the stories have not been shared online (and therefore are unseen by the models), and to obtain informed evaluations...

Recent advancements in large language models have significantly expedited the process of generating front-end code. This allows users to rapidly prototype user interfaces and ideate through code, a process known as exploratory programming. However, existing LLM code-generation tools focus more on technical implementation details rather than finding...

Generative text-to-image models are disrupting the lives of creative professionals. Specifically, illustrators are threatened by models that claim to extract and reproduce their style. Yet, research on style transfer has rarely focused on their perspectives. We provided four illustrators with a model fine-tuned to their style and conducted semi-str...

Large language models (LLMs) are remarkably good at writing code. A particularly valuable case of human-LLM collaboration is code-based UI prototyping, a method for creating interactive prototypes that allows users to view and fully engage with a user interface. We conduct a formative study of GPT Pilot, a leading LLM-generated code-prototyping sys...

Human evaluation has been the gold standard for checking faithfulness in abstractive summarization. However, with a challenging source domain like narrative, multiple annotators can agree a summary is faithful, while missing details that are obvious errors only once pointed out. We therefore introduce a new dataset, STORYSUMM, comprising LLM summar...

Generative AI (Gen-AI) is rapidly changing the landscape of User�Generated Content (UGC) on social media. AI tools for generating text, images, and videos, such as Large-Language Models (LLM), image generation AI, AI-powered video material tools, and deep fake technologies, are accelerating creators in obtaining content ideas, drafting outlines, an...

Communicating science and technology is essential for the public to understand and engage in a rapidly changing world. Tweetorials are an emerging phenomenon where experts explain STEM topics on social media in creative and engaging ways. However, STEM experts struggle to write an engaging "hook" in the first tweet that captures the reader's attent...

How I came to love design and used AI to alleviate the most frustrating parts of the process.

Advances in voice technology and voice user interfaces (VUIs) -- such as Alexa, Siri, and Google Home -- have opened up the potential for many new types of interaction. However, despite the potential of these devices reflected by the growing market and body of VUI research, there is a lingering sense that the technology is still underused. In this...

Communicating science and technology is essential for the public to understand and engage in a rapidly changing world. Tweetorials are an emerging phenomenon where experts explain STEM topics on social media in creative and engaging ways. However, STEM experts struggle to write an engaging "hook" in the first tweet that captures the reader's attent...

Collaborative stories, which are texts created through the collaborative efforts of multiple authors with different writing styles and intentions, pose unique challenges for NLP models. Understanding and generating such stories remains an underexplored area due to the lack of open-domain corpora. To address this, we introduce STORYWARS, a new datas...

Short videos on social media are a prime way many young people find and consume content. News outlets would like to reach audiences through news reels, but currently struggle to translate traditional journalistic formats into the short, entertaining videos that match the style of the platform. There are many ways to frame a reel-style narrative aro...

Visuals are a core part of our experience of music, owing to the way they can amplify the emotions and messages conveyed through the music. However, creating music visualization is a complex, time-consuming, and resource-intensive process. We introduce Generative Disco, a generative AI system that helps generate music visualizations with large lang...

Understanding what constitutes safe text is an important issue in natural language processing and can often prevent the deployment of models deemed harmful and unsafe. One such type of safety that has been scarcely studied is commonsense physical safety, i.e. text that is not explicitly violent and requires additional commonsense knowledge to compr...

Architects and systems designers artfully balance multiple competing design constraints during the design process but are unable to translate between system metrics and end user experience. This work presents three methodologies to fill in this gap. The first is an incentive-compatible methodology that determines a "ground truth" measurement of use...

Multimodal AI advancements have presented people with powerful ways to create images from text. Recent work has shown that text-to-image generations are able to represent a broad range of subjects and artistic styles. However, translating text prompts into visual messages is difficult. In this paper, we address this challenge with Opal, a system th...

Handwriting recognition is improving in leaps and bounds, and this opens up new opportunities for stylus-based interactions. In particular, note-taking applications can become a more intelligent user interface, incorporating new features like autocomplete and integrated search. In this work we ran a gesture elicitation study, asking 21 participants...

AmbiTeam is an ambient display designed to communicate contextual or background information in the periphery of the user's awareness and only requires the user's attention when it is appropriate or desired.

Pop culture is a pervasive and important aspect of communication and self-expression. When people wish to communicate using pop culture references, they need to find connections between their message and the things, people, location and actions of a movie, tv series, or other pop culture domain. However, finding an appropriate match from memory is...

Language models are known to produce vague and generic outputs. We propose two unsupervised decoding strategies based on either word-frequency or point-wise mutual information to increase the specificity of any model that outputs a probability distribution over its vocabulary at generation time. We test the strategies in a prompt completion task; w...

Large-scale language models are rapidly improving, performing well on a wide variety of tasks with little to no customization. In this work we investigate how language models can support science writing, a challenging writing task that is both open-ended and highly constrained. We present a system for generating "sparks", sentences related to a sci...

People are increasingly getting information and news from social media. On Twitter we are seeing the emergence of "tweetorials"- long, explanatory Twitter threads written by experts. In this work we study tweetorials as a form of science writing. While scientists have begun to champion the importance of Twitter as a science communication medium, fe...

Text-to-image generative models are a new and powerful way to generate visual artwork. The free-form nature of text as interaction is double-edged; while users have access to an infinite range of generations, they also must engage in brute-force trial and error with the text prompt when the result quality is poor. We conduct a study exploring what...

Every day we are surrounded by spoken dialog. This medium delivers rich diverse streams of information auditorily; however, systematically understanding dialog can often be non-trivial. Despite the pervasiveness of spoken dialog, automated speech understanding and quality information extraction remains markedly poor, especially when compared to wri...

Social media platforms are deeply ingrained in society, and they offer many different spaces for people to engage with others. Unfortunately, accessibility barriers prevent people with disabilities from fully participating in these spaces. Social media users commonly post inaccessible media, including videos without captions (which are important fo...

Despite the success of style transfer in image processing, it has seen limited progress in natural language generation. Part of the problem is that content is not as easily decoupled from style in the text domain. Curiously, in the field of stylometry, content does not figure prominently in practical methods of discriminating stylistic elements, su...

Images on social media platforms are inaccessible to people with vision impairments due to a lack of descriptions that can be read by screen readers. Providing accurate alternative text for all visual content on social media is not yet feasible, but certain subsets of images, such as internet memes, offer affordances for automatic or semi-automatic...

Social media platforms are deeply ingrained in society, and they ofer many diferent spaces for people to engage with others. Unfortunately, accessibility barriers prevent people with disabilities from fully participating in these spaces. Social media users commonly post inaccessible media, including videos without captions (which are important for...

Visual metaphors are a creative technique used in print media to convey a message through images. This message is not said directly, but implied through symbols and how those symbols are juxtaposed in the image. The messages we see affect our thoughts and lives, and it is an open research challenge to get machines to automatically understand the im...

Writers regularly use a thesaurus to help them write well; the thesaurus is one of the few widespread writing support tools and many writers find it integral to their writing practice. A normal thesaurus is hand-crafted and structured around strict synonymy for a given word sense. However, writers rarely look for a perfectly synonymous word -- inst...

" Traditional approaches for ensuring high quality crowdwork have failed to achieve high-accuracy on difficult problems. Aggregating redundant answers often fails on the hardest problems when the majority is confused. Argumentation has been shown to be effective in mitigating these drawbacks. However, existing argumentation systems only support lim...

Visual blends are an advanced graphic design technique to draw attention to a message. They combine two objects in a way that is novel and useful in conveying a message symbolically. This paper presents VisiBlends, a flexible workflow for creating visual blends that follows the iterative design process. We introduce a design pattern for blending sy...

Creative writing, from poetry to journalism, is at the crux of human ingenuity and social interaction. Existing creative writing support tools produce entire passages or fully formed sentences, but these approaches fail to adapt to the writer's own ideas and intentions. Instead we posit to build tools that generate ideas coherent with the writer's...

Traditional approaches for ensuring high quality crowdwork have failed to achieve high-accuracy on difficult problems. Aggregating redundant answers often fails on the hardest problems when the majority is confused. Argumentation has been shown to be effective in mitigating these drawbacks. However, existing argumentation systems only support limit...

Visual blends are an advanced graphic design technique to draw users' attention to a message. They blend together two objects in a way that is novel and useful in conveying a message symbolically. This demo presents an interactive pipeline for creating visual blends that follows the iterative design process. Our pipeline decomposes the process into...

Effectively planning a large multi-track conference requires an understanding of the preferences and constraints of organizers, authors, and attendees. Traditionally, the onus of scheduling the program falls on a few dedicated organizers. Resolving conflicts becomes difficult due to the size and complexity of the schedule and the lack of insight in...

Creating a good schedule for a large conference such as CHI requires taking into account the preferences and constraints of organizers, authors, and attendees. Traditionally, the onus of planning is placed entirely on the organizers and involves only a few individuals. Cobi presents an alternative approach to conference scheduling that engages the...

The rapidly growing field of collective intelligence - encompassing crowdsourcing, human computation, and social computing - is having a tremendous impact on the way we work, live, and play. Building on the success of a CHI 2012 CrowdCamp, this two-day event focuses on developing ideas into concrete outputs: in-depth thoughts on hard problems, pape...

Crowdsourcing and human computation are transforming human-computer interaction, and CHI has led the way. The seminal publication in human computation was initially published in CHI in 2004 [1], and the first paper investigating Mechanical Turk as a user study platform has amassed over one hundred citations in two years [5]. However, we are just be...

Web search engines have historically focused on connecting people with information resources. For example, if a person wanted to know when their flight to Hyderabad was leaving, a search engine might connect them with the airline where they could find flight status information. However, search engines have recently begun to try to meet people's sea...

This chapter examines the landscape of Web customization, particularly for the Mozilla Firefox. It specifically focuses on browser-hosted customizations, but a serious limitation of this approach is that the user's customizations do not easily move with them, as they use different browsers on different computers. CoScripter has an advantage here, b...

Mechanical Turk (MTurk) provides an on-demand source of human computation. This provides a tremendous opportunity to explore algorithms which incorporate human computation as a function call. However, various systems challenges make this difficult in practice, and most uses of MTurk post large numbers of independent tasks. TurKit is a toolkit for p...

A professor and several PhD students at MIT examine the challenges and opportunities in human computation.

In order to understand how a labor market for human com-putation functions, it is important to know how workers search for tasks. This paper uses two complementary meth-ods to gain insight into how workers search for tasks on Mechanical Turk. First, we perform a high frequency scrape of 36 pages of search results and analyze it by looking at the ra...

Services like Amazon's Mechanical Turk have opened the door for exploration of processes that outsource computation to humans. These human computation processes hold tremendous potential to solve a variety of problems in novel and interesting ways. However, we are only just beginning to understand how to design such processes. This paper explores t...

Crowdsourcing is a form of "peer production" in which work traditionally performed by an employee is outsourced to an "undefined, generally large group of people in the form of an open call." We present a model of workers supplying labor to paid crowdsourcing projects. We also introduce a novel method for estimating a worker's reservation wage--the...

Seaweed is a web application for experimental economists with no programming background to design two-player symmetric games in a visual-oriented interface. Games are automatically published to the web where players can play against each other remotely and game play is logged so that the game's designer can analyze the data. Seaweed's interface dra...