Priyan Vaithilingam’s research while affiliated with Harvard University and other places


Ad

What is this page?


This page lists works of an author who doesn't have a ResearchGate profile or hasn't added the works to their profile yet. It is automatically generated from public (personal) data to further our legitimate goal of comprehensive and accurate scientific recordkeeping. If you are this author and want this page removed, please let us know.

Publications (13)


Figure 4. Grammar for the regex domain
Figure 5. Success rate of the literal L0 and ranking-based Lanneal synthesizers inferring the correct regex as a function of numbers of examples given (turn). Lanneal achieves a success rate of 93.75%, L0 achieves only 65.63%. The ranking-based synthesizer also achieves higher success with fewer utterances. Bands indicate 95% CI over 24 regexes for each condition.
Amortizing Pragmatic Program Synthesis with Rankings
  • Preprint
  • File available

June 2024

·

9 Reads

Yewen Pu

·

Saujas Vaduguru

·

Priyan Vaithilingam

·

[...]

·

Daniel Fried

The usage of Rational Speech Acts (RSA) framework has been successful in building \emph{pragmatic} program synthesizers that return programs which, in addition to being logically consistent with user-generated examples, account for the fact that a user chooses their examples informatively. We present a general method of amortizing the slow, exact RSA synthesizer. Our method first query the exact RSA synthesizer to compile a communication dataset. The dataset contains a number of example-dependent rankings of subsets of programs. It then distills a \textit{single} global ranking of all programs as an approximation to every ranking in the dataset. This global ranking is then used at inference time to rank multiple logically consistent candidate programs generated from a fast, non-pragmatic synthesizer. Experiments on two program synthesis domains using our ranking method resulted in orders of magnitudes of speed ups compared to the exact RSA synthesizer, while being more accurate than a non-pragmatic synthesizer when communicating with humans. Finally, we prove that in the special case of synthesis from a single example, this approximation is exact.

Download





Figure 2: A boolean lexicon for a small reference game of regular expressions. The rows are the utterances (strings) and the columns are hypotheses (regexes), and each entry denotes if a string is consistent with a regex.
Figure 11: Fraction of stable orders that were formed in σ L1 as a function of increasing lexicon size. Points are raw samples (n=100 per lexicon size and Ptrue), bars are 95% bootstrapped CI (nboot = 1000). Overall, increasing Ptrue and lexicon size increases the fraction of stable orders that were formed in σ L1
Figure 13: Two patterns in our layout domain and their corresponding programs, represented as a sequence of production rules: [Program, Shape, Left, Right, Top, Bottom, Thickness, O, I, Colour, A1, A2]. The symbol · indicates rules which only have 1 choice of expansion (Program, Shape, and Colour). The rules where these two programs differ are marked with a box .
Figure 14: User interface for the regex domain
Amortizing Pragmatic Program Synthesis with Rankings

September 2023

·

26 Reads

In program synthesis, an intelligent system takes in a set of user-generated examples and returns a program that is logically consistent with these examples. The usage of Rational Speech Acts (RSA) framework has been successful in building \emph{pragmatic} program synthesizers that return programs which -- in addition to being logically consistent -- account for the fact that a user chooses their examples informatively. However, the computational burden of running the RSA algorithm has restricted the application of pragmatic program synthesis to domains with a small number of possible programs. This work presents a novel method of amortizing the RSA algorithm by leveraging a \emph{global pragmatic ranking} -- a single, total ordering of all the hypotheses. We prove that for a pragmatic synthesizer that uses a single demonstration, our global ranking method exactly replicates RSA's ranked responses. We further empirically show that global rankings effectively approximate the full pragmatic synthesizer in an online, multi-demonstration setting. Experiments on two program synthesis domains using our pragmatic ranking method resulted in orders of magnitudes of speed ups compared to the RSA synthesizer, while outperforming the standard, non-pragmatic synthesizer.


Fig. 1. A simple example of pragmatic communication using rational speech acts
The Usability of Pragmatic Communication in Regular Expression Synthesis

August 2023

·

30 Reads

Programming-by-example (PBE) systems aim to alleviate the burden of programming. However, user-specified examples are often ambiguous, leaving multiple programs to satisfy the specification. Consequently, in most prior work, users have had to provide additional examples, particularly negative ones, to further constrain the search over compatible programs. Recent work resolves additional ambiguity by modeling program synthesis tasks as pragmatic communication, showing promising results on a graphics domain using a rudimentary user-study. We adapt pragmatic reasoning to a sub-domain of regular expressions and rigorously study its usability as a means of communication both with and without the ability to provide negative examples. Our user study (N=30) demonstrates that, with a pragmatic synthesizer, end-users can more successfully communicate a target regex using positive examples alone (95%) compared to using a non-pragmatic synthesizer (51%). Further, users can communicate more efficiently (57% fewer examples) with a pragmatic synthesizer compared to a non-pragmatic one.





Ad

Citations (8)


... customization widgets for data visualizations on an ad-hoc basis [6,10,62]. Another approach is to generate user interface elements that help the user refine specific aspects of the generated artifact (e.g., code) [8]. The latter is an example of a general category of techniques that use graphical interface elements for manipulating prompts, sometimes referred to as "prompt middleware" [36], to ease the cognitive burdens of text entry and manipulation. ...

Reference:

Dynamic Prompt Middleware: Contextual Prompt Refinement Controls for Comprehension Tasks
DynaVis: Dynamically Synthesized UI Widgets for Visualization Editing
  • Citing Conference Paper
  • May 2024

... HCI research has been an important driver of algorithm auditing methods and frameworks, in shaping best practices [97] and in documenting the struggles of auditors in practice [41]. Researchers have recently developed tools to facilitate algorithm audits, such as for LLM assessments [14], fairness metric selection [114] and code reviews [80]. For user-facing algorithms, user-driven audit frameworks have been proposed [43,87,118] and are increasingly adopted by practitioners [41]. ...

ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing
  • Citing Conference Paper
  • May 2024

... Meanwhile, Visual ChatGPT [65] combines conversational capabilities with visual foundation models, enabling users to talk, draw, and edit visuals simultaneously, creating a richer and more dynamic engagement experience. ChainForge [5] serves as a visual toolkit for prompt engineering and hypothesis testing, allowing users to iteratively refine their inputs and analyze outputs in an intuitive visual format, thereby enhancing the overall interaction with LLMs. Compared to these works, we use graphical control and direct manipulation to specify both input and output. ...

ChainForge: An open-source visual programming environment for prompt engineering
  • Citing Conference Paper
  • October 2023

... The proactivity of CCTs has also been investigated, with research encouraging its implementation [37]. Vaithilingam et al. suggested that automatic inline gray-text suggestions, as opposed to traditional lightbulb interfaces, significantly improve discoverability and usability of CCTs [38]. The authors established five design principles to guide future AI-driven code suggestion interfaces: ...

Towards More Effective AI-Assisted Programming: A Systematic Design Exploration to Improve Visual Studio IntelliCode’s User Experience
  • Citing Conference Paper
  • May 2023

... In recent years, researchers have also started to focus on the experience of developers when using AI code generation tools and how the tools can improve productivity by observing their behavior. Vaithilingam et al. [77] studied how developer use and perceive Copilot and found that although Copilot may not necessarily improve task completion time or success rate, it often provides a useful starting point. They also noted that the participants had difficulties understanding, editing, and debugging the code snippets generated by Copilot. ...

Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models
  • Citing Conference Paper
  • April 2022

... Task decomposition is a long-standing challenge in program synthesis and code generation [23,31,39,71]. Several approaches support task decomposition by asking users to specify intermediate steps [27,33,82]. For instance, Wranger [33] recommends a ranked list of operators at each synthesis step and asks users to select which operator to use and fill in the parameters. ...

Assuage: Assembly Synthesis Using A Guided Exploration
  • Citing Conference Paper
  • October 2021

... We instantiate pragmatic PBE synthesis in a subdomain of regular expressions for the first time, in order to characterize the user experience of PBE with a pragmatic inductive bias in a domain where PBE has been helpful in the past [5,9,32,57,59,60]. However, we restricted the synthesizer to a sufficiently small subdomain of regular expressions to make the computation currently necessary for the pragmatic approach practical and interactive. ...

Interpretable Program Synthesis

... Heer et al. [28] employ query relaxation techniques, allowing users to broaden their selection effectively. Bespoke [50] utilizes user demonstrations to generate custom UIs for command-line applications. It applies rule-based heuristics to infer the semantic types of parameters in bash commands and thus dynamically create widgets for parameter manipulation. ...

Bespoke: Interactively Synthesizing Custom GUIs from Command-Line Applications By Demonstration
  • Citing Conference Paper
  • October 2019