Ryen W. White’s research while affiliated with Microsoft and other places

Envisioning the future of information interaction using multiple modalities enabled by the emergent capabilities of generative artificial intelligence.

Generative Artificial Intelligence (GenAI) is revolutionizing how people access information and how they tackle and complete complex information tasks. This report is a summary of a recent workshop at Microsoft on this important and pressing topic. The event brought together a diverse mix of attendees from different professions and at different career stages for an engaging day of presentations and discussions. The emergent themes are described in detail in this summary. Date : 27 September 2024. Website : https://ir-ai.github.io.

In the midst of the growing integration of Artificial Intelligence (AI) into various aspects of our lives, agents are experiencing a resurgence. These autonomous programs that act on behalf of humans are neither new nor exclusive to the mainstream AI movement. By exploring past incarnations of agents, we can understand what has been done previously, what worked, and more importantly, what did not pan out and why. This understanding lets us to examine what distinguishes the current focus on agents. While generative AI is appealing, this technology alone is insufficient to make new generations of agents more successful. To make the current wave of agents effective and sustainable, we envision an ecosystem that includes not only agents but also Sims, which represent user preferences and behaviors, as well as Assistants, which directly interact with the user and coordinate the execution of user tasks with the help of the agents.

Social influence is a strong determinant of food consumption, which in turn influences the environment and health. Purchasing mimicry, a phenomenon where a person copies another person’s purchases, has been identified as the key governing mechanism. Although consistent observations have been made on the role of purchasing mimicry in driving similarities in food consumption, much less is known about the precise prevalence, the affected subpopulations, and the food types most strongly associated with mimicry effects. Here, we study social influence on food choice through carefully designed causal analyses, leveraging the sequential nature of shop queues on a large university campus. In particular, we consider a large number of adjacent purchases where a focal user immediately follows another user (“partner”) in the checkout queue and both make a purchase. Across food additions purchased during lunchtime together with a meal, we find that the focal user is significantly more likely to purchase the food item when the partner buys the item, vs. when the partner does not, increasing the purchasing probability by 14% in absolute terms, or by 83% in relative terms. The effect is observed across all food types, but largest for condiments. Furthermore, purchasing mimicry is present across age, gender, and status subpopulations, but strongest for students and the youngest. We elucidate the behavioral mechanism of purchasing mimicry, and derive direct implications for interventions improving dietary behaviors on campus, such as facilitating pre-ordering to reduce detrimental interactions.

Dek AI agents are extending the capabilities of traditional search engines to help users tackle complex tasks.

Although diets influence health and the environment, measuring and changing nutrition is challenging. Traditional measurement methods face challenges, and designing and conducting behavior-changing interventions is conceptually and logistically complicated. Situated local communities such as university campuses offer unique opportunities to shape the nutritional environment and promote health and sustainability. The present study investigates how passively sensed food purchase logs typically collected as part of regular business operations can be used to monitor and measure on-campus food consumption and understand food choice determinants. First, based on 38 million sales logs collected on a large university campus over eight years, we perform statistical analyses to quantify spatio-temporal determinants of food choice and characterize harmful patterns in dietary behaviors, in a case study of food purchasing at EPFL campus. We identify spatial proximity, food item pairing, and academic schedules (yearly and daily) as important determinants driving the on-campus food choice. The case studies demonstrate the potential of food sales logs for measuring nutrition and highlight the breadth and depth of future possibilities to study individual food-choice determinants. We describe how these insights provide an opportunity for stakeholders, such as campus offices responsible for managing food services, to shape the nutritional environment and improve health and sustainability by designing policies and behavioral interventions. Finally, based on the insights derived through the case study of food purchases at EPFL campus, we identify five future opportunities and offer a call to action for the nutrition research community to contribute to ensuring the health and sustainability of on-campus populations—the very communities to which many researchers belong.

The emergence of generative artificial intelligence (GenAI) is transforming information interaction. For decades, search engines such as Google and Bing have been the primary means of locating relevant information for the general population. They have provided search results in the same standard format (the so-called "10 blue links"). The recent ability to chat via natural language with AI-based agents and have GenAI automatically synthesize answers in real-time (grounded in top-ranked results) is changing how people interact with and consume information at massive scale. These two information interaction modalities (traditional search and AI-powered chat) coexist in current search engines, either loosely coupled (e.g., as separate options/tabs) or tightly coupled (e.g., integrated as a chat answer embedded directly within a traditional search result page). We believe that the existence of these two different modalities, and potentially many others, is creating an opportunity to re-imagine the search experience, capitalize on the strengths of many modalities, and develop systems and strategies to support seamless flow between them. We refer to these as panmodal experiences. Unlike monomodal experiences, where only one modality is available and/or used for the task at hand, panmodal experiences make multiple modalities available to users (multimodal), directly support transitions between modalities (crossmodal), and seamlessly combine modalities to tailor task assistance (transmodal). While our focus is search and chat, with learnings from insights from a survey of over 100 individuals who have recently performed common tasks on these two modalities, we also present a more general vision for the future of information interaction using multiple modalities and the emergent capabilities of GenAI.

The success of graph neural networks (GNNs) in graph-based web mining highly relies on abundant human-annotated data, which is laborious to obtain in practice. When only a few labeled nodes are available, how to improve their robustness is key to achieving replicable and sustainable graph semi-supervised learning. Though self-training is powerful for semi-supervised learning, its application on graph-structured data may fail because 1) larger receptive fields are not leveraged to capture long-range node interactions, which exacerbates the difficulty of propagating feature-label patterns from labeled nodes to unlabeled nodes and 2) limited labeled data makes it challenging to learn well-separated decision boundaries for different node classes without explicitly capturing the underlying semantic structure. To address the challenges of capturing informative structural and semantic knowledge, we propose a new graph data augmentation framework, augmented graph self-training (AGST), which is built with two new (i.e., structural and semantic) augmentation modules on top of a decoupled GST backbone. In this work, we investigate whether this novel framework can learn a robust graph predictive model under the low-data context. We conduct comprehensive evaluations on semi-supervised node classification under different scenarios of limited labeled-node data. The experimental results demonstrate the unique contributions of the novel data augmentation framework for node classification with few labeled data.

Search is far from being a solved problem. While search engines may cope well with simple tasks, searchers and systems struggle as task complexity increases. Task is central to the search process, motivating the search and driving search behavior. Complex search tasks require more than support for rudimentary fact finding or re-finding. Various support options have been offered by search systems over time (e.g., query suggestions, contextual search) to help search engine users more effectively tackle complex tasks. The recent emergence of generative artificial intelligence (AI) and the arrival of assistive agents, or copilots , based on this technology, has the potential to offer further assistance to searchers, especially those engaged in complex tasks. The implications from these advances for the design of intelligent systems and for the future of search itself are significant. This overview of the keynote that I gave at the 2023 ACM SIGIR Conference introduces AI copilots and briefly presents some of the challenges and opportunities for researching, developing, and deploying search copilots. Date : 26 July 2023.