Wendy E. Mackay’s research while affiliated with University of Paris-Saclay and other places

Visually oriented designers often struggle to create effective generative AI (GenAI) prompts. A preliminary study identified specific issues in composing and fine-tuning prompts, as well as needs in accurately translating intentions into rich input. We developed DesignPrompt, a moodboard tool that lets designers combine multiple modalities — images, color, text — into a single GenAI prompt and tweak the results. We ran a comparative structured observation study with 12 professional designers to better understand their intent expression, expectation alignment and transparency perception using DesignPrompt and text input GenAI. We found that multimodal prompt input encouraged designers to explore and express themselves more effectively. Designer’s interaction preferences change according to their overall sense of control over the GenAI and whether they are seeking inspiration or a specific image. Designers developed innovative uses of DesignPrompt, including developing elaborate multimodal prompts and creating a multimodal prompt pattern to maximize novelty while ensuring consistency.

This Special Interest Group (SIG) explores the transformative impact of Generative Artificial Intelligence (GenAI) on Human-Computer Interaction (HCI) research processes. The theme here is to answer “question zero”: when to use and when to refrain from using AI tools during the research cycle? The discussion is guided by five research phases commonly used in HCI: research planning, prototyping, data collection, analysis and synthesis, and dissemination and communication. We investigate how GenAI accelerates project cycles, enhances reproducibility, and influences inclusivity in research. We also address the challenging ethical considerations about the ownership of generated content. Our goal is to build a community of HCI enthusiasts to harness the early advantages of the recent groundbreaking technology and foresee challenges arising from its prevalence in the scientific community.

How can we can design “human-computer partnerships” that take optimal advantage of human skills and system capabilities? Artificial Intelligence research is usually measured in terms of the effectiveness of an algorithm, whereas Human-Computer Interaction research focuses on enhancing human abilities. I argue that better AI algorithms are neither necessary nor sufficient for creating more effective intelligent systems. Instead, we need to focus on the details of interaction and how to successfully balance the simplicity of the user’s interaction with the expressive power of the system. After describing our approach to “generative theories of interaction”, I illustrate how to create interactive intelligent systems where users can discover relevant functionality, express individual differences and appropriate the system for their own personal use.

Designing interactive technology to support safety-critical systems poses multiple challenges with respect to security, access to operators and the proprietary nature of the data. We conducted a two-year participatory design project with French power grid operators to both understand their specific needs and to collaborate on the design of a novel collaborative tool called StoryLines . Our primary objective was to capture detailed, in-context data about operators’ work practices as part of a larger project designed to provide bi-directional assistance between an intelligent agent and human operator. We targeted handovers between shifts to take advantage of the operators’ existing practice of articulating the current status of the grid and expected future events. We use information that would otherwise be lost to gather valuable information about the operator’s decision rationale and decision-making patterns. This paper describes how we combined a bottom-up participatory design approach with a top-down generative theory approach to design StoryLines, an interactive timeline that helps operators collect information from diverse tools, record reminders and share relevant information with the next shift’s operator. We conclude with a discussion of the challenges of working with users in safety-critical environments and directions for future research.