Sebastian Möller’s research while affiliated with Deutsches Forschungszentrum für Künstliche Intelligenz and other places

Counterfactual examples are widely used in natural language processing (NLP) as valuable data to improve models, and in explainable artificial intelligence (XAI) to understand model behavior. The automated generation of counterfactual examples remains a challenging task even for large language models (LLMs), despite their impressive performance on many tasks. In this paper, we first introduce ZeroCF, a faithful approach for leveraging important words derived from feature attribution methods to generate counterfactual examples in a zero-shot setting. Second, we present a new framework, FitCF, which further verifies aforementioned counterfactuals by label flip verification and then inserts them as demonstrations for few-shot prompting, outperforming two state-of-the-art baselines. Through ablation studies, we identify the importance of each of FitCF's core components in improving the quality of counterfactuals, as assessed through flip rate, perplexity, and similarity measures. Furthermore, we show the effectiveness of LIME and Integrated Gradients as backbone attribution methods for FitCF and find that the number of demonstrations has the largest effect on performance. Finally, we reveal a strong correlation between the faithfulness of feature attribution scores and the quality of generated counterfactuals.

In the last decade, crowdsourcing has become a popular method for conducting quantitative empirical studies in human-machine interaction. The remote work on a given task in crowdworking settings suits the character of typical speech/language-based interactive systems for instance with regard to argumentative conversations and information retrieval. Thus, crowdworking promises a valuable opportunity to study and evaluate the usability and user experience of real humans in interactions with such interactive systems. In contrast to physical attendance in laboratory studies, crowdsourcing studies offer much more flexible and easier access to large numbers of heterogeneous participants with a specific background, e.g., native speakers or domain expertise. On the other hand, the experimental and environmental conditions as well as the participant's compliance and reliability (at least better monitoring of the latter) are much better controllable in a laboratory. This paper seeks to present a (self-)critical examination of crowdsourcing-based studies in the context of complex (spoken) dialogue systems. It describes and discusses observed issues in crowdsourcing studies involving complex tasks and suggests solutions to improve and ensure the quality of the study results. Thereby, our work contributes to a better understanding and what needs to be considered when designing and evaluating studies with crowdworkers for complex dialogue systems.

This study explored a concept for using Augmented Reality (AR) glasses to customize augmented table setup and lighting in a restaurant. The aim was to provide insights into AR usage in restaurants and contribute to existing research by introducing an extendable and versatile concept for scholars and restaurateurs. A controlled laboratory study, using a within-subjects design, was conducted to investigate the effects of a customizable augmented table setup and lighting on user experience (UX), perceived waiting time, psychological ownership, and social acceptability. A simulated restaurant environment was created using a 360-degree image in Virtual Reality (VR). The study implemented default and customizable table setup and lighting. Results from a paired samples t-test showed a statistically significant effect of table setup and lighting on the pragmatic quality of UX, hedonic quality of UX, overall UX, valence, dominance, psychological ownership, and affect. Furthermore, table setup had a significant effect on arousal and perceived waiting time. Moreover, table setup significantly affected AR interaction, isolation, and safety acceptability, while lighting only affected AR interaction acceptability. Findings suggest that these investigated variables are worth considering for AR applications in a restaurant, especially when offering customizable augmented table setup and lighting.

In this work, we introduce a methodology for alignment designed to enhance the ability of large language models (LLMs) to articulate their reasoning (self-explanation) even in the absence of annotated rationale explanations. Our alignment methodology comprises three key components: explanation quality assessment, self-instruction dataset generation, and model alignment. Additionally, we present a novel technique called Alignment with Anchor Preference Pairs, which improves the selection of preference pairs by categorizing model outputs into three groups: consistently correct, consistently incorrect, and variable. By applying tailored strategies to each category, we enhance the effectiveness of Direct Preference Optimization (DPO). Our experimental results demonstrate that this approach significantly improves explanation quality while maintaining accuracy compared to other fine-tuning strategies.

Nowadays, Mixed Reality (MR) headsets are a game-changer for knowledge work. Unlike stationary monitors, MR headsets allow users to work with large virtual displays anywhere they wear the headset, whether in a professional office, a public setting like a cafe, or a quiet space like a library. This study compares four different layouts (eye level-close, eye level-far, below eye level-close, below eye level-far) of virtual displays regarding feelings of safety, perceived productivity, and social acceptability when working with MR in public. We test which layout is most preferred by users and seek to understand which factors affect users' layout preferences. The aim is to derive useful insights for designing better MR layouts. A field study in a public library was conducted using a within-subject design. While the participants interact with a layout, they are asked to work on a planning task. The results from a repeated measure ANOVA show a statistically significant effect on productivity but not on safety and social acceptability. Additionally, we report preferences expressed by the users regarding the layouts and using MR in public.

The EyeSight feature, introduced with the new Apple Vision Pro XR headset, promises to revolutionize user interaction by simulating real human eye expressions on a digital display. This feature could enhance XR devices' social acceptability and social presence when communicating with others outside the XR experience. In this pilot study, we explore the implications of the EyeSight feature by examining social acceptability, social presence, emotional responses, and technology acceptance. Eight participants engaged in conversational tasks in three conditions to contrast experiencing the Apple Vision Pro with EyeSight, the Meta Quest 3 as a reference XR headset, and a face-to-face setting. Our preliminary findings indicate that while the EyeSight feature improves perceptions of social presence and acceptability compared to the reference headsets, it does not match the social connectivity of direct human interactions.

Introduction As healthcare is shifting from a paternalistic to a patient-centred approach, medical decision making becomes more collaborative involving patients, their support persons (SPs) and physicians. Implementing shared decision-making (SDM) into clinical practice can be challenging and becomes even more complex with the introduction of artificial intelligence (AI) as a potential actant in the communicative network. Although there is more empirical research on patients’ and physicians’ perceptions of AI, little is known about the impact of AI on SDM. This study will help to fill this gap. To the best of our knowledge, this is the first systematic empirical investigation to prospectively assess the views of patients, their SPs and physicians on how AI affects SDM in physician–patient communication after kidney transplantation. Using a transdisciplinary approach, this study will explore the role and impact of an AI-decision support system (DSS) designed to assist with medical decision making in the clinical encounter. Methods and analysis This is a plan to roll out a 2 year, longitudinal qualitative interview study in a German kidney transplant centre. Semi-structured interviews with patients, SPs and physicians will be conducted at baseline and in 3-, 6-, 12- and 24-month follow-up. A total of 50 patient–SP dyads and their treating physicians will be recruited at baseline. Assuming a dropout rate of 20% per year, it is anticipated that 30 patient–SP dyads will be included in the last follow-up with the aim of achieving data saturation. Interviews will be audio-recorded and transcribed verbatim. Transcripts will be analysed using framework analysis. Participants will be asked to report on their (a) communication experiences and preferences, (b) views on the influence of the AI-based DSS on the normative foundations of the use of AI in medical decision-making, focusing on agency along with trustworthiness, transparency and responsibility and (c) perceptions of the use of the AI-based DSS, as well as barriers and facilitators to its implementation into routine care. Ethics and dissemination Approval has been granted by the local ethics committee of Charité—Universitätsmedizin Berlin (EA1/177/23 on 08 August 2023). This research will be conducted in accordance with the principles of the Declaration of Helsinki (1996). The study findings will be used to develop communication guidance for physicians on how to introduce and sustainably implement AI-assisted SDM. The study results will also be used to develop lay language patient information on AI-assisted SDM. A broad dissemination strategy will help communicate the results of this research to a variety of target groups, including scientific and non-scientific audiences, to allow for a more informed discourse among different actors from policy, science and society on the role and impact of AI in physician–patient communication.