Leif Azzopardi

• B.Info. Sci (Hons I), PhD. Comp. Sci.
• Professor (Associate) at University of Glasgow

The volume of personal information, accessible online about individuals is unprecedented. Such information may be pieced together by others, to create a more detailed picture of a person, exposing them to potential harms, such as employment loss, unwanted attention , fraud, and more. In this context, relevance is contextual, situational and depende...

Incomplete relevance judgments limit the reusability of test collections. When new systems are compared to previous systems that contributed to the pool, they often face a disadvantage. This is due to pockets of unjudged documents (called holes) in the test collection that the new systems return. The very nature of Conversational Search (CS) means...

The rise of personalized conversational search systems has been driven by advancements in Large Language Models (LLMs), enabling these systems to retrieve and generate answers for complex information needs. However, the automatic evaluation of responses generated by Retrieval Augmented Generation (RAG) systems remains an understudied challenge. In...

Auditing Large Language Models (LLMs) to discover their biases and preferences is an emerging challenge in creating Responsible Artificial Intelligence (AI). While various methods have been proposed to elicit the preferences of such models, countermeasures have been taken by LLM trainers, such that LLMs hide, obfuscate or point blank refuse to disc...

Attributing answers to source documents is an approach used to enhance the verifiability of a model's output in retrieval augmented generation (RAG). Prior work has mainly focused on improving and evaluating the attribution quality of large language models (LLMs) in RAG, but this may come at the expense of inducing biases in the attribution of answ...

We present a report on the Collab-a-thon, a series of sessions at the European Conference on Information Retrieval (ECIR) 2024 designed to help foster new collaborations during a conference. This report presents the motivation and design of the Collab-a-thon, a summary of the discussions covered at each session, and a set of recommendations for con...

The First Search Futures Workshop, in conjunction with the Fourty-sixth European Conference on Information Retrieval (ECIR) 2024, looked into the future of search to ask questions such as: • How can we harness the power of generative AI to enhance, improve and re-imagine Information Retrieval (IR)? • What are the principles and fundamental rights t...

In most recent studies, gender bias in document ranking is evaluated with the NFaiRR metric, which measures bias in a ranked list based on an aggregation over the unbiasedness scores of each ranked document. This perspective in measuring the bias of a ranked list has a key limitation: individual documents of a ranked list might be biased while the...

In most recent studies, gender bias in document ranking is evaluated with the NFaiRR metric, which measures bias in a ranked list based on an aggregation over the unbiasedness scores of each ranked document. This perspective in measuring the bias of a ranked list has a key limitation: individual documents of a ranked list might be biased while the...

Digital libraries aim to provide value to users by housing content that is accessible and searchable. Often such access is afforded through external web search engines. In this article, we measure how easily digital library content can be retrieved (i.e., how retrievable) through a well‐known search engine (Google) using its analytics platforms. Us...

The Probability Ranking Principle (PRP) ranks search results based on their expected utility derived solely from document contents, often overlooking the nuances of presentation and user interaction. However, with the evolution of Search Engine Result Pages (SERPs), now comprising a variety of result cards, the manner in which these results are pre...

Cumulative disclosure Post-mortem privacy Data legacy Privacy a b s t r a c t The paper argues that protecting post-mortem privacy is not solely beneficial for the deceased and their relatives but enables intergenerational data-sharing. However, legal approaches alone are unlikely to generate the trust required and need to be supplemented with tool...

During the Information Search and Retrieval (ISR) process, user-system interactions such as submitting queries, examining results, and engaging with information, impose some degree of demand on the user’s resources. Within ISR, these demands are well recognised, and numerous studies have demonstrated that the Cost, Effort, and Load (CEL) experience...

When pieces from an individual's personal information available online are connected over time and across multiple platforms, this more complete digital trace can give unintended insights into their life and opinions. In a data narrative interview study with 26 currently employed participants, we examined risks and harms to individuals and employer...

Increasing agent transparency is an ongoing challenge for Human-Agent Collaboration (HAC). Chen et al. proposed the three level SAT framework to improve Agent Transparency and users’ Situational Awareness (SA) by informing about (1) what the agent is doing, (2) why the agent is doing it and (3) what the agent will do next. Explanations can be descr...

In the past few years, individuals choose increasingly to share online about themselves, which reflect greatly about their personality and self-image. In particular, individuals with mental health issues tend to discuss or mention such issues online, in an obvious or concealed way. Thus, analysing individuals’ online posts for mental health, became...

Searching for people online is a common search task that most of us have performed at some point or other. With so much information about people available online it is often amazing what one can fnd out about someone else – especially when information taken from diferent sources is pieced together to create a more detailed picture of the individual...

Various conceptual and descriptive models of conversational search have been proposed in the literature – while useful, they do not provide insights into how interaction between the agent and user would change in response to the costs and benefits of the different interactions. In this paper, we develop two economic models of conversational search...

When pieces from an individual's personal information available online are connected over time and across multiple platforms, this more complete digital trace can give unintended insights into their life and opinions. In a data narrative interview study with 26 currently employed participants, we examined risks and harms to individuals and employer...

Various conceptual and descriptive models of conversational search have been proposed in the literature -- while useful, they do not provide insights into how interaction between the agent and user would change in response to the costs and benefits of the different interactions. In this paper, we develop two economic models of conversational search...

Procurement or tender search is where suppliers seek opportunities for providing goods, works or services that authorities, organisations and businesses require. Such opportunities are listed as procurement contract notices for which suppliers can submit tenders. Typically, an E-Procurement system is used to help find and carry out one or more of t...

The law does not concern itself with trifles. If a risk is deemed minimal, or an infraction negligible, invoking the authority of the law often seems unnecessary. However, there are increasingly fields of human activity where this principle leads to gaps in the protection necessary for a flourishing society. This paper reports findings and ideas fr...

Every day, people post information about themselves and others on online social networks, making such information accessible within their social circles but also, potentially, way beyond. While the information posted may seem benign or innocuous, small pieces of information, when tied together, can potentially reveal much more about the person than...

Information seeking conversations between users and Conversational Search Agents (CSAs) consist of multiple turns of interaction. While users initiate a search session, ideally a CSA should sometimes take the lead in the conversation by obtaining feedback from the user by offering query suggestions or asking for query clarifications i.e. mixed init...

This paper briefly describes our research groups’ efforts in tackling Task 1 (Early Detection of Signs of Self-Harm), and Task 2 (Measuring the Severity of the Signs of Depression) from the CLEF eRisk Track. Core to how we approached these problems was the use of BERT-based classifiers which were trained specifically for each task. Our results on b...

Uncertainty in Human-Agent interactions is often studied in terms of transparency and understandability of agent actions. Less work, however, has focused on how Visual Environmental Uncertainty (VEU) that restricts or occludes visual information affects the Human-Agent Teaming (HAT) in terms of trust, reliance, performance, cognitive load and situa...

Collaborative virtual agents help human operators to perform tasks in real-time. For this collaboration to be effective, human operators must appropriately trust the agent(s) they are interacting with. Multiple factors influence trust, such as the context of interaction, prior experiences with automated systems and the quality of the help offered b...

Sentiment analysis (SA) is the key element for a variety of opinion and attitude mining tasks. While various unsupervised SA tools already exist, a central problem is that they are lexicon-based where the lexicons used are limited, leading to a vocabulary mismatch. In this paper, we present an unsupervised word embedding-based sentiment scoring fra...

Trust in automation is often strongly tied to an agent’s performance. However, our understanding of imperfect agents’ behaviours and its impact on trust is limited. In this paper, we study the relationship between performance, reliance and trust in a set of human-agent collaborative tasks. Participants collaborated with different automated agents t...

Trust in automation is often strongly tied to an agent's performance. However, our understanding of imperfect agents' behaviours and its impact on trust is limited. In this paper, we study the relationship between performance, reliance and trust in a set of human-agent collaborative tasks. Participants collaborated with different automated agents t...

Undertaking an interactive evaluation of goal-oriented conversational agents (CAs) is challenging, it requires the search task to be realistic and relatable while accounting for the users cognitive limitations. In the current paper we discuss findings of two Wizard of Oz studies and provide our reflections regarding the impact of different interact...

Conversational search relies on an interactive, natural language exchange between a user, who has an information need, and a search system, which elicits and reveals information. Prior research posits that due to the non-persistent nature of speech, conversational agents (CAs) should support users in their search task by: (1) actively suggesting qu...

Result diversification aims to provide searchers with a broader view of a given topic while attempting to maximise the chances of retrieving relevant material. Diversifying results also aims to reduce search bias by increasing the coverage over different aspects of the topic. As such, searchers should learn more about the given topic in general. De...

In this paper, we provide an overview of the seventh annual edition of the CLEF eHealth evaluation lab. CLEF eHealth 2019 continues our evaluation resource building efforts around the easing and support of patients, their next-of-kins, clinical staff, and health scientists in understanding, accessing, and authoring electronic health information in...

Procurement legislation stipulates that information about the goods, services, or works, that tax-funded authorities wish to purchase are made publicly available in a procurement contract notice. However, for businesses wishing to tender for such competitive opportunities, finding relevant procurement contract notices presents a challenging profess...

We present a tool ("cwl_eval") which unifies many metrics typically used to evaluate information retrieval systems using test collections. In the CWL framework metrics are specified via a single function which can be used to derive a number of related measurements: Expected Utility per item, Expected Total Utility, Expected Cost per item, Expected...

Economics provides an intuitive and natural way to formally represent the costs and benefits of interacting with applications, interfaces and devices. By using economic models it is possible to reason about interaction, make predictions about how changes to the system will affect behavior, and measure the performance of people's interactions with t...

For human-agent collaborations to prosper, end-users need to trust the agent(s) they interact with. This is especially important in scenarios where the users and agents negotiate control in order to achieve objectives in real time (e.g. from helping surgeons with precision tasks to parking a semi-autonomous car or completing objectives in a video-g...

Economics provides an intuitive and natural way to formally represent the cost and benefits of interacting with applications, interfaces and devices. By using economics models it is possible to reason about interaction and make predictions about how changes to the system will affect performance and behavior. In this course, we provided an overview...

Since 2012 CLEF eHealth has focused on evaluation resource building efforts around the easing and support of patients, their next-of-kins, clinical staff, and health scientists in understanding, accessing, and authoring eHealth information in a multilingual setting. This year’s lab offers three tasks: Task 1 on multilingual information extraction;...

This two-volume set LNCS 11437 and 11438 constitutes the refereed proceedings of the 41st European Conference on IR Research, ECIR 2019, held in Cologne, Germany, in April 2019. The 48 full papers presented together with 2 keynote papers, 44 short papers, 8 demonstration papers, 8 invited CLEF papers, 11 doctoral consortium papers, 4 workshop paper...

This two-volume set LNCS 11437 and 11438 constitutes the refereed proceedings of the 41st European Conference on IR Research, ECIR 2019, held in Cologne, Germany, in April 2019. The 48 full papers presented together with 2 keynote papers, 44 short papers, 8 demonstration papers, 8 invited CLEF papers, 11 doctoral consortium papers, 4 workshop paper...

In this paper we provide an overview of the CLEF 2018 Dynamic Search Lab. The lab ran for the first time in 2017 as a workshop. The outcomes of the workshop were used to define the tasks of this year’s evaluation lab. The lab strives to answer one key question: how can we evaluate, and consequently build, dynamic search algorithms? Unlike static se...

Legal researchers, recruitment professionals, healthcare information professionals, and patent analysts all undertake work tasks where search forms a core part of their duties. In these instances, the search task is often complex and time-consuming and requires specialist expertise to identify relevant documents and insights within large 15 domain-...

Despite significant improvements in automatic speech recognition and spoken language understanding - human interaction with Virtual Personal Assistants (VPAs) through speech remains irregular and sporadic. According to recent studies, currently the usage of VPAs is constrained to basic tasks such as checking facts, playing music, and obtaining weat...

Voice based search systems currently do not support natural conversational interaction. Consequently, people tend to limit their use of voice search to simple navigational tasks, as more complex search tasks require more sophisticated dialogue modelling. In this paper, we explore how people's search behaviour, performance and perception of usabilit...

The purpose of the Strategic Workshop in Information Retrieval in Lorne is to explore the long-range issues of the Information Retrieval field, to recognize challenges that are on-or even over-the horizon, to build consensus on some of the key challenges, and to disseminate the resulting information to the research community. The intent is that thi...

Web Search Engine Result Pages (SERPs) are complex responses to queries, containing many heterogeneous result elements (web results, advertisements, and specialised "answers'') positioned in a variety of layouts. This poses numerous challenges when trying to measure the quality of a SERP because standard measures were designed for homogeneous ranke...

When conducting systematic reviews, medical researchers heavily deliberate over the final query to pose to the information retrieval system. Given the possible query variations that they could construct, selecting the best performing query is difficult. This motivates a new type of query performance prediction (QPP) task where the challenge is to e...

Within many domains, such as news, medicine and patent, documents contain a variety of fields such as title, author, body, source, etc. As such fielded retrieval models that query across these fields are often employed. It is largely presumed that fielding provides a better representation of the document and offers more control when querying and th...

Prior work on using retrievability measures in the evaluation of information retrieval (IR) systems has laid out the foundations for investigating the relation between retrieval performance and retrieval bias. While various factors influencing retrievability have been examined, showing how the retrieval model may influence bias, no prior work has e...

The PICO process is a technique used in evidence based practice to frame and answer clinical questions. It involves structuring the question around four types of clinical information: population, intervention, control or comparison and outcome. The PICO framework is used extensively in the compilation of systematic reviews as the means of framing r...

Algorithmic bias presents a difficult challenge within Information Retrieval. Long has it been known that certain algorithms favour particular documents due to attributes of these documents that are not directly related to relevance. The evaluation of bias has recently been made possible through the use of retrievability, a quantifiable measure of...

For improving the effectiveness of Interactive Information Retrieval (IIR), a system should minimise the search time by guiding the user appropriately. As a prerequisite, in any search situation, the system must be able to estimate the time the user will need for finding the next relevant document. In this paper, we show how Markov models derived f...

In this work, the relationship between performance and retrievability bias is explored when various query expansion methods are employed to aide retrieval. Several parameters are altered, independently, to identify those that have an impact on bias. Parameters altered include; Rocchio's beta, length normalisation parameters, the number of terms add...

This paper introduces a test collection for evaluating the effectiveness of different methods used to retrieve research studies for inclusion in systematic reviews. Systematic reviews appraise and synthesise studies that meet specific inclusion criteria. Systematic reviews intended for a biomedical science audience use boolean queries with many, of...

People tend to type short queries, however, the belief is that longer queries are more effective. Consequently, a number of attempts have been made to encourage and motivate people to enter longer queries. While most have failed, a recent attempt - conducted in a laboratory setup - in which the query box has a halo or glow effect, that changes as t...

As an empirical discipline, information access and retrieval research requires substantial software infrastructure to index and search large collections. This workshop is motivated by the desire to better align information retrieval research with the practice of building search applications from the perspective of open-source information retrieval...

The design and presentation of a Search Engine Results Page (SERP) has been subject to much research. With many contemporary aspects of the SERP now under scrutiny, work still remains in investigating more traditional SERP components, such as the result summary. Prior studies have examined a variety of different aspects of result summaries, but in...

The Information Retrieval Festival took place in April 2017 in Glasgow. The focus of the workshop was to bring together IR researchers from the various Scottish universities and beyond in order to facilitate more awareness, increased interaction and reflection on the status of the field and its future. The program included an industry session, rese...

A searcher’s interaction with a retrieval system consists of actions such as query formulation, search result list interaction and document interaction. The simulation of searcher interaction has recently gained momentum in the analysis and evaluation of interactive information retrieval (IIR). However, a key issue that has not yet been adequately...

Providing support for users during their search sessions has been hailed as a major challenge in interactive information retrieval (IIR). Providing such support requires considering the context of the search and facilitating the work task at hand. In this paper, we consider the work tasks associated with air traffic analysts, who perform numerous s...

Modeling how people interact with search interfaces has been of particular interest and importance to the field of Interactive Information Retrieval. Recently, there has been a move to developing formal models of the interaction between the user and the system, whether it be to: (i) run a simulation, (ii) conduct an economic analysis, (iii) measure...

The workshop and hackathon on developing Information Retrieval Evaluation Resources using Lucene (L4IR) was held on the 8th and 9th of September, 2016 at the University of Strathclyde in Glasgow, UK and funded by the ESF Elias Network. The event featured three main elements: (i) a series of keynote and invited talks on industry, teaching and evalua...

Pienapple is a search interface that aims to combine bookmarking and searching within a blended experience to facilitate improved access, serendipity, and sharing. While personal and social bookmarking platforms already exist, they are often separated from the search system, resulting in an increased effort and complexity because two or more system...

Most of the current models that are used to simulate users in Interactive Information Retrieval (IIR) lack realism and agency. Such models generally make decisions in a stochastic manner, without recourse to the actual information encountered or the underlying information need. In this paper, we develop a more sophisticated model of the user that i...

Searching is performed in the context of a task and as such the value of the information found is with respect to the task. Recently, there has been a drive to developing formal models of information seeking and retrieval that consider the costs and benefits arising through the interaction with the interface/system and the information surfaced duri...

Interactive Information Retrieval (IR) systems often provide various features and functions, such as query suggestions and relevance feedback, that a user may or may not decide to use. The decision to take such an option has associated costs and may lead to some benefit. Thus, a savvy user would take decisions that maximises their net benefit. In t...

Retrievability is an independent evaluation measure that offers insights to an aspect of retrieval systems that performance and efficiency measures do not. Retrievability is often used to calculate the retrievability bias, an indication of how accessible a system makes all the documents in a collection. Generally, computing the retrievability bias...

Simulation provides a powerful and cost-effective approach to explore and evaluate how interactions between a searcher and system influence search behaviour and performance. With a growing interest in simulation and an increasing number of papers using such an approach, there is a need for a flexible framework for simulation. Thus, we present SimII...