Nicholas J. Belkin

• PhD
• Professor (Full) at Rutgers, The State University of New Jersey

The Third Workshop on Evaluation of Personalisation in Information Retrieval (WEPIR 2021) was held in conjunction with the ACM SIGIR Conference on Human Information Interaction & Retrieval (CHIIR 2021) in Canberra, Australia, as a virtual event. WEPIR 2021 followed on from the first and second WEPIRs held at CHIIR 2018 and 2019. The purpose of the...

Purpose The purpose of this paper is to explore effects of individuals' social context on their perception of a task, for better understanding of social aspects of task-based information seeking behavior. Design/methodology/approach This study took a qualitative case approach and conducted semi-structured one-on-one interviews with 12 participants...

This study aims to examine users' knowledge change characteristics and knowledge change process, and examined factors that may influence users' knowledge change process. A user search experiment was conducted, in which participants were asked to search for two learning‐related search tasks and use a mind‐map to organize their thoughts during the se...

The Second Workshop on Evaluation of Personalisation in Information Retrieval (WEPIR 2019) was held in conjunction with the ACM SIGIR Conference on Human Information Interaction & Retrieval (CHIIR 2019) in Glasgow, Scotland. WEPIR 2019 followed on from the first WEPIR held at CHIIR 2018. The purpose of the workshop was again to bring together resea...

Personalization of information retrieval (PIR) is aimed at tailoring a search toward individual users and user groups by taking account of additional information about users besides their queries. In the past two decades or so, PIR has received extensive attention in both academia and industry. This article surveys the literature of personalization...

The second WEPIR 2019 workshop brings together researchers with different backgrounds interested in continuing to explore and advance the evaluation of personalisation in information retrieval. The workshop builds on the first WEPIR workshop held at CHIIR 2018, and will focus on further developing a common understanding of the challenges, requireme...

In Interactive Information Retrieval (IIR) experiments the user's gaze motion on web pages is often recorded with eye tracking. The data is used to analyze gaze behavior or to identify Areas of Interest (AOI) the user has looked at. So far, tools for analyzing eye tracking data have certain limitations in supporting the analysis of gaze behavior in...

According to the cognitive viewpoint of information retrieval (IR) research, a search task can be conceptualized as a sequence of information seeking intentions which both motivate and are influenced by search behaviors. While the behavioral effects of task features have been thoroughly discussed in a large body of literature, how different informa...

In Interactive Information Retrieval (IIR) experiments the user's gaze motion on web pages is often recorded with eye tracking. The data is used to analyze gaze behavior or to identify Areas of Interest (AOI) the user has looked at. So far, tools for analyzing eye tracking data have certain limitations in supporting the analysis of gaze behavior in...

Two key, but usually ignored, issues for the evaluation of methods of personalization for information retrieval are: that such evaluation must be of a search session as a whole; and, that people, during the course of an information search session, engage in a variety of activities, intended to accomplish differ- ent goals or intentions. Taking seri...

The Workshop on Evaluation of Personalisation in Information Retrieval (WEPIR 2018) was held in conjunction with the ACM SIGIR Conference on Human Information Interaction & Retrieval (CHIIR 2018) in New Brunswick, USA. The purpose of WEPIR 2018 was to bring together researchers from different backgrounds, interested in advancing the evaluation of p...

When users are looking for information on the Web, they show different behavior for different task types, e.g., for fact finding vs. information gathering tasks. For example, related work in this area has investigated how this behavior can be measured and applied to distinguish between easy and difficult tasks. In this work, we look at the searcher...

Two key, but usually ignored, issues for the evaluation of methods of personalization for information retrieval are: that such evaluation must be of a search session as a whole; and, that people, during the course of an information search session, engage in a variety of activities, intended to accomplish different goals or intentions. Taking seriou...

Cambridge Core - Knowledge Management, Databases and Data Mining - Interactive Information Seeking, Behaviour and Retrieval - edited by Ian Ruthven

Interactive Information Seeking, Behaviour and Retrieval - edited by Ian Ruthven December 2013

The purpose of the WEPIR 2018 workshop is to bring together researchers from different backgrounds, interested in advancing the evaluation of personalisation in information retrieval. The workshop focus is on the development of a common understanding of the challenges, requirements and practical limitations of meaningful evaluation of personalisati...

When users are looking for information on the Web, they show different behavior for different task types, e.g., for fact finding vs. information gathering tasks. For example, related work in this area has investigated how this behavior can be measured and applied to distinguish between easy and difficult tasks. In this work, we look at the searcher...

Query reformulation is the major method by which information seekers can communicate their information searching intentions to information retrieval systems. Despite a good deal of significant research on types of query reformulations, we still know little about why people reformulate their queries in various ways. We report on a study which elicit...

Query reformulation is the major method by which information seekers can communicate their information searching intentions to information retrieval systems. Despite a good deal of significant research on types of query reformulations, we still know little about why people reformulate their queries in various ways. We report on a study which elicit...

It has been shown that people attempt to accomplish a variety of intentions during the course of an information seeking session, and there is reason to believe that these different information seeking intentions can benefit from system support tailored to each such intention. We address the problem of predicting the presence of such intentions duri...

There is broad consensus in the field of IR that search is complex in many use cases and applications, both on the Web and in domain-specific collections, and both in our professional and in our daily life. Yet our understanding of complex search tasks, in comparison to simple look up tasks, is fragmented at best. The workshop addressed many open r...

Harman, D. K. and Kelly, D., editors

There is broad consensus in the field of IR that search is complex in many use cases and applications, both on the Web and in domain specific collections, and both professionally and in our daily life. Yet our understanding of complex search tasks, in comparison to simple look up tasks, is fragmented at best. The workshop addresses many open resear...

The established research domain of information retrieval and the interdisciplinary domain of knowledge mapping have mainly been independent from each other. Both strands are driven by quite different epistemic perspectives. The use of information visualization in the area of information retrieval is predominantly focused on the support of informati...

The management of chronic conditions is often accompanied with the long-term management of health information. By examining how patients manage their health information over time, we can develop guidelines and design technologies to support this patient work, and also contribute the patients' perspective to the existing literature on personal infor...

We report on a study investigating the relationships among query reformulations and different search intentions during an information seeking session. Twenty-four participants were each asked to search for information useful for two (of four) different journalism tasks; after completing each search, the search was replayed, and participants were as...

As a significant contextual factor in information search, topic knowledge has been gaining increased research attention. We report on a study of the relationship between information searchers' topic knowledge and their search behaviors, and on an attempt to predict searchers' topic knowledge from their behaviors during the search. Data were collect...

We present a method for extracting the self-reported intentions of users engaged in an information seeking episode. We recruited participants to conduct search sessions and subsequently asked them to self-report their intentions. A total of 27 users participated in a lab study, during which they worked on two search tasks. After each search session...

Recent recognition in IR that people engage in information seeking sessions; attempts to formally model search sessions; and, the move toward evaluation of IR systems over entire search sessions, are evidence of a new, broader understanding of IR's goals. This move takes IR's goal beyond identification of relevant information objects, to supporting...

Colleagues, friends, let me begin by expressing how pleased, and humbly honored I am to be a recipient of the Gerard Salton Award. Gerry was a great man, and to receive the award named for him is very special. For me personally, it is especially meaningful, given the sometime disputatious nature of our professional interactions, and what seemed, on...

Personalization of support for information seeking depends crucially on the information retrieval system's knowledge of the task that led the person to engage in information seeking. Users work during information search sessions to satisfy their task goals, and their activity is not random. To what degree are there patterns in the user activity dur...

Although people engage in a range of information behaviors, the majority of previous models and empirical research in information behavior tend to focus just on information seeking and use. This panel will discuss the need of extending the territories of information behavior research beyond seeking and use. Panelists will explain their own research...

Understanding the motivating factors for personal health information management (PHIM) activities can inform the design of interventions to help people engage in healthful PHIM activities. This paper examines the attributes of health information that can motivate people's PHIM activities. We explored this topic through a qualitative study using sem...

People around the world have developed many strategies to thrive with diabetes by harnessing health information about themselves and the health condition in general. This study examines the information management strategies (i.e., what people do with the information they gathered) employed by diabetes patients and how these strategies help them rea...

Knowing, in real time, whether a current searcher in an information retrieval system finds the search task difficult can be valuable for tailoring the system's support for that searcher. This study investigated searcher's behaviors at different stages of the search process; they are: 1) first-round point at the beginning of the search, right before...

Purpose – People often engage in different information-seeking strategies (ISSs) within a single information-seeking episode. A critical concern for the design of information retrieval (IR) systems is how to provide support for these different behaviors in a manner which searchers can easily understand, navigate and use, as they move from one ISS t...

Can the activity patterns of page use during information search sessions discriminate between different types of information seeking tasks? We model sequences of interactions with search result and content pages during information search sessions. Two representations are created: the sequences of page use and a cognitive representation of page inte...

User domain knowledge affects search behaviors and search success. Predicting a user's knowledge level from implicit evidence such as search behaviors could allow an adaptive information retrieval system to better personalize its interaction with users. This study examines whether user domain knowledge can be predicted from search behaviors by appl...

Personalization of information retrieval tailors search towards individual users to meet their particular information needs by taking into account information about users and their contexts, often through implicit sources of evidence such as user behaviors. This study looks at users' dwelling behavior on documents and several contextual factors: th...

Information search, quite often, is not an isolated activity, but is accompanied by the use of the located information to generate some outcome. Frequently seen are “complex” tasks that consist of multiple aspects and can be divided into sub-tasks and/or finished in multiple sessions. This paper explores how search systems may help users with their...

This study examined personal information items that are difficult to categorize, and how people deal with these information items. For this research study, 18 participants were asked to keep a diary over a week and record diary entries whenever they decided to save or organize any electronic personal information items. Then, two post-diary semi-str...

The acquisition of information and the search interaction process is influenced strongly by a person’s use of their knowledge of the domain and the task. In this paper we show that a user’s level of domain knowledge can be inferred from their interactive search behaviors without considering the content of queries or documents. A technique is presen...

This paper examines the changes of information searchers’ topic knowledge levels in the process of completing information tasks. Multi-session tasks were used in the study, which enables the convenience of eliciting users’ topic knowledge during their process of completing the whole tasks. The study was a 3-session laboratory experiment with 24 par...

In this paper, we examined the effects of information searchers' familiarity with task topics on their search behaviors, in different task types. Data were collected in a controlled laboratory experiment, participated by 32 undergraduate journalism students. Each participant searched on four tasks varying along four task facets: task product (being...

We report on an investigation of behavioral differences between users in difficult and easy search tasks. Behavioral factors that can be used in real-time to predict task difficulty are identified. User data was collected in a controlled lab experiment (n=38) where each participant completed four search tasks in the genomics domain. We looked at us...

Personalization of search results offers the potential for significant improvement in information retrieval performance. User interactions with the system and documents during information-seeking sessions provide a wealth of information about user preferences and their task goals. In this paper, we propose methods for analyzing and modeling user se...

Searching for information is often driven by some work tasks that involve information use and require certain types of outcomes other than finding information. To explore how search systems can help with work tasks calls for examining factors that influence work task performance. A 3-stage controlled lab experiment was conducted with 24 participant...

While search behavior using dynamic query suggestions is understudied, it is virtually non-existent for dynamic search results (as currently experienced with Google Instant). We report results from a controlled lab study aimed at exploring the effects of these recent search interface developments – dynamic query suggestions and dynamic search resul...

This study explores the effect of task difficulty on search behavior changes for users with high and low domain knowledge. A user experiment (n=40) was conducted using 5 search tasks. Participants rated their knowledge of MeSH terms (n=409) and were divided into two domain knowledge levels. Three of the tasks were designed to be difficult tasks wit...

Click through events in search results pages (SERPs) are not reliable implicit indicators of document relevance. A user's task and domain knowledge are key factors in recognition and link selection and the most useful SERP document links may be those that best match the user's domain knowledge. User study participants rated their knowledge of genom...

This study uses regression modeling to predict a user's domain knowledge level (DK) from implicit evidence provided by certain search behaviors. A user study (n=35) with recall-oriented search tasks in the genomic domain was conducted. A number of regression models of a person's DK, were generated using different behavior variable selection methods...

We report findings on how the user's perception of task difficulty changes before and after searching for information to solve tasks. We found that while in one type of task, the dependent task, this did not change, in another, the parallel task, it did. The findings have implications on designing systems that can provide assistance to users with t...

a b s t r a c t We report on an investigation into people's behaviors on information search tasks, specifically the rela-tion between eye movement patterns and task characteristics. We conducted two independent user stud-ies (n = 32 and n = 40), one with journalism tasks and the other with genomics tasks. The tasks were constructed to represent inf...

One major problem of most current information retrieval systems is that they provide uniform access and retrieval results to all users solely based on the query terms users issued to the system. In this paper, we propose a model to personalize the search results according to the user's search context, in particular the type of task that led the use...

This paper examines keeping behavior of personal information in different forms by reviewing and analyzing previous empirical studies on keeping personal information. By adopting user-centered and cross-form perspectives, similar behaviors as well as unique behaviors in keeping different forms of personal information are reviewed and comparatively...

Searchers with a complex information need typically slice-and-dice their problem into several queries and subqueries, and laboriously combine the answers post hoc to solve their tasks. Consider planning a social event at the last day of SIGIR, in the unknown city of Beijing, factoring in distances, timing, and preferences on budget, cuisine, and en...

In this study, we attempted to use dwell time on content pages as a predictor of document usefulness, and evaluated the prediction performance in different types of tasks. A user study was conducted to address this research problem. A total of 32 participants conducted searches associated with 4 different tasks, varying along several task type dime...

In this paper, we propose to personalize search result content through modeling multiple user behavioral measures in different ways as evidence for implicit relevance feedback for different types of search tasks. The point of this personalization is to predict potentially useful documents based on the type of task, and on multiple behaviors indicat...

Users of digital libraries and other information systems typically carry out searches with very short queries, on the order of two words or so. This makes it very difficult for the systems to disambiguate their queries and identify potentially relevant documents, resulting in sub-optimal retrieval performance. We hypothesize that users will provide...

We present a method to measure some aspects of cognitive effort by a user while reading during a search session. We measured reading eye movement properties and patterns of eye movement in a user study (n=32) of participants carrying out realistic journalism IR work tasks. The results show the cognitive effort measures correlate positively with the...

We report findings that help us better understand the difficulty of tasks which involve information seeking, retrieving, gathering, and use. We examined the data gathered from two interactive information retrieval user studies on how users' perception of task difficulty changes before and after searching for information to solve tasks, and how the...

Users engaged in information search often reformulate or modify their queries. This paper reports on an investigation of how task type and task situation influence users' query reformulation behavior. A controlled experiment was conducted with 48 participants, each working on six web search tasks classified into three types according to the task st...

This paper reports our investigation of differences in users' behavior between difficult and easy search tasks, as well as how these differences vary with different types of tasks. We also report how behavioral predictors of task difficulty vary across task types. In addition, we explored how whole-task-session level user behaviors and within-task-...

Self-assessment of topic/task knowledge is a human metacognitive capacity that impacts information behavior, for example through selection of learning and search strategies. It is often used as a measure in experiments for evaluation of results and those measurements are taken to be generally reliable. We conducted a user study (n=40) to test this...

Many studies have demonstrated that people engage in a variety of different information behaviors when engaging in information seeking. However, standard information retrieval systems such as Web search engines continue to be designed to support mainly one such behavior, specified searching. This situation has led to suggestions that people would b...

Motivation -- On-the-task detection of the task type and task attributes can benefit personalization and adaptation of information systems. Research approach -- A web-based information search experiment was conducted with 32 participants using a multi-stream logging system. The realistic tasks were related directly to the backgrounds of the partici...

We report on a study that investigated the efficacy of four different interactive information retrieval (IIR) systems, each designed to support a specific information-seeking strategy (ISS). These systems were constructed using different combinations of IR techniques (i.e., combinations of different methods of representation, comparison, presentati...

In this paper, we report findings on how user behaviors vary in tasks with different difficulty levels as well as of different types. Two behavioral signals: document dwell time and number of content pages viewed per query, were found to be able to help the system detect when users are working with difficult tasks.

Dwell time as a user behavior has been found in previous studies to be an unreliable predictor of document usefulness, with contextual factors such as the user's task needing to be considered in its interpretation. Task stage has been shown to influence search behaviors including usefulness judgments, as has task type. This paper reports on an inve...

Personalization of information retrieval tailors search towards individual users to meet their particular information needs by taking into account information about users and their contexts, often through implicit sources of evidence such as user behaviors. Task types have been shown to influence search behaviors including usefulness judgments. Thi...

As Digital Libraries (DL) become more aligned with the web architecture, their functional components need to be fundamentally rethought in terms of URIs and HTTP. Annotation, a core scholarly activity enabled by many DL solutions, exhibits a clearly ...

Over the last two decades, Interactive Information Retrieval (IIR) has established a new direction within the long tradition of IR that introduces the user at its center and poses new challenges for system evaluation. IR systems can improve performance by utilizing information about the entire interactive process of search. This approach has so far...

The 3rd Information Interaction in Context Symposium (IIiX'10) was held at Rutgers University in New Brunswick, NJ, USA on 18-21 August, 2010. This report provides an overview of the purpose and program of the conference, as well as information about how to access the Proceedings.

This paper presents an experiment and analysis system framework that allows researchers to design and conduct interactive experiments and analyze data for the evaluation of contextual relationships.

On July 23, 2009 the SIGIR Workshop on the Future of IR Evaluation was held as part of SIGIR in Boston. The program consisted of four keynotes, a boaster and poster session with 20 accepted papers, four breakout groups, and a final panel discussion of ...

Clustering of search results has been shown to be advantageous over the simple list presentation of search results. However, in most clustering inter- faces, the clusters are not adaptive to a user's interaction with the clustering re- sults, and the important question "how to optimize the benefit of a clustering inter- face for a user" has not bee...

There is need for more foundational research in the development of interactive information retrieval systems. The results of a week long discussion by a group of multi­disciplinary researchers have reported here. A brief description of main activities and major recommendations of the workshop are reported here. @InProceedings{fuhr_et_al:DSP:2009:21...

Modern information search systems can benefit greatly from using additional information about the user and the user's behavior, and research in this area is active and growing. Feedback data based on direct interaction (e.g., clicks, scrolling, etc.) as well as on user profiles/preferences has been proven valuable for personalizing the search proce...

Although we see the positive results of information retrieval research embodied throughout the Internet, on our computer desktops, and in many other aspects of daily life, at the same time we notice that people still have a wide variety of difficulties in finding information that is useful in resolving their problematic situations. This suggests th...

This paper reports on an experiment comparing the retrieval effectiveness of an interactive information retrieval (IIR) system which adapts to support different information seeking strategies, with that of a standard baseline IIR system. The experiment, with 32 subjects each searching on 8 different topics, indicates that using the integrated IIR s...

The process of information seeking involves a varied set of tasks and interactions. Exactly how the information seeker judges the relevance of what is retrieved has been a renewed area of interest in information retrieval studies. Various studies have identified facets or categories of relevance which go beyond simple topical relevance, and there h...

People use questions to elicit information from other people in their everyday lives and yet the most common method of obtaining information from a search engine is by posing keywords. There has been research that suggests users are better at expressing their information needs in natural language, however the vast majority of work to improve docume...

In an ideal of personal information management or PIM, people always have just the right information, in the right form and at the right place, to meet their current information needs. Panelists all participated in a special workshop sponsored by the National Science Foundation to consider the challenges of PIM that must be met in order to make sig...

We report on an evaluation of the relationship between document readability, an objective measure related to the length and complexity of words and sentences, and the subjective perception of document relevance of users with a certain level of familiarity to a topic. The research reported here, follow‐up work to our TREC 2004 effort, tries to expla...

The purpose of this study is to investigate what level of difference in precision would be significantly perceived by a human user of an information retrieval system. Not many researches have been conducted with regards to this issue in information retrieval field. Despite the non-significant results, there were several interesting findings in reco...

We present a user modeling system for personalized interaction and tailored retrieval that (1) tracks interactions over time, (2) represents multiple information needs, both short and long term, (3) allows for changes in information needs over time, (4) acquires and updates the user model automatically, without explicit assistance from the user, an...

Designing information systems with the user in mind, generally referred to as ‘user-centered design’ has become an established goal of much of the work in information science. According to this approach, activities such as user studies, usability evaluations, and user analyses (amongst others) must be performed if we are to either advance our field...

Most information systems share a common assumption: information seeking is discrete. Such an assumption neither reflects real-life information seeking processes nor conforms to the perspective of phenomenology, “life is a journey constituted by continuous acquisition of knowledge.” Thus, this study develops and validates a theoretical model that ex...