Search Interface Design and Evaluation
Abstract
Information seeking and use is now routine in people’s everyday lives. Searching through various information retrieval systems such as web search engines or search functions within information systems allows users to gain access to information on the Internet. Whereas most research in this area has focused on the algorithms behind the search engines from technical perspectives, in this monograph, the authors focus on the search interface, the place where searchers interact with the search system. Search Interface Design and Evaluation reviews the research on the design and evaluation of search user interfaces of the past 10 years. The authors’ primary goal is to integrate state-of-the-art search interface research in the areas of information seeking behavior, information retrieval, and human-computer interaction. The monograph describes the history and background of the development of the search interface and introduces information search behavior models that help conceptualize users’ information needs. The authors also characterize the major components of search interfaces that support different subprocesses based on Marchonini’s information seeking process model, review the design of search interfaces for different user groups, and identify evaluation methods of search interfaces and how they were implemented. Lastly, they provide an outlook on the future trends of search interfaces that includes conversational search interfaces, search interfaces supporting serendipity and creativity, and searching in immersive and virtual reality environments.
... Most previous studies on traditional text-based search services developed a method to evaluate web search service quality for measuring the effectiveness of search engine performance (Can et al., 2004;Choudhary et al., 2017;S anchez et al., 2018;Spink, 2002;Vaughan, 2004). The objective of these effectiveness measurement is to determine the search engine performance that most effectively aligns with the individual preferences of users (Liu et al., 2021). From the technical perspectives, this study has an objective to deliver the evaluation aspect of VSS performance, which has not been done in prior studies. ...
... It not only identifies the service's quality performance using a technology-centered perspective, but these items also measure the behavior with the VSS users with the behavior-centered perspective. The integration of these two viewpoints is also supported by the work of Liu et al. (2021), which emphasizes the importance of comprehending the design of VSSs by considering both behavioral and technological factors to ensure that the VSSs developed are suitable to a wide variety of people and can satisfy their search needs. The development of VSS effectiveness scale was done using the perceptions of actual users to determine how well VSS performs and satisfies their needs for information acquisition, which is also in line with the methodology used from Wu et al. (2019) research on intelligence search engine. ...
Despite the potential of visual search services (VSSs) to increase the size of the global visual search market, there is no VSS effectiveness measurement scale in the literature. Prior studies of traditional text-based search services have aimed to establish a method for assessing the performance of web search services. Our primary objective is to integrate state-of-the-art research in the fields of information-seeking behavior, visual search performance, and human–computer interaction pertaining to evaluating VSS effectiveness. The key dimensions of VSS effectiveness were identified by adopting the Delphi method, and the associated measures were developed. Then, survey data were collected from 426 VSS users, and the developed measures of VSS effectiveness were validated using the exploratory factor analysis technique. Finally, the scale developed includes 19 measures belonging to three key dimensions: user dependency on VSSs, the perceived ability to use VSSs, and VSS retrieval performance. The nomological validity of the developed scale’s three dimensions was also ensured and was evident from the positive correlations between them and VSS user continuance intention. Overall, we find that the better the functionality of a VSS is, the more users with a high perceived ability can acquire better information. This study also contributes to the theory development in the literature on search service effectiveness measures in terms of highlighting the VSS feature to be considered when providing a VSS that is suitable for the user’s preference.
Recently, virtual reality (VR) technology has become more widespread. Humans increasingly interact with information in VR, and a detailed look into those activities is warranted. Thus, a scoping literature review (PRISMA-ScR) is conducted. It overviews all relevant literature about information-seeking behaviour in VR, focusing on existing models and theories. Out of 536 publications, 37 qualify for this review. Eight publications show an understanding related to information behaviour theories from information science. Pressingly, no publications relate models, frameworks or general theories of information seeking to VR. This review overviews VR-related cognitive and behavioural human factors based on this research gap. Those factors include immersion and presence, affordances, embodiment, cognitive load, human error, flow and engagement. The review is concluded with an explorative framework for future research that is constructed with Marchionini’s process model of information seeking as a baseline and in conjunction with the discussed human factors.
The interdisciplinary field known as digital humanities (DH) is represented in various forms in the teaching and research practiced in iSchools. Building on the work of an iSchools organization committee charged with exploring digital humanities curricula, we present findings from a series of related studies exploring aspects of DH teaching, education, and research in iSchools, often in collaboration with other units and disciplines. Through a survey of iSchool programs and an online DH course registry, we investigate the various education models for DH training found in iSchools, followed by a detailed look at DH courses and curricula, explored through analysis of course syllabi and course descriptions. We take a brief look at collaborative disciplines with which iSchools cooperate on DH research projects or in offering DH education. Next, we explore DH careers through an analysis of relevant job advertisements. Finally, we offer some observations about the management and administrative challenges and opportunities related to offering a new iSchool DH program. Our results provide a snapshot of the current state of digital humanities in iSchools which may usefully inform the design and evolution of new DH programs, degrees, and related initiatives.
Creativity is a crucial factor in finding novel and useful visualization and interaction techniques, but its emergence is contingent on the right conditions. The focus of visualization research has traditionally been on techniques, and to a lesser degree on the process of creating them with domain experts and end users. This paper focuses on the collaborative design of visualizations for information seeking and knowledge management. The difficult, yet common challenge in any visualization project is to find meaningful visual representations and useful interaction techniques to carry out complex analysis tasks. The unique difficulty for preparing co-design activities for visualization lies in the gap between the abstract nature of data and the concrete form of visual representations. To bridge this gap, our co-design framework for visualization places particular emphasis on actors, activities, and artifacts as categories that expand the focus of visualization design beyond the traditional triad of users, tasks, and data. Drawing from general co-design principles, the framework is developed and validated during the course of two case studies in the context of information management systems and library collection databases. Based on observed patterns during the case studies, practical tactics provide advice on carrying out co-design in information visualization.
The use of conversational agents within the aerospace industry offers quick and concise answers to complex situations. The aerospace domain is characterized by products and systems that are built over decades of engineering to reach high levels of performance within complex environments. Current development in conversational agents can leverage the latest retrieval and language model to refine the system's question-answering capabilities. However, evaluating the added-value of such a system in the context of industrial applications such as pilots in a cockpit is complex. This paper describes how a conversational agent is implemented and evaluated , with particular references to how state-of-the-art technologies can be adapted to the domain specificity. Preliminary findings of a controlled user experiment suggest that user perception of the usefulness of the system in completing the search task and the system's responses to the relevance of the topic are good predictors of user search performance. User satisfaction with the system's responses may not be a good predictor of user search performance.
The development of usable visualization solutions is essential for ensuring both their adoption and effectiveness. User-centered design principles, which involve users throughout the entire development process, have been shown to be effective in numerous information visualization endeavors. We describe how we applied these principles in scientific visualization over a two year collaboration to develop a hybrid in situ/post hoc solution tailored towards combustion researcher needs. Furthermore, we examine the importance of user-centered design and lessons learned over the design process in an effort to aid others seeking to develop effective scientific visualization solutions.
Overview of the challenge of bridging the gap between research and design through an examination of the human act of reading.
Search tasks play an important role in the study and development of interactive information retrieval (IIR) systems. Prior work has examined how search tasks vary along dimensions such as the task's main activity, end goal, structure, and complexity. Recently, researchers have been exploring task complexity from the perspective of cognitive complexity---related to the types (and variety) of mental activities required by the task. Anderson & Krathwohl's two-dimensional taxonomy of learning has been a commonly used framework for investigating tasks from the perspective of cognitive complexity. A&K's 2D taxonomy involves a cognitive process dimension and an orthogonal knowledge dimension. Prior IIR research has successfully leveraged the cognitive process dimension of this 2D taxonomy to develop search tasks and investigate their effects on searchers' needs, perceptions, and behaviors. However, the knowledge dimension of the taxonomy has been largely ignored. In this conceptual paper, we argue that future IIR research should consider both dimensions of A&K's taxonomy. Specifically, we discuss related work, present details on both dimensions of A&K's taxonomy, and explain how to use the taxonomy to develop search tasks and learning assessment materials. Additionally, we discuss how considering both dimensions of A&K's taxonomy has important implications for future IIR research.
Introduction. Previous studies on music information behaviour have focused on describing all the different activities used when interacting with music. However, a general picture of this behaviour has yet to emerge. The purpose of this research is to model the information behaviour of music fans through the perspective of social practice theory. Method. Qualitative method was employed, using in-depth interviews and the observation of participants' homes and practices. Eighteen music fans with diverse socio-demographic characteristics living in Medellín, Colombia, participated in the research. Analysis. Qualitative deductive analysis was performed using a tree of categories extracted from a conceptual framework. Categories were assigned to the data, and then compared and contrasted. Results. Information activities that comprise practices are presented as existing within continuums from active behaviour to passive behaviour. Factors that affect and influence practices are presented in four groups: personal benefits, social benefits, extrinsic conditions and worldview. A model for music information practices that integrates all these elements is presented as the main result from this study. Conclusions. This research contributes to the ongoing discussion about music information behaviour, presenting a model that describes an information practice which has the potential to describe other types of information behaviour.
Conversational agents (CAs) are software-based systems designed to interact with humans using natural language and have attracted considerable research interest in recent years. Following the Computers Are Social Actors paradigm, many studies have shown that humans react socially to CAs when they display social cues such as small talk, gender, age, gestures, or facial expressions. However, research on social cues for CAs is scattered across different fields, often using their specific terminology, which makes it challenging to identify, classify, and accumulate existing knowledge. To address this problem, we conducted a systematic literature review to identify an initial set of social cues of CAs from existing research. Building on classifications from interpersonal communication theory, we developed a taxonomy that classifies the identified social cues into four major categories (i.e., verbal, visual, auditory, invisible) and ten subcategories. Subsequently, we evaluated the mapping between the identified social cues and the categories using a card sorting approach in order to verify that the taxonomy is natural, simple, and parsimonious. Finally, we demonstrate the usefulness of the taxonomy by classifying a broader and more generic set of social cues of CAs from existing research and practice. Our main contribution is a comprehensive taxonomy of social cues for CAs. For researchers, the taxonomy helps to systematically classify research about social cues into one of the taxonomy's categories and corresponding subcategories. Therefore, it builds a bridge between different research fields and provides a starting point for interdisciplinary research and knowledge accumulation. For practitioners, the taxonomy provides a systematic overview of relevant categories of social cues in order to identify, implement, and test their effects in the design of a CA.
Users often fail to formulate their complex information needs in a single query. As a consequence, they may need to scan multiple result pages or reformulate their queries, which may be a frustrating experience. Alternatively, systems can improve user satisfaction by proactively asking questions of the users to clarify their information needs. Asking clarifying questions is especially important in conversational systems since they can only return a limited number of (often only one) result(s).
In this paper, we formulate the task of asking clarifying questions in open-domain information-seeking conversational systems. To this end, we propose an offline evaluation methodology for the task and collect a dataset, called Qulac, through crowdsourcing. Our dataset is built on top of the TREC Web Track 2009-2012 data and consists of over 10K question-answer pairs for 198 TREC topics with 762 facets. Our experiments on an oracle model demonstrate that asking only one good question leads to over 170% retrieval performance improvement in terms of [email protected], which clearly demonstrates the potential impact of the task. We further propose a retrieval framework consisting of three components: question retrieval, question selection, and document retrieval. In particular, our question selection model takes into account the original query and previous question-answer interactions while selecting the next question. Our model significantly outperforms competitive baselines. To foster research in this area, we have made Qulac publicly available.
The validity of environmental simulations depends on their capacity to replicate responses produced in physical environments. However, very few studies validate navigation differences in immersive virtual environments, even though these can radically condition space perception and therefore alter the various evoked responses. The objective of this paper is to validate environmental simulations using 3D environments and head-mounted display devices, at behavioural level through navigation. A comparison is undertaken between the free exploration of an art exhibition in a physical museum and a simulation of the same experience. As a first perception validation, the virtual museum shows a high degree of presence. Movement patterns in both ‘museums’ show close similarities, and present significant differences at the beginning of the exploration in terms of the percentage of area explored and the time taken to undertake the tours. Therefore, the results show there are significant time-dependent differences in navigation patterns during the first 2 minutes of the tours. Subsequently, there are no significant differences in navigation in physical and virtual museums. These findings support the use of immersive virtual environments as empirical tools in human behavioural research at navigation level.
Research highlights
The latest generation HMDs show a high degree of presence. There are significant differences in navigation patterns during the first 2 minutes of a tour. Adaptation time need to be considered in future research. Training rooms need to be realistic, to avoid the ‘wow’ effect in the main experiment. Results support the use of Virtual Reality and the latest HMDs as empirical tools in human behavioural research at navigation level.
Two general challenges faced by data analysis are the existence of noise and the extraction of meaningful information from collected data. In this study, we used a multiscale framework to reduce the effects caused by noise and to extract explainable geometric properties to characterize finite metric spaces. We conducted lab experiments that integrated the use of eye‐tracking, electrodermal activity (EDA), and user logs to explore users' information‐seeking behaviors on search engine result pages (SERPs). Experimental results of 1,590 search queries showed that the proposed strategies effectively predicted query‐level user satisfaction using EDA and eye‐tracking data. The bootstrap analysis showed that combining EDA and eye‐tracking data with user behavior data extracted from user logs led to a significantly better linear model fit than using user behavior data alone. Furthermore, cross‐user and cross‐task validations showed that our methods can be generalized to different search engine users performing different preassigned tasks.
Modern User Interfaces (UIs) are increasingly expected to be plastic, in the sense that they retain a constant level of usability, even when subjected to context (platform, user, and environment) changes at runtime. Adaptive UIs have been promoted as a solution for context variability due to their ability to automatically adapt to the context-of-use at runtime. However, evaluating end-user satisfaction of adaptive UIs is a challenging task, because the UI and the context-of-use are both constantly changing. Thus, an acceptance analysis of UI adaptation features should consider the context-of-use when adaptations are triggered. Classical usability evaluation methods like usability tests mostly focus on a posteriori analysis techniques and do not fully exploit the potential of collecting implicit and explicit user feedback at runtime. To address this challenge, we present an on-the-fly usability testing solution that combines continuous context monitoring together with collection of instant user feedback to assess end-user satisfaction of UI adaptation features. The solution was applied to a mobile Android mail application, which served as basis for a usability study with 23 participants. A data-driven end-user satisfaction analysis based on the collected context information and user feedback was conducted. The main results show that most of the triggered UI adaptation features were positively rated.
Visual analytics (VA) research provides helpful solutions for interactive visual data analysis when exploring large and complex datasets. Due to recent advances in eye tracking technology, promising opportunities arise to extend these traditional VA approaches. Therefore, we discuss foundations for eye tracking support in VA systems. We first review and discuss the structure and range of typical VA systems. Based on a widely used VA model, we present five comprehensive examples that cover a wide range of usage scenarios. Then, we demonstrate that the VA model can be used to systematically explore how concrete VA systems could be extended with eye tracking, to create supportive and adaptive analytics systems. This allows us to identify general research and application opportunities, and classify them into research themes. In a call for action, we map the road for future research to broaden the use of eye tracking and advance visual analytics.
Although various methods have been developed to evaluate conversational interfaces, there has been a lack of methods specifically focusing on evaluating user experience. This paper reviews the understandings of user experience (UX) in conversational interfaces literature and examines the six questionnaires commonly used for evaluating conversational systems in order to assess the potential suitability of these questionnaires to measure different UX dimensions in that context. The method to examine the questionnaires involved developing an assessment framework for main UX dimensions with relevant attributes and coding the items in the questionnaires according to the framework. The results show that (i) the understandings of UX notably differed in literature; (ii) four questionnaires included assessment items, in varying extents, to measure hedonic, aesthetic and pragmatic dimensions of UX; (iii) while the dimension of affect was covered by two questionnaires, playfulness, motivation, and frustration dimensions were covered by one questionnaire only. The largest coverage of UX dimensions has been provided by the Subjective Assessment of Speech System Interfaces (SASSI). We recommend using multiple questionnaires to obtain a more complete measurement of user experience or improve the assessment of a particular UX dimension.
RESEARCH HIGHLIGHTS
Varying understandings of UX in conversational interfaces literature. A UX assessment framework with UX dimensions and their relevant attributes. Descriptions of the six main questionnaires for evaluating conversational interfaces. A comparison of the six questionnaires based on their coverage of UX dimensions.
Information seeking and access are essential for users in all walks of life, from addressing personal needs such as finding flights to locating information needed to complete work tasks. Over the past decade or so, the general needs of people with impairments have increasingly been recognized as something to be addressed, an issue embedded both in international treaties and in state legislation. The same tendency can be found in research, where a growing number of user studies including people with impairments have been conducted. The purpose of these studies is typically to uncover potential barriers for access to information, especially in the context of inaccessible search user interfaces. This literature review provides an overview of research on the information seeking and searching of users with impairments. The aim is to provide an overview to both researchers and practitioners who work with any of the user groups identified. Some diagnoses are relatively well represented in the literature (for instance, visual impairment), but there is very little work in other areas (for instance, autism) and in some cases no work at all (for instance, aphasia). Gaps are identified in the research, and suggestions are made regarding areas where further research is needed.
Depression is an affective disorder with distinctive autobiographical memory impairments, including negative bias, overgeneralization and reduced positivity. Several clinical therapies address these impairments, and there is an opportunity to develop new supports for treatment by considering depression-associated memory impairments within design. We report on interviews with ten experts in treating depression, with expertise in both neuropsychology and cognitive behavioral therapies. The interviews explore approaches for addressing each of these memory impairments. We found consistent use of positive memories for treating all memory impairments, the challenge of direct retrieval, and the need to support the experience of positive memories. We aim to sensitize HCI researchers to the limitations of memory technologies, broaden their awareness of memory impairments beyond episodic memory recall, and inspire them to engage with this less explored design space. Our findings open up new design opportunities for memory technologies for depression, including positive memory banks for active encoding and selective retrieval, novel cues for supporting generative retrieval, and novel interfaces to strengthen the reliving of positive memories.
Popular messaging platforms such as Slack have given rise to thousands of applications (or bots) that users can engage with individually or as a group. In this paper, we study the use of searchbots (i.e., bots that perform specific types of searches) during collaborative information-seeking tasks mediated through Slack. We report on a user study in which 27 pairs of participants were exposed to three searchbot conditions (a within-subjects design). In the first condition, participants completed the task by searching independently and coordinating through Slack (no searchbot). In the second condition, participants could only search inside of Slack using the searchbot. In the third condition, participants could both search inside of Slack using the searchbot and outside of Slack using their own independent search interfaces. We investigate four research questions focusing on the influence of the searchbot condition on outcomes associated with: (RQ1) participants' levels of workload, (RQ2) collaborative awareness, (RQ3) experiences interacting with the searchbot, and (RQ4) search behaviors. Our results suggest opportunities and challenges in designing searchbots to support collaborative search. On one hand, access to the searchbot resulted in more collaborative awareness, ease of coordination, and fewer duplicated searches. On the other hand, forcing participants to share the querying environment resulted in fewer overall queries, fewer query refinements by individuals, and greater levels of effort. We discuss the implications of our findings for designing effective searchbots to support collaborative search.
Purpose
This study explores the effects of cognitive load on the propensity to reformulate queries during information seeking on the web.
Design/methodology/approach
This study employs an experimental design to analyze the effect of manipulations of cognitive load on the propensity for query reformulation between experimental and control groups. In total, three affective components that contribute to cognitive load were manipulated: mental demand, temporal demand and frustration.
Findings
A significant difference in the propensity of query reformulation behavior was found between searchers exposed to cognitive load manipulations and searchers who were not exposed. Those exposed to cognitive load manipulations made half as many search query reformulations as searchers not exposed. Furthermore, the National Aeronautical and Space Administration Task Load Index (NASA-TLX) cognitive load scores of searchers who were exposed to the three cognitive load manipulations were higher than those of searchers who were not exposed indicating that the manipulation was effective. Query reformulation behavior did not differ across task types.
Originality/value
The findings suggest that a dual-task method and NASA-TLX assessment serve as good indicators of cognitive load. Because the findings show that cognitive load hinders a searcher's interaction with information search tools, this study provides empirical support for reducing cognitive load when designing information systems or user interfaces.
Information visualizations are an efficient means to support the users in understanding large amounts of complex, interconnected data; user comprehension, however, depends on individual factors such as their cognitive abilities. The research literature provides evidence that user-adaptive information visualizations positively impact the users' performance in visualization tasks. This study attempts to contribute towards the development of a computational model to predict the users' success in visual search tasks from eye gaze data and thereby drive such user-adaptive systems. State-of-the-art deep learning models for time series classification have been trained on sequential eye gaze data obtained from 40 study participants' interaction with a circular and an organizational graph. The results suggest that such models yield higher accuracy than a baseline classifier and previously used models for this purpose. In particular, a Multivariate Long Short Term Memory Fully Convolutional Network (MLSTM-FCN) shows encouraging performance for its use in on-line user-adaptive systems. Given this finding, such a computational model can infer the users' need for support during interaction with a graph and trigger appropriate interventions in user-adaptive information visualization systems. This facilitates the design of such systems since further interaction data like mouse clicks is not required.
Significant amounts of our time and energy are devoted to creating, managing, and avoiding information. Computers and telecommunications technology have extended our regard for information and are driving changes in how we learn, work, and play. One result of these developments is that skills and strategies for storing and retrieving information have become more essential and more pervasive in our culture. This book considers how electronic technologies have changed these skills and strategies and augmented the fundamental human activity of information seeking. The author makes a case for creating new interface designs that allow the information seeker to choose what strategy to apply according to their immediate needs. Such systems may be designed by providing information seekers with alternative interface mechanisms for displaying and manipulating multiple levels of representation for information objects. Information Seeking in Electronic Environments is essential reading for researchers and graduate students in information science, human-computer interaction, and education, as well as for designers of information retrieval systems and interfaces for digital libraries and archives.
Purpose
A substantial number of models have been developed over the years, with the purpose of describing the information seeking and searching of people in various user groups and contexts. Several models have been frequently applied in user studies, but are rarely included in research on participants with impairments. Models are purposeful when developing theories. Consequently, it might be valuable to apply models when studying this user group, as well. The purpose of this study was to explore whether existing models are applicable in describing the online information seeking and searching of users with impairments, with an overall aim to increase the use of models in studies involving impairments.
Design/methodology/approach
Six models were selected according to the following criteria: the model should address information seeking or searching, include the interaction between users and systems whilst incorporating assistive technology. Two user groups were selected from each of the categories: cognitive, sensory and motor impairments, namely dyslexia, autism, blindness, deafness, paralysation and Parkinson's. The models were then analysed based on known barriers reported for these cohorts.
Findings
All the selected models had potential to be applied in user studies involving impairments. While three of the models had the highest potential to be used in the current form, the other three models were applicable either through minor revisions or by combining models.
Originality/value
This study contributes with a new perspective on the use of models in information seeking and searching research on users with impairments.
Appropriate evaluation is a key component in visualization research. It is typically based on empirical studies that assess visualization components or complete systems. While such studies often include the user of the visualization, empirical research is not necessarily restricted to user studies but may also address the technical performance of a visualization system such as its computational speed or memory consumption. Any such empirical experiment faces the issue that the underlying visualization is becoming increasingly sophisticated, leading to an increasingly difficult evaluation in complex environments. Therefore, many of the established methods of empirical studies can no longer capture the full complexity of the evaluation. One promising solution is the use of data-rich observations that we can acquire during studies to obtain more reliable interpretations of empirical research. For example, we have been witnessing an increasing availability and use of physiological sensor information from eye tracking, electrodermal activity sensors, electroencephalography, etc. Other examples are various kinds of logs of user activities such as mouse, keyboard, or touch interaction. Such data-rich empirical studies promise to be especially useful for studies in the wild and similar scenarios outside of the controlled laboratory environment. However, with the growing availability of large, complex, time-dependent, heterogeneous, and unstructured observational data, we are facing the new challenge of how we can analyze such data. This challenge can be addressed by establishing the subfield of visualization for visualization (Vis4Vis): visualization as a means of analyzing and communicating data from empirical studies to advance visualization research.
Previous work has shown that some user cognitive abilities relevant for processing information visualizations can be predicted from eye-tracking data. Performing this type of user modeling is important for devising visualizations that can detect a user's abilities and adapt accordingly during the interaction. In this article, we extend previous user modeling work by investigating for the first time interaction data as an alternative source to predict cognitive abilities during visualization processing when it is not feasible to collect eye-tracking data. We present an extensive comparison of user models based solely on eye-tracking data, on interaction data, as well as on a combination of the two. Although we found that eye-tracking data generate the most accurate predictions, results show that interaction data can still outperform a majority-class baseline, meaning that adaptation for interactive visualizations could be enabled even when it is not feasible to perform eye tracking, using solely interaction data. Furthermore, we found that interaction data can predict several cognitive abilities with better accuracy at the very beginning of the task than eye-tracking data, which are valuable for delivering adaptation early in the task. We also extend previous work by examining the value of multimodal classifiers combining interaction data and eye-tracking data, with promising results for some of our target user cognitive abilities. Next, we contribute to previous work by extending the type of visualizations considered and the set of cognitive abilities that can be predicted from either eye-tracking data and interaction data. Finally, we evaluate how noise in gaze data impacts prediction accuracy and find that retaining rather noisy gaze datapoints can yield equal or even better predictions than discarding them, a novel and important contribution for devising adaptive visualizations in real settings where eye-tracking data are typically noisier than in laboratory settings.
User engagement has become an important outcome measure in interactive information retrieval (IIR) research, as commercial (e.g., search engines and e-commerce companies) and educational (e.g., libraries) enterprises focus on capturing and retaining customers. User engagement pertains to the kind of investment – emotional, cognitive, behavioural – the user is willing to make in an application. While research has shown how characteristics of users (e.g., individual differences and preferences) and the systems and content with which they interact influence engagement, less is understood about how the tasks people perform using digital applications affect their engagement. Drawing upon a wealth of literature in IIR, this study examined the effects of task on search engagement in a within-subjects Amazon Mechanical Turk (MTurk) experiment. Participants completed six search tasks on different task topics using task versions that included or excluded items and dimensions in the task descriptions. Items refer to things being compared (alternatives) and dimensions correspond to attributes by which items may differ. The task topics were meant to influence user interest in the task, and the versions were intended to manipulate the task doer's degree of certainty as they planned and performed the task, with the expectation that these factors would affect their self-reported engagement. We captured self-reported task perceptions (e.g., complexity, difficulty, interest) and logged search behaviours (e.g., querying, bookmarking) to both validate our manipulations and to understand how these variables related to engagement. Using multi-level modelling (MLM) we discovered that task topic affected user engagement, whereas task version had limited effects. However, participants’ perceptions of the tasks as interesting, difficult, and so on affected their engagement. Through the self-report and behavioural data, we observed that effort (more search engine results page exploration, greater perceived task difficulty) had a negative effect on engagement, while bookmarking pages and the ability to understand the task and how to complete it was associated with positive engagement. These results have implications for designing search tasks, deciphering the relationship between user experience and task complexity in IIR experiments, and aligning self-reports and search behaviours in evaluating online search engagement.
Purpose
In order to understand the totality, diversity and richness of human information behavior, increasing research attention has been paid to examining serendipity in the context of information acquisition. However, several issues have arisen as this research subfield has tried to find its feet; we have used different, inconsistent terminology to define this phenomenon (e.g. information encountering, accidental information discovery, incidental information acquisition), the scope of the phenomenon has not been clearly defined and its nature was not fully understood or fleshed-out.
Design/methodology/approach
In this paper, information encountering (IE) was proposed as the preferred term for serendipity in the context of information acquisition.
Findings
A reconceptualized definition and scope of IE was presented, a temporal model of IE and a refined model of IE that integrates the IE process with contextual factors and extends previous models of IE to include additional information acquisition activities pre- and postencounter.
Originality/value
By providing a more precise definition, clearer scope and richer theoretical description of the nature of IE, there was hope to make the phenomenon of serendipity in the context of information acquisition more accessible, encouraging future research consistency and thereby promoting deeper, more unified theoretical development.
Models of human information seeking reveal that search, in particular ad-hoc retrieval, is non-linear and iterative. Despite these findings, today’s search user interfaces do not support non-linear navigation, like for example backtracking in time. We propose QueryCrumbs, a compact and easy-to-understand visualization for navigating the search query history supporting iterative query refinement. We apply a multi-layered interface design to support novices and first-time users as well as intermediate and expert users. The visualization is evaluated with novice users in a formative user study, with experts in a think aloud test and its usage in a long-term study with software logging. The formative evaluation showed that the interactions can be easily performed, and the visual encodings were well understood without instructions. Results indicate that QueryCrumbs can support users when searching for information in an iterative manner. The evaluation with experts showed that expert users can gain valuable insights into the back-end search engine by identifying specific patterns in the visualization. In a long-term usage study, we observed an uptake of the visualization, indicating that users deem QueryCrumbs beneficial for their search interactions.
This paper presents an analytical review of the existing literature about Human Information Behaviour (HIB) in the context of Serious Leisure (SL). Various forms of information activities in this context have been identified and categorised to depict common patterns in information seeking and sharing. The findings show the informational aspect of SL is a rich and productive topic in HIB because it entails the continuous pursuit and recreation of knowledge and often involves several types of information-related actions including information seeking, searching, browsing, retrieving, gathering, saving, organising, sharing, evaluating, measuring, analysing, producing and disseminating. The paper also presents a tentative model of predominant information sources in SL based on the analytical literature review and theoretical speculation. This preliminary model categorises SL activities into three main groups including (1) intellectual pursuits, (2) creating or collecting physical objects/materials/products and (3) experiential activities. Similarly, the paper categorises SL participants into three major groups of appreciators, producers/collectors and performers. Each category has its own priorities, source preferences and information behaviour. The findings also indicate that exploring various aspects of HIB in the SL domain is still an emerging ground and that the majority of studies have been thus far been qualitative. As a result, further research needs to be done to gain a more comprehensive picture of this area and to validate the growing knowledge base with larger samples and further settings.
People with visual impairments often rely on screen readers when interacting with computer systems. Increasingly, these individuals also make extensive use of voice-based virtual assistants (VAs). We conducted a survey of 53 people who are legally blind to identify the strengths and weaknesses of both technologies, and the unmet opportunities at their intersection. We learned that virtual assistants are convenient and accessible, but lack the ability to deeply engage with content (e.g., read beyond the first few sentences of an article), and the ability to get a quick overview of the landscape (e.g., list alternative search results and suggestions). In contrast, screen readers allow for deep engagement with content (when content is accessible), and provide fine-grained navigation and control, but at the cost of reduced walk-up-and-use convenience. Based on these findings, we implemented VERSE (Voice Exploration, Retrieval, and SEarch), a prototype that extends a VA with screen-reader-inspired capabilities, and allows other devices (e.g., smartwatches) to serve as optional input accelerators. In a usability study with 12 blind screen reader users we found that VERSE meaningfully extended VA functionality. Participants especially valued having access to multiple search results and search verticals.
In order to investigate how a VR study context influence participants' User Experience responses of an interactive system an UX evaluation of the same in-vehicle systems was conducted in the field and in virtual reality. The virtual environment featured a virtual road scene and an interactive in-car environment, paired with a physical set-up containing a table-mounted steering wheel and a touch-sensitive panel. The VR system enabled a high estimation of presence and focus, however participants voiced less affect in the virtual setting and had difficulty in separating judgments of the VR experience from the UX of the in-vehicle systems. No significant differences in UX questionnaire data were identified between VR and the field, but there were correlations between rated presence in the VR system and UX ratings, especially for reported stimulation. Based on the lessons learned, a number of methodological and technological consequences are recommended in the paper, such as the need for more dynamic movement behaviour, improved resolution of graphics of the virtual vehicle and introducing the test leader as visually present in the virtual environment.
Music-streaming platforms offer users a large amount of content for consumption. Finding the right music can be challenging and users often need to search through extensive catalogs provided by these platforms. Prior research has focused on general-domain web search, which is designed to meet a broad range of user goals. Here, we study search in the domain of music, seeking to understand how and why people use search and how they evaluate their search experiences on a music-streaming platform. Over two studies, we conducted semi-structured interviews with 27 participants, asking about their search habits and preferences, and observing their behavior while searching for music. Analysis revealed participants evaluated their search experiences along two dimensions: success and effort. Importantly, how participants perceived success and effort differed by their mindset, or the way they assessed the results of their query. We conclude with recommendations to improve the user experience of music search.
Implicit detection of relevance has been approached by many during the last decade. From the use of individual measures to the use of multiple features from different sources (multimodality), studies have shown the feasibility to automatically detect whether a document is relevant. Despite promising results, it is not clear yet to what extent multimodality constitutes an effective approach compared to unimodality. In this article, we hypothesize that it is possible to build unimodal models capable of outperforming multimodal models in the detection of perceived relevance. To test this hypothesis, we conducted three experiments to compare unimodal and multimodal classification models built using a combination of 24 features. Our classification experiments showed that a univariate unimodal model based on the left‐click feature supports our hypothesis. On the other hand, our prediction experiment suggests that multimodality slightly improves early classification compared to the best unimodal models. Based on our results, we argue that the feasibility for practical applications of state‐of‐the‐art multimodal approaches may be strongly constrained by technology, cultural, ethical, and legal aspects, in which case unimodality may offer a better alternative today for supporting relevance detection in interactive information retrieval systems.
Increasingly popular, long-distance running events (LDRE) attract not just runners but an exponentially increasing number of spectators. Due to the long duration and broad geographic spread of such events, interactions between them are limited to brief moments when runners (R) pass by their supporting spectators (S). Current technology is limited in its potential for supporting interactions and mainly measures and displays basic running information to spectators who passively consume it. In this paper, we conducted qualitative studies for an in-depth understanding of the R&S' shared experience during LDRE and how technology can enrich this experience. We propose a two-layer DyPECS framework, highlighting the rich dynamics of the R&S multi-faceted running journey and of their micro-encounters. DyPECS is enriched by the findings from our in depth qualitative studies. We finally present design implications for the multi-facet co-experience of R&S during LDRE.