Carol Peters’s research while affiliated with Institute of Information Science and Technologies "Alessandro Faedo", Italian National Research Council and other places

This introductory chapter begins by explaining briefly what is intended by experimental evaluation in information retrieval in order to provide the necessary background for the rest of this volume. The major international evaluation initiatives that have adopted and implemented in various ways this common framework are then presented and their relationship to CLEF indicated. The second part of the chapter details how the experimental evaluation paradigm has been implemented in CLEF by providing a brief overview of the main activities and results obtained over the last two decades. The aim has been to build a strong multidisciplinary research community and to create a sustainable technical framework that would not simply support but would also empower both research and development and evaluation activities, while meeting and at times anticipating the demands of a rapidly evolving information society.

This volume celebrates the twentieth anniversary of CLEF - the Cross-Language Evaluation Forum for the first ten years, and the Conference and Labs of the Evaluation Forum since – and traces its evolution over these first two decades. CLEF’s main mission is to promote research, innovation and development of information retrieval (IR) systems by anticipating trends in information management in order to stimulate advances in the field of IR system experimentation and evaluation. The book is divided into six parts. Parts I and II provide background and context, with the first part explaining what is meant by experimental evaluation and the underlying theory, and describing how this has been interpreted in CLEF and in other internationally recognized evaluation initiatives. Part II presents research architectures and infrastructures that have been developed to manage experimental data and to provide evaluation services in CLEF and elsewhere. Parts III, IV and V represent the core of the book, presenting some of the most significant evaluation activities in CLEF, ranging from the early multilingual text processing exercises to the later, more sophisticated experiments on multimodal collections in diverse genres and media. In all cases, the focus is not only on describing “what has been achieved”, but above all on “what has been learnt”. The final part examines the impact CLEF has had on the research world and discusses current and future challenges, both academic and industrial, including the relevance of IR benchmarking in industrial settings. Mainly intended for researchers in academia and industry, it also offers useful insights and tips for practitioners in industry working on the evaluation and performance issues of IR tools, and graduate students specializing in information retrieval.

Increasingly people are required to interact or communicate with Information Retrieval (IR) applications in order to find useful information. This interaction commonly takes place through the user interface, which should help users to formulate their queries, refine their searches and understand and examine search results. In multilingual information access, the design of an effective search interface and the provision of functionality to assist the user with searching are vital as users cross the language boundary and interact with materials written in foreign, and potentially unknown, languages. Designing the user interface may also involve the localisation of existing material and services, in addition to providing cross-language search functionalities. This chapter focuses on the users of MLIR/CLIR systems and highlights the issues involved in developing user interfaces, including what form of multilingual search assistance can help users and how to design an interface to support multilingual information access.

This chapter discusses IR system evaluation with particular reference to the multilingual context, and presents the most commonly used measures and models. The main focus is on system performance from the viewpoint of retrieval effectiveness. However, we also discuss evaluation from a user-oriented perspective and address questions such as how to assess whether the system satisfies the requirements of its users. The objective is to give the reader a working knowledge of how to set about MLIR/CLIR system evaluation. In addition, we report on some of the ways in which evaluation experiments and evaluation campaigns have helped to achieve greater understanding of the issues involved in MLIR/CLIR system development and have contributed to advancing the state-of-the-art.

The information retrieval system stands at the core of many information acquisition cycles. Its task is the retrieval of relevant information from document collections in response to a coded query based on an information need. In its general form, when searching unstructured, natural language text produced by a large range of authors, this is a difficult task: in such text there are many different valid ways to convey the same information. Adding to the complexity of the task is an often incomplete understanding of the desired information by the user. In this chapter, we discuss the mechanisms employed for matching queries and (textual) documents within one language, covering some of the peculiarities of a number of widely spoken languages. Effective within-language retrieval is an essential prerequisite for effective multilingual information access. The discussion of within-language information retrieval or monolingual information retrieval can be structured into two main phases: the indexing phase, commonly implemented as a pipeline of indexing steps, producing a representation that is suitable for matching; and the matching phase, which operates on the indexed representations and produces a ranked list of documents that are most likely to satisfy the user’s underlying information need.

We are living in a multilingual world and the diversity in languages which are used to interact with information access systems has generated a wide variety of challenges to be addressed by computer and information scientists. The growing amount of non-English information accessible globally and the increased worldwide exposure of enterprises also necessitates the adaptation of Information Retrieval (IR) methods to new, multilingual settings. Peters, Braschler and Clough present a comprehensive description of the technologies involved in designing and developing systems for Multilingual Information Retrieval (MLIR). They provide readers with broad coverage of the various issues involved in creating systems to make accessible digitally stored materials regardless of the language(s) they are written in. Details on Cross-Language Information Retrieval (CLIR) are also covered that help readers to understand how to develop retrieval systems that cross language boundaries. Their work is divided into six chapters and accompanies the reader step-by-step through the various stages involved in building, using and evaluating MLIR systems. The book concludes with some examples of recent applications that utilise MLIR technologies. Some of the techniques described have recently started to appear in commercial search systems, while others have the potential to be part of future incarnations. The book is intended for graduate students, scholars, and practitioners with a basic understanding of classical text retrieval methods. It offers guidelines and information on all aspects that need to be taken into consideration when building MLIR systems, while avoiding too many ‘hands-on details’ that could rapidly become obsolete. Thus it bridges the gap between the material covered by most of the classical IR textbooks and the novel requirements related to the acquisition and dissemination of information in whatever language it is stored.

The continually growing number of applications creating and/or using collections of natural language digital documents in diverse media and languages means that there is increasing demand for technologies to access the information contained in these archives. When the documents are comprised of media other than text and in languages unfamiliar to the user of the application, then a more complex approach is required than that used for monolingual textual document retrieval. Technologies employed for Multilingual Information Access (MLIA) must be integrated with others in a seamless fashion. In this chapter, we discuss ways in which the technologies developed to implement systems for Multilingual and Cross-Language Information Retrieval (MLIR/CLIR) are adopted in areas that go beyond textual document search such as multimedia retrieval and information extraction over language boundaries. We also describe a range of practical application domains which employ these technologies, thus helping to motivate the need for further developments in MLIR/CLIR.

The special issue of Transactions on Asian Language Information Processing (TALIP) discusses six research papers on Indian language Information Retrieval (IR). The first article, 'The FIRE 2008 Evaluation Exercise' by Prasenjit Majumder and co-workers, provides the motivation and background for the FIRE initiative. It describes how the FIRE 2008 test collection was constructed, and summarizes the approaches adopted by various participants. The authors also discuss the limitations of the datasets, and outline the tasks planned for the next iteration of FIRE. Leveling and Jones in their article,'Sub-word Indexing and Blind Relevance Feedback for English, Bengali, Hindi, and Marathi IR,' try a corpus-based stemming approach based on morpheme induction, as well as sub-word indexing units. The final article, An Information Extraction System for Urdu - A Resource Poor Language' by Smruthi, addresses Natural Language Processing (NLP) tasks for Urdu, a language that is not addressed by any of the other articles.

The goal of the panel is to present some of the main lessons that we have learned in well over a decade of experimental evaluation and to promote discussion with respect to what the future objectives in this field should be.