Table 2 - uploaded by Nicola Ferro
Content may be subject to copyright.
Language Modes for CHiC Experiments

Language Modes for CHiC Experiments

Source publication
Article
Full-text available
The paper for the CHiC pilot lab describes the motivation, tasks, Europeana collections and topics, evaluation measures as well as the submitted and analyzed information retrieval runs. In its first year, CHiC offered three tasks: ad-hoc, which measured retrieval effectiveness according to relevance of the ranked retrieval results (standard 1000 do...

Context in source publication

Context 1
... tasks were offered with the same set of topics and in three language modes: (i) monolingual (query and document language are the same), (ii) bilingual (query and document languages are different), (iii) multilingual (documents in multiple lan- guages, i.e. the whole Europeana collection will be searched). This allowed the partic- ipants to experiment with a number of language variations (table 2). ...

Similar publications

Chapter
Full-text available
This chapter describes the lessons learnt from the ad hoc track at CLEF in the years 2000 to 2009. This contribution focuses on Information Retrieval (IR) for languages other than English (monolingual IR), as well as bilingual IR (also termed “cross-lingual”; the request is written in one language and the searched collection in another), and multil...
Preprint
Full-text available
The majority of the published studies on multilingualism concentrate on education, while only a few published papers describe this concept outside the education environment. The current descriptive study uses Makkah City as an example to describe two multilingual phenomena: Umrah and Hajj as examples of permanent and temporary multilingualism pheno...
Article
Full-text available
El nivel actual de internacionalización de las universidades españolas está demandando un aumento en el dominio de lenguas extranjeras entre los estudiantes. Numerosas universidades españolas están introduciendo asignaturas impartidas en lenguas diferentes del español en sus planes de estudio, principalmente en inglés, a través de diferentes progra...
Article
Full-text available
Second-language learners frequently encounter challenges when solving word problems that are not written in their first language. This study compares the mathematics word-problem performance of 5th-grade learners using English and Filipino as the languages of assessment. The study consists of 32 5th-grade students from a public elementary schoo...

Citations

... Cultural Heritage in CLEF (CHiC, 2011-2013) promoted systematic and large-scale evaluation of digital libraries and, more in general, cultural heritage information access systems, using the huge Europeana dataset, aggregating information from libraries, museums, and archives [144,371,372]; aimed at evaluating multimedia analysis and retrieval techniques on biodiversity data for species identification, namely images for plants, audio for birds, and video for fishes [200][201][202][204][205][206]; News Recommendation Evaluation (NewsREEL, 2014-2017) focused on evaluation of news recommender systems in real-time by offering access to the APIs of a commercial system [188,233,234,272] Living Labs (LL4IR, 2015-2016) dealt with evaluation of ranking systems in a live setting with real users in their natural task environments, acting as a proxy between commercial organizations (live environments) and lab participants (experimental systems) [423]; Social Book Search (SBS, 2015-2016) investigated techniques to support users in complex book search tasks that involve more than just a query and results list [242,243]. Microblog Cultural Contextualization (MC2, 2016-2017) investigated techniques to support users in complex book search tasks that involve more than just a query and results list [117,162]. ...
Conference Paper
Full-text available
2019 marks the 20th birthday for CLEF, an evaluation campaign activity which has applied the Cranfield evaluation paradigm to the testing of multilingual and multimodal information access systems in Europe. This paper provides a summary of the motivations which led to the establishment of CLEF, and a description of how it has evolved over the years, the major achievements, and what we see as the next challenges.
... We used the Cultural Heritage in CLEF (CHiC) 2012 English collection for ad hoc retrieval. This collection is composed of 1 107 176 documents containing "metadata records describing digital representations of cultural heritage objects" (Petras et al., 2012) and 50 queries for ad hoc retrieval tasks. Below is a With #d and #q being respectively the number of documents in the collection and the number of queries. ...
Article
Full-text available
In this paper, we explore the usage of Word Embedding semantic resources for Information Retrieval (IR) task. This embedding, produced by a shallow neural network, have been shown to catch semantic similarities between words (Mikolov et al., 2013). Hence, our goal is to enhance IR Language Models by addressing the term mismatch problem. To do so, we applied the model presented in the paper Integrating and Evaluating Neural Word Embedding in Information Retrieval by Zuccon et al. (2015) that proposes to estimate the translation probability of a Translation Language Model using the cosine similarity between Word Embedding. The results we obtained so far did not show a statistically significant improvement compared to classical Language Model.
... -metadata quality [12,30,31] -impact of semantic enrichments [38,52,54] and components of workflow [51] -performance of item similarity algorithms [3,4,10,25,26] -content characteristics compared to other DLs [2,14] -usage of particular content [8,9,36,37] -accessibility of content [2] -information retrieval criteria, e.g. precision of search results [1,11,40,41,46] -performance of enrichment tools [32]. ...
Conference Paper
This meta-analysis of 41 evaluation studies of the Europeana Digital Library categorizes them by their constructs, contexts, criteria, and methodologies using Saracevic’s digital library evaluation framework. The analysis shows that system-centered evaluations prevail over user-centered evaluations and evaluations from a societal or institutional perspective are missing. The study reveals, which Europeana components have received focused attention in the last decade (e.g. the metadata) and can serve as a reference for identifying gaps, selecting methodologies and re-using data for future evaluations.
... While there has been a slight increase in the number of long queries, the most prevalent queries are still those of one, two, and three words. The cultural heritage domain is also an example of search systems where users express their information needs using short queries [Akasereh, 2013;Petras et al., 2012]. The third example is the medical images search 1 , where short queries are used to search image captions [Clough & Sanderson, 2004]. ...
... • Besides, we also consider best result of the evaluation campaign. Best MAP achieved ,in CHIC2012 campaign for semantic enrichment task, is 0.34 [Petras et al., 2012]. Table 6.2, shows the baselines and comparison methods for the two smoothing methods used in our experiments: Jelinek-Mercer and Dirichlet. ...
Thesis
Even though modern retrieval systems typically use a multitude of features to rank documents, the backbone for search ranking is usually the standard retrieval models.This thesis addresses a limitation of the standard retrieval models, the term mismatch problem. The term mismatch problem is a long standing problem in information retrieval. However, it was not well understood how often term mismatch happens in retrieval, how important it is for retrieval, or how it affects retrieval performance. This thesis answers the above questions.This research is enabled by the formal definition of term mismatch. In this thesis, term mismatch is defined as the probability that a term does not appear in a document given that this document is relevant. We propose several approaches for reducing term mismatch probability through modifying documents or queries. Our proposals are then followed by a quantitative analysis of term mismatch probability that shows how much the proposed approaches reduce term mismatch probability with maintaining the system performance. An essential component for achieving term mismatch probability reduction is the knowledge resource that defines terms and their relationships.First, we propose a document modification approach according to a user query. The main idea of our document modification approach is to deal with mismatched query terms. While prior research on document enrichment provides a static approach for document modification, we are concerned to only modify the document in case of mismatch. The modified document is then used in a standard retrieval model in order to obtain a mismatch aware retrieval model.Second, we propose a semantic query expansion approach based on a collaborative knowledge resource. We focus on the collaborative resource structure to obtain interesting expansion terms that contribute to reduce term mismatch probability, and as a result, improve the effectiveness of search.Third, we propose a query expansion approach based on neural language models. Neural language models are proposed to learn term vector representations, called distributed neural embeddings. Distributed neural embeddings capture relationships between terms, and they obtained impressive results comparing with state of the art approaches in term similarity tasks. However, in information retrieval, distributed neural embeddings are newly started to be exploited. We propose to use distributed neural embeddings as a knowledge resource in a query expansion scenario.Fourth, we apply the term mismatch probability definition for each contribution of the above contributions. We show how we use standard retrieval corpora with queries and relevance judgments to estimate the term mismatch probability. We estimate the term mismatch probability using original documents and queries, and we figure out how mismatch problem is clearly found in search systems for different types of indexing terms. Then, we point out how much our contributions reduce the estimated mismatch probability, and improve the system recall. As a result, we present how the modified document and query representations contribute to build a mismatch aware retrieval model that mitigate term mismatch problem theoretically and practically.This dissertation shows the effectiveness of our proposals to improve retrieval performance. Our experiments are conducted on corpora from two different domains: medical domain and cultural heritage domain. Moreover, we use two different types of indexing terms for representing documents and queries: words and concepts, and we exploit several types of relationships between indexing terms: hierarchical relationships, relationships based on a collaborative resource structure, relationships defined on distributed neural embeddings.Promising research directions are identified where the term mismatch research may make a significance impact on improving the search scenarios.
... CHiC (2011-2013) promoted systematic and large-scale evaluation of digital libraries and, more in general, cultural heritage information access systems, using the huge Europeana dataset, aggregating information from libraries, museums, and archives [89,212,213]; ...
Article
Full-text available
2014 marks the 15th birthday for CLEF, an evaluation campaign activity which has applied the Cranfield evaluation paradigm to the testing of multilingual and multimodal information access systems in Europe. This paper provides a summary of the motivations which led to the establishment of CLEF, and a description of how it has evolved over the years, the major achievements, and what we see as the next challenges.
... The Europeana information retrieval document collection was prepared for the CHiC pilot lab in 2012 (Petras et al., 2012). It consists of the complete Europeana metadata index as downloaded from the production system in March 2012. ...
... A set of 50 topics was created for the 2013 edition of CHiC, where topic selection was determined partially by the potential for retrieving a sufficient number of relevant documents in each of the collection languages. CHiC 2012 used topics from the Europeana query logs alone, which resulted in zero results for some of the 3 languages [13] . The problem of having zero relevant results is aggravated when collection languages are varied, especially in the cultural heritage area. ...
Conference Paper
Full-text available
The Cultural Heritage in CLEF 2013 lab comprised three tasks: multilingual ad-hoc retrieval and semantic enrichment in 13 languages (Dutch, English, German, Greek, Finnish, French, Hungarian, Italian, Norwegian, Polish, Slovenian, Spanish, and Swedish), Polish ad-hoc retrieval and the interactive task, which studied user behavior via log analysis and questionnaires. For the multilingual and Polish sub-tasks, more than 170,000 documents were assessed for relevance on a tertiary scale. The multilingual task had 7 participants submitting 30 multilingual and 41 monolingual runs. The Polish task comprised 3 participating groups submitting manual and automatic runs. The interactive task had 4 participating research groups and 208 user participants in the study. For the multilingual task, results show that more participants are necessary in order to provide comparative analyses. The interactive task created a rich data set comprising of questionnaire of log data. Further analysis of the data is planned in the future.
... In this paper we describe our experiments done inside the CLEF – CHiC 2013 evaluation campaign [1] focusing both on the multilingual and Polish tasks. Searching for pertinent cultural heritage (CH) objects in response to a short user's query is a challenging task for various reasons. ...
Article
This paper presents and analyzes the experiments done at the University of Neuchatel for both the multilingual and Polish CHiC tasks at CLEF 2013. Within these two tasks, our experiments explore the problem when facing with short text descriptions expressed in various languages having a richer morphology than English. For the multilingual task, each language and its corresponding CH object collection is managed separately. Thus for each query, the broker needs to merge 13 result lists to form a single ranked list of retrieved items. In this context, the best retrieval performance levels tend to be achieved when applying a stopword list for each language. The use of a languagedependent light stemmer may have either a positive or a negative but always slight impact. For the Polish task, we found that the use of a short stopword list and a light stemmer improves retrieval effectiveness. The use of words as indexing units is better than considering n-gram or trunc-n indexing schemes. Considering automatically generated enrichment descriptors does not improve the retrieval effectiveness neither does the use of pseudo-relevance feedback. Finally, the application of data fusion operator was not able to enhance the retrieval performance.
Chapter
INEX ran as an independent evaluation forum for 10 years before it teamed up with CLEF in 2012. Even before 2012 there was considerable collaboration between INEX and CLEF, and these collaborations increased in intensity when CLEF moved beyond its traditional cross-lingual focus in 2009/2010 shifting to include all experimental IR. This led to the merger of CLEF and INEX, and effectively to the inclusion of INEX as a large track or lab into CLEF in 2012. This chapter details the efforts of the INEX lab in CLEF (2012–2014), as well as the ongoing activities as separate labs, under the labels Social Book Search (2015–2016), and Microblog Contextualization (2016–2018).
Article
Europeana is a large-scale search engine for digitised cultural heritage material. It aggregates metadata from various European institutions such as libraries, archives, museums and galleries. The heterogeneous data and the enormous scale (53 million objects in over 50 languages) pose specific challenges for search and exploration. In this paper, we address the different challenges and solutions for information access within Europeana including information needs, data enrichment, ranking and other search aspects for digital cultural heritage.
Article
Full-text available
W niniejszym artykule prezentujemy realizacje laboratorium ewaluacyjnego CLEF (Conference and Labs of the Evaluation Forum) ze specjalnym uwzglednieniem kampanii CHiC (Cultural Heritage in CLEF). Opisujemy realizacje oraz wyniki zadania Polish Task in ChiC. W artykule zaprezentowano wnioski z realzacji zadania. Zostaly omowione wyniki uzyskane przez uczestnikow zadania przy uzyciu roznych strategii indeksowania oraz wyszukiwania zasobow. Porownaliśmy efektywnośc metod tf-idf, OKAPI, DFR oraz data fusion.