Conference Paper

Multiple Example Queries in Content-Based Image Retrieval

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Content-Based Image Retrieval (cbir) is the practical class of techniques used for information retrieval from large image collections. Many CBIR systems allow users to specify their information need by providing an example image. This query-by-example paradigm can be extended to support multiple example images. In this work, we present a large-scale experiment that shows the average performance of querying with multiple examples is significantly better than single-example querying. We also investigate the effects of providing different numbers of example images, the impact of example quality, and the relative performance of functions used to combine image features. Our experiments indicate that three-example queries are more effective than other numbers of examples, and that the minimum combining function is robust for most query types.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... In both papers, they evaluated their approach using 90 versions of 30 different songs. The fourth method performs the best, with a precision value of 54.5% and a recall of 30.8% recall. 1 In the image IR field, frequency distributions (the term histograms are often used instead) are also often used as summary values to represent images [8,41,142,158,159]. Colour is the primary feature used for image matching. ...
... Tahaghoghi et al. [142] experimented with frequency distributions of colour components in various colour spaces. 2 The total frequency of every colour component is normalised to 1. For matching images based on their content using histograms, they found the Manhattan distance to be more accurate than other distance measures. ...
... The paper also examined combination functions from Fox and Shaw [43], some of which are COMBMIN, COMBMAX, and COMBSUM. They are essentially the same as the MINIMUM, MAXIMUM, and SUM methods in Tahaghoghi et al. [142], respectively. ...
... To detect cuts, we rank the frames of the moving query window based on their similarity to the current frame [27]. The frame most similar to the current frame is ranked highest.Figure 2 shows how a cut can be detected using similarity ranking. ...
... Our algorithm applies the concept of query-by-example (qbe), popular in content-based image retrieval [27], to shot boundary detection. The observation that all transitions except cuts stretch over several adjacent frames suggests that we ought to evaluate a set of frames together. ...
... To detect cuts, we rank the frames of the moving query window based on their similarity to the current frame [27]. The frame most similar to the current frame is ranked highest. ...
Article
Full-text available
Digital video is widely used in multimedia databases and requires effective retrieval techniques. Shot bound-ary detection is a common first step in analysing video content. The effective detection of gradual transitions is an especially difficult task. Building upon our past research work, we have designed a novel decision stage for detection of gradual transitions. Its strength lies particularly in the accurate detection of gradual tran-sition boundaries. In this paper, we describe our mov-ing query window method and discuss its performance in the context of the trec-12 shot boundary detection task. We believe this approach is a valuable contribu-tion to video retrieval and worth persueing in the fu-ture.
... The GNU Image Finding Tool (GIFT) 5 supports single-and multiple-image QBIE (Squire et al. 2000). Tahaghoghi et al. (2001, 2002) have shown that using multiple query images generally improves retrieval ef- fectiveness. ...
... In this paper, the results that we present use Maximum similarity as the combining function (which is the minimum distance). This was found to be the most effective combining function by Tahaghoghi et al. (2002). To reduce the problem of object segmentation, we test our approach on a domain where regions are easily separated: a collection of comic strips. ...
... In this work we focus on combiningSquire et al. 2000). Tahaghoghi et al. (2001 Tahaghoghi et al. ( , 2002) have shown that using multiple query images generally improves retrieval ef- fectiveness. The need to describe shapes mathematically leads to the two broad methods for shape representation and description — region-and contour-based (Zhang & Lu 2004). ...
Conference Paper
Full-text available
A common approach to content-based image retrieval is to use example images as queries; images in the collection that have low-level features similar to the query examples are returned in response to the query. In this paper, we explore the use of image regions as query examples. We compare the retrieval eec- tiveness of using whole images, single regions, and multiple regions as examples. We also compare two approaches for combining shape features: an equal- weight linear combination, and classication using machine learning algorithms. We show that using im- age regions as query examples leads to higher eec- tiveness than using whole images, and that an equal- weight linear combination of shape features is simpler and at least as eective as using a machine learning algorithm.
Article
Multi-query is a powerful approach in the context of Content-Based Image Retrieval (CBIR) that is designed to optimize retrieval performance. Notably, multiquery contains several query images selected by the user and each image has a visual representation with respect to the user. Each image can often be represented as a high dimensional vector of the low-level features. The question remains as to how a system can integrate those vectors for multi-query processing; in this work, we will explore several active-dimension strategies for multi-query image retrieval. The active-dimension is selected according to both the mean and coefficient of variation of the feature histograms for the multi-query process. The experimental results show that the similarity measurement design, based on active-dimension strategies, shows superior retrieval performance with respect to consumer images.
Conference Paper
This paper presents a cluster-based relevance feedback technique. Our method combines two popular techniques of relevance feedback: query point movement and query expansion. The query expansion gives better result than the query point movement but there is still a problem with the insufficiency and the overlap of relevant and irrelevant images. To enhance the efficiency in such cases, irrelevant images are used to modify the multiple point queries of the query expansion technique by using the query point movement technique. To learn the multiple point queries, the irrelevant images are classified into query points which are clustered from relevant images in the query expansion technique. Inspired from text retrieval, query point movement and query expansion has given good result for image retrieval. In this paper, another technique inspired from text retrieval - the state-of-the-art Bag of Words model is used. The experiments show that our method gives better results in comparison with traditional methods of relevance feedback.
Article
This paper presents a novel probabilistic framework to process multiple sample queries in content based image retrieval (CBIR). This framework is independent from the underlying distance or (dis)similarity measures which support the retrieval system, and only assumes mutual independence among their outcomes.The proposed framework gives rise to a relevance feedback mechanism in which positive and negative data are combined in order to optimally retrieve images according to the available information. A particular setting in which users interactively supply feedback and iteratively retrieve images is set both to model the system and to perform some objective performance measures.Several repositories using different image descriptors and corresponding similarity measures have been considered for benchmarking purposes. The results have been compared to those obtained with other representative strategies, suggesting that a significant improvement in performance can be obtained.
Conference Paper
Full-text available
Benchmarking has always been a crucial problem in content-based image retrieval (CBIR).A key issue is the lack of a common access method to retrieval systems, such as SQL for relational databases. The Multimedia Retrieval Mark-up Language (MRML) solves this problem by standardizing access to CBIR systems (CBIRSs). Other difficult problems are also shortly addressed, such as obtaining relevance judgments and choosing a database for performance comparison. In this article we present a fully automated benchmark for CBIRSs based on MRML, which can be adapted to any image database and almost any kind of relevance judgment. The test evaluates the performance of positive and negative relevance feedback, which can be generated automatically from the relevance judgments. To illustrate our purpose, a freely available, non-copyright image collection is used to evaluate our CBIRS, Viper. All scripts described here are also freely available for download.
Article
Full-text available
Content-based image retrieval (CBIR) algorithms have been seen as a promising access method for digital photograph collections. Unfortunately, we have very little evidence of the usefulness of these algorithms in real user needs and contexts. In this paper, we introduce a test collection for the evaluation of CBIR algorithms. In the test collection, the performance testing is based on photograph similarity perceived by end-users in the context of realistic illustration tasks and environment. The building process and the characteristics of the resulting test collection are outlined, including a typology of similarity criteria expressed by the subjects judging the similarity of photographs. A small-scale study on the consistency of similarity assessments is presented. A case evaluation of two CBIR algorithms is reported. The results show clear correlation between the subjects' similarity assessments and the functioning of feature parameters of the tested algorithms.
Conference Paper
Full-text available
A major hurdle in practical content based image retrieval (CBIR) is conveying the user's information need to the system. One common method of query specification is to express the query using one or more example images. The authors consider whether using more examples improves the effectiveness of CBIR in meeting a user's information need. We show that using multiple examples improves retrieval effectiveness by around 9%-20% over single-example queries, but that further improvements in using more than two examples may not justify the added processing required
Conference Paper
Full-text available
Content based retrieval systems have been developed for querying image data in which the users can pose queries based on visual properties such as color and texture. These systems, which have advanced the state of the art in image database systems, remarkably lack formal query languages. The traditional query languages are unable to capture the inherent fuzzy nature of the image data and content based querying. The fuzzy object query language (FOQL) presented in this paper addresses the need to support fuzzy values and fuzzy collections required for image databases. It can be used for defining schemas and high level concepts, and for querying image databases. It captures the inherent fuzzy nature of content based retrieval by keeping the query results fuzzy as against other query languages. The users can interactively refine their queries and high level concept definitions using recursive and named query definition constructs in FOQL. Being an extension of ODMG-OQL, FOQL can be easily mapped to ODMG-compliant visual query languages
Conference Paper
Full-text available
Retrieving images from large and varied collections using image content as a key is a challenging and important problem. In this paper, we present a new image representation which provides a transformation from the raw pixel data to a small set of localized coherent regions in color and texture space. This so-called “blobworld” representation is based on segmentation using the expectation-maximization algorithm on combined color and texture features. The texture features we use for the segmentation arise from a new approach to texture description and scale selection. We describe a system that uses the blobworld representation to retrieve images. An important and unique aspect of the system is that, in the context of similarity-based querying, the user is allowed to view the internal representation of the submitted image and the query results. Similar systems do not offer the user this view into the workings of the system; consequently, the outcome of many queries on these systems can be quite inexplicable, despite the availability of knobs for adjusting the similarity metric
Article
Full-text available
Research on ways to extend and improve query methods for image databases is widespread. We have developed the QBIC (Query by Image Content) system to explore content-based retrieval methods. QBIC allows queries on large image and video databases based on example images, user-constructed sketches and drawings, selected color and texture patterns, camera and object motion, and other graphical information. Two key properties of QBIC are (1) its use of image and video content-computable properties of color, texture, shape and motion of images, videos and their objects-in the queries, and (2) its graphical query language, in which queries are posed by drawing, selecting and other graphical means. This article describes the QBIC system and demonstrates its query capabilities. QBIC technology is part of several IBM products
Article
Full-text available
We describe two new color indexing techniques. The first one is a more robust version of the commonly used color histogram indexing. In the index we store the cumulative color histograms. The L 1 -, L 2 -, or L1 -distance between two cumulative color histograms can be used to define a similarity measure of these two color distributions. We show that while this method produces only slightly better results than color histogram methods, it is more robust with respect to the quantization parameter of the histograms. The second technique is an example of a new approach to color indexing. Instead of storing the complete color distributions, the index contains only their dominant features. We implement this approach by storing the first three moments of each color channel of an image in the index, i.e., for a HSV image we store only 9 floating point numbers per image. The similarity function which is used for the retrieval is a weighted sum of the absolute differences between corresponding mo...
Article
Full-text available
In this paper, we address an important yet controversial issue in content-based image retrieval (CBIR): performance evaluation (PE). The goal is to develop an objective and comprehensive protocol for quantitative evaluation and comparison of CBIR performances among different approaches. The protocol consists of two elements: acquisition of appropriate ground truth data, and definition of quantitative performance metrics. A large image data set with human-label ground truths has been built as part of the protocol and made available for public use. An application of this protocol to our MiAlbum system is also presented, which shows suitability of such a protocol.
Chapter
Content Based Image Retrieval (CBIR) systems that are able to “retrieve images of Clinton with Lewinsky” are unrealistic at present. However, this area has seen much research and development activity since IBM’s QBIC announcement in 1994. The CHITRA CBIR system under development at the RMIT and Monash Universities, addresses the need for a test bed system. Users can dynamically incorporate new features and similarity measures in to the system, enabling it to act as a testbed for CBIR research. The system uses a 4-level data model we have developed and supports definition and querying of high level concepts such as MOUNTAIN and SUNSET. These advanced capabilities are supported by a powerful graphical query mechanism and a high-dimensional indexing structure based on linear mapping. In this paper we describe the design of the system, our contributions to the state of the art and provide some implementation details.
Article
In studying actual Web searching by the public at large, we analyzed over one million Web queries by users of the Excite search engine. We found that most people use few search terms, few modified queries, view few Web pages, and rarely use advanced search features. A small number of search terms are used with high frequency, and a great many terms are unique; the language of Web queries is distinctive. Queries about recreation and entertainment rank highest. Findings are compared to data from two other large studies of Web queries. This study provides an insight into the public practices and choices in Web searching.
Article
A common form of queries encountered in Content-Based Image Retrieval (CBIR) systems, such as QBIC and Virage, are query by example image (QBE). We encountered this problem in our implementation of a prototype CBIR system, called CHITRA. This system supports four layer image data model and enable high level concept definition such as SUNSET. Users can pose queries of the form retrieve all images that have SUNSET and MOUNTAINS. We are addressing the problem of processing queries of the form retrieve all images similar to I1, I2,..., In based on color. We refer to such queries as SF-QBME. Essentially the same problem is encountered in processing high level concept queries such as SUNSET and MOUNTAINS above. Processing of such queries has received some recent research attention. Processing SF-QBME queries involves dealing with multiple points in a single feature space. We first provide the motivation for use of such queries in the context of similarity based retrieval. We then define the exact low level semantics of such queries, and provide the corresponding processing strategies. The experimental performance results that demonstrate the capability of SF-QBME are also presented.
Article
In studying actual Web searching by the public at large, we analyzed over one million Web queries by users of the Excite search engine. We found that most people use few search terms, few modified queries, view few Web pages, and rarely use advanced search features. A small number of search terms are used with high frequency, and a great many terms are unique; the language of Web queries is distinctive. Queries about recreation and entertainment rank highest. Findings are compared to data from two other large studies of Web queries. This study provides an insight into the public practices and choices in Web searching.
Article
In the QBIC (Query By Image Content) project we are studying methods to query large on-line image databases using the images'' content as the basis of the queries. Examples of the content we use include color, texture, shape, position, and dominant edges of image objects and regions. Potential applications include medical (Give me other images that contain a tumor with a texture like this one), photo-journalism (Give me images that have blue at the top and red at the bottom), and many others in art, fashion, cataloging, retailing, and industry. We describe a set of novel features and similarity measures allowing query by image content, together with the QBIC system we implemented. We demonstrate the effectiveness of our system with normalized precision and recall experiments on test databases containing over 1000 images and 1000 objects populated from commercially available photo clip art images, and of images of airplane silhouettes. We also present new methods for efficient processing of QBIC queries that consist of filtering and indexing steps. We specifically address two problems: (a) non Euclidean distance measures; and (b) the high dimensionality of feature vectors. For the first problem, we introduce a new theorem that makes efficient filtering possible by bounding the non-Euclidean, full cross-term quadratic distance expression with a simple Euclidean distance. For the second, we illustrate how orthogonal transforms, such as Karhunen Loeve, can help reduce the dimensionality of the search space. Our methods are general and allow some false hits but no false dismissals. The resulting QBIC system offers effective retrieval using image content, and for large image databases significant speedup over straightforward indexing alternatives. The system is implemented in X/Motif and C running on an RS/6000.
Article
This paper provides a comprehensive survey of the technical achievements in the research area of image retrieval, especially content-based image retrieval, an area that has been so active and prosperous in the past few years. The survey includes 100+ papers covering the research aspects of image feature representation and extraction, multidimensional indexing, and system design, three of the fundamental bases of content-based image retrieval. Furthermore, based on the state-of-the-art technology available now and the demand from real-world applications, open research issues are identified and future promising research directions are suggested.
Conference Paper
Unlike traditional database management systems, in multi- media databases that support content-based retrieval over multimedia objects, it is difficult for users to express their exact information need directly in the form of a precise query. Atypical interface supported by content-based retrieval systems allows users to express their query in the form of examples of objects similar to the ones they wish to retrieve. Such a user interface, however, requires mechanisms to learn the query representation from the examples provided by the user. In our previous work, we proposed a query refinement mechanism in which a query rep- resentation is modified by adding new relevant examples based on user feedback. In this paper, we describe query processing mechanisms that can efficiently support query expansion using multidimensional index structures.
Article
Efficient dynamic programming algorithms are available for a broad class of protein and DNA sequence comparison problems. These algorithms require computer time proportional to the product of the lengths of the two sequences being compared [O(N2)] but require memory space proportional only to the sum of these lengths [O(N)]. Although the requirement for O(N2) time limits use of the algorithms to the largest computers when searching protein and DNA sequence databases, many other applications of these algorithms, such as calculation of distances for evolutionary trees and comparison of a new sequence to a library of sequence profiles, are well within the capabilities of desktop computers. In particular, the results of library searches with rapid searching programs, such as FASTA or BLAST, should be confirmed by performing a rigorous optimal alignment. Whereas rapid methods do not overlook significant sequence similarities, FASTA limits the number of gaps that can be inserted into an alignment, so that a rigorous alignment may extend the alignment substantially in some cases. BLAST does not allow gaps in the local regions that it reports; a calculation that allows gaps is very likely to extend the alignment substantially. Although a Monte Carlo evaluation of the statistical significance of a similarity score with a rigorous algorithm is much slower than the heuristic approach used by the RDF2 program, the dynamic programming approach should take less than 1 hr on a 386-based PC or desktop Unix workstation. For descriptive purposes, we have limited our discussion to methods for calculating similarity scores and distances that use gap penalties of the form g = rk. Nevertheless, programs for the more general case (g = q+rk) are readily available. Versions of these programs that run either on Unix workstations, IBM-PC class computers, or the Macintosh can be obtained from either of the authors.
Conference Paper
With the proliferation of multimedia, the World Wide Web and digital imaging, there now exists a high demand for intelligent tools for image management; most importantly, indexing and retrieval-commonly referred to as “query-by-image-content”. Existing systems often make use of global attributes, such as overall color distributions, which ignore the actual composition of the image in terms of internal structures. In this paper, we present an experimental image retrieval system designed and based on the principle that it is the user who is most qualified to specify the “content” in an image, and not the computer. Consequently, the user is asked to provide salient “regions-of-interest” (ROIs) and to specify the importance of their relative spatial relationships in the query image. Our technique has led to acceptable retrievals (equal to if not better than global-based searches), and it provides an intuitive user interface and more flexibility in specifying image content
Conference Paper
We present an implementation of NeTra, a prototype image retrieval system that uses color texture, shape and spatial location information in segmented image database. A distinguishing aspect of this system is its incorporation of a robust automated image segmentation algorithm that allows object or region based search. Image segmentation significantly improves the quality of image retrieval when images contain multiple complex objects. Other important components of the system include an efficient color representation, and indexing of color, texture, and shape features for fast search and retrieval. This representation allows the user to compose interesting queries such as “retrieve all images that contain regions that have the color of object A, texture of object B, shape of object C, and lie in the upper one-third of the image” where the individual objects could be regions belonging to different images
Article
Selecting from a large, expanding collection of images requires carefully chosen search criteria. We present an approach that integrates a relational database retrieval system with a color analysis technique. The Chabot project was initiated at our university to study storage and retrieval of a vast collection of digitized images. These images are from the State of California Department of Water Resources. The goal was to integrate a relational database retrieval system with content analysis techniques that would give our querying system a better method for handling images. Our simple color analysis method, if used in conjunction with other search criteria, improves our ability to retrieve images efficiently. The best result is obtained when text-based search criteria are combined with content-based criteria and when a coarse granularity is used for content analysis
Article
This paper provides a comprehensive survey of the technical achievements in the research area of Image Retrieval, especially Content-Based Image Retrieval, an area so active and prosperous in the past few years. The survey includes 100+ papers covering the research aspects of image feature representation and extraction, multi-dimensional indexing, and system design, three of the fundamental bases of Content-Based Image Retrieval. Furthermore, based on the state-of-the-art technology available now and the demand from real-world applications, open research issues are identified, and future promising research directions are suggested.
Article
The growth of digital image and video archives is increasing the need for tools that effectively filter and efficiently search through large amounts of visual data. Towards this goal we propose a technique by which the color content of images and videos is automatically extracted to form a class of meta-data that is easily indexed. The color indexing algorithm uses the back-projection of binary color sets to extract color regions from images. This technique provides for both the automated extraction of regions and representation of their color content. It overcomes some of the problems with color histogram techniques such as high-dimensional feature vectors, spatial localization, indexing and distance computation. We present the binary color set back-projection technique and discuss its implementation in the VisualSEEk content-based image/video retrieval system for the World Wide Web. We also evaluate the retrieval effectiveness of the color set back-projection method and compare its performance to other color retrieveal methods.
Article
Users often can not easily express their queries. For example, in a multimedia#image by content setting, the user mightwant photographs with sunsets; in current systems, like QBIC, the user has to give a sample query, and to specify the relative importance of color, shape and texture. Even worse, the user mightwant correlations between attributes, like, for example, in a traditional, medical record database, a medical researcher mightwant to #nd #mildly overweight patients", where the implied query would be #weight#height # 4 lb#inch". Our goal is to provide a user-friendly, but theoretically solid method, to handle such queries. We allow the user to give several examples, and, optionally, their 'goodness' scores, and we propose a novel method to #guess" which attributes are important, which correlations are important, and with what weight. Our contributions are twofold: #a# we formalize the problem as a minimization problem and showhow to solve for the optimal solution, completely av...
Article
We present a method for searching in an image database using a query image that is similar to the intended target. The query image may be a hand-drawn sketch or a (potentially low-quality) scan of the image to be retrieved. Our searching algorithm makes use of multiresolution wavelet decompositions of the query and database images. The coefficients of these decompositions are distilled into small "signatures" for each image. We introduce an "image querying metric" that operates on these signatures. This metric essentially compares how many significant wavelet coefficients the query has in common with potential targets. The metric includes parameters that can be tuned, using a statistical analysis, to accommodate the kinds of image distortions found in different types of image queries. The resulting algorithm is simple, requires very little storage overhead for the database of signatures, and is fast enough to be performed on a database of 20,000 images at interactive rates (on standard...
Article
1 We describe a highly functional prototype system for searching by visual features in an image database. The VisualSEEk system is novel in that the user forms the queries by diagramming spatial arrangements of color regions. The system finds the images that contain the most similar arrangements of similar regions. Prior to the queries, the system automatically extracts and indexes salient color regions from the images. By utilizing efficient indexing techniques for color information, region sizes and absolute and relative spatial locations, a wide variety of complex joint color/spatial queries may be computed. KEYWORDS: image databases, content-based retrieval, image indexing, similarity retrieval, spatial query. 1 INTRODUCTION In this paper we investigate a new content-based image query system that enables querying by image regions and spatial layout. VisualSEEk is a hybrid system in that it integrates feature-based image indexing with spatial query methods. The integration reli...
Article
Image Database Management Systems (IDBMS) aim to store large collections of images, and to support efficient content-based retrieval. In this paper we explore the idea of image data modeling as a tool for describing image domains, with the twofold objectives of guiding feature extraction and incorporating semantics into the extracted summaries of images. We discuss the implementation of these ideas in the PIQ image DBMS, and demonstrate that significat gains in the expressiveness of content-based queries can be achieved. Image Database Management Systems (IDBMS) aim to store large collections of images, and to support efficient content-based retrieval of the images. Like any other DBMS, an IDBMS has a storage facility, the data is arranged according to some data model, and it has a query facility. The main IDBMS issue this paper explores is the data model and its impact on the expressiveness of the query language. The kind of queries we focus on are selection queries. In other words...
Colour features in content-based image retrieval
  • S M M Tahaghoghi
  • J A Thom
  • H E Williams