Mingjing Li’s research while affiliated with University of Science and Technology of China and other places

Statistical approaches to large-scale image annotation are described. Generally, the annotation technique includes compiling visual features and textual information from a number of images, hashing the images visual features, and clustering the images based on their hash values. An example system builds statistical language models from the clustered images and annotates the image by applying one of the statistical language models.

A dual cross-media relevance model (DCMRM) is used for automatic image annotation. In contrast to the traditional relevance models which calculate the joint probability of words and images over a training image database, the DCMRM model estimates the joint probability by calculating the expectation over words in a predefined lexicon. The DCMRM model may be advantageous because a predefined lexicon potentially has better behavior than a training image database. The DCMRM model also takes advantage of content-based techniques and image search techniques to define the word-to-image and word-to-word relations involved in image annotation. Both relations can be estimated by using image search techniques on the web data as well as available training data.

Improvements are provided to effectively assess a user's face and head pose such that a computer or like device can track the user's attention towards a display device(s). Then the region of the display or graphical user interface that the user is turned towards can be automatically selected without requiring the user to provide further inputs. A frontal face detector is applied to detect the user's frontal face and then key facial points such as left/right eye center, left/right mouth corner, nose tip, etc., are detected by component detectors. The system then tracks the user's head by an image tracker and determines yaw, tilt and roll angle and other pose information of the user's head through a coarse to fine process according to key facial points and/or confidence outputs by pose estimator.

Word correlations are estimated using a content-based method, which uses visual features of image representations of the words. The image representations of the subject words may be generated by retrieving images from data sources (such as the Internet) using image search with the subject words as query words. One aspect of the techniques is based on calculating the visual distance or visual similarity between the sets of retrieved images corresponding to each query word. The other is based on calculating the visual consistence among the set of the retrieved images corresponding to a conjunctive query word. The combination of the content-based method and a text-based method may produce even better result.

This paper presents a latent topic visual language model to handle variation problem in object categorization. Variations including different views, styles, poses, etc., have greatly affected the spatial arrangement and distribution of visual features, on which previous categorization models largely depend. Taking the object variations as hidden topics within each category, the proposed model explores the relationship between object variations and visual feature arrangement in the traditional visual language modeling process. With this improvement, the accuracy of object categorization is further boosted. Experiments on Caltech101 dataset have shown that this model makes sense and is effective.

Automatic annotation of digital pictures is a key technology for managing and retrieving images from large image collections. Typical algorithms only deal with the problem of monolingual image annotation. In this paper, we propose a framework to deal with the problem of multilingual image annotation, which can annotate images in multiple languages. The framework can not only benefit users with different native languages, but also provide more accurate annotations. In this framework, image annotation is performed in two stages, including parallel monolingual image annotation and the fusion of annotation results in multiple languages. In the first stage, candidate annotations for each language are extracted by leveraging multilingual large scale Web image database. Due to the incompleteness and inaccuracy problem of candidate annotations, we proposed a multilingual annotation fusion algorithm (MAF). By modeling candidate annotations for each language as an n-partite graph, MAF algorithm can improve and re-rank multilingual annotations. Finally, annotations with the highest ranking values in each language are selected and translated as the result. Experimental results for English-Chinese image annotations demonstrate the effectiveness of the proposed framework.

Modern web search engines are expected to return the top-k results efficiently. Although many dynamic index pruning strategies have been proposed for efficient top-k computation, most of them are prone to ignoring some especially important factors in ranking functions, such as term-proximity (the distance relationship between query terms in a document). In our recent work [Zhu, M., Shi, S., Li, M., & Wen, J. (2007). Effective top-k computation in retrieving structured documents with term-proximity support. In Proceedings of 16th CIKM conference (pp. 771–780)], we demonstrated that, when term-proximity is incorporated into ranking functions, most existing index structures and top-k strategies become quite inefficient. To solve this problem, we built the inverted index based on web page structure and proposed the query processing strategies accordingly. The experimental results indicate that the proposed index structures and query processing strategies significantly improve the top-k efficiency. In this paper, we study the possibility of adopting additional techniques to further improve top-k computation efficiency. We propose a Proximity-Probe Heuristic to make our top-k algorithms more efficient. We also test the efficiency of our approaches on various settings (linear or non-linear ranking functions, exact or approximate top-k processing, etc.).

Image annotation has been an active research topic in recent years due to its potential impact on both image understanding and web image search. In this paper, we propose a graph learning framework for image annotation. First, the image-based graph learning is performed to obtain the candidate annotations for each image. In order to capture the complex distribution of image data, we propose a Nearest Spanning Chain (NSC) method to construct the image-based graph, whose edge-weights are derived from the chain-wise statistical information instead of the traditional pairwise similarities. Second, the word-based graph learning is developed to refine the relationships between images and words to get final annotations for each image. To enrich the representation of the word-based graph, we design two types of word correlations based on web search results besides the word co-occurrence in the training set. The effectiveness of the proposed solution is demonstrated from the experiments on the Corel dataset and a web image dataset.

In recent years, ldquobag-of-wordsrdquo models, which treat an image as a collection of unordered visual words, have been widely applied in the multimedia and computer vision fields. However, their ignorance of the spatial structure among visual words makes them indiscriminative for objects with similar word frequencies but different word spatial distributions. In this paper, we propose a visual language modeling method (VLM), which incorporates the spatial context of the local appearance features into the statistical language model. To represent the object categories, models with different orders of statistical dependencies have been exploited. In addition, the multilayer extension to the VLM makes it more resistant to scale variations of objects. The model is effective and applicable to large scale image categorization. We train scale invariant visual language models based on the images which are grouped by Flickr tags, and use these models for object categorization. Experimental results show they achieve better performance than single layer visual language models and ldquobag-of-wordsrdquo models. They also achieve comparable performance with 2-D MHMM and SVM-based methods, while costing much less computational time.

In image annotation refinement, word correlations among candidate annotations are used to reserve high relevant words and remove irrelevant words. Existing methods build word correlations on textual annotations of images. In this paper, visual contents of images are utilized to explore better word correlations by using multi-graph similarity reinforcement method. Firstly, image visual similarity graph and word correlations graph are built respectively. Secondly, the two graphs are iteratively reinforced by each other through image-word transfer matrix. Once the two graphs converge to steady states, the new word correlations graph is used to refine the candidate annotations. The experiments show that our method performs better than method not considering visual content of images.