No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Towards content-based patent image retrieval: A framework perspective
Abstract
In this article, we discuss the potential benefits, the requirements and the challenges involved in patent image retrieval and subsequently, we propose a framework that encompasses advanced image analysis and indexing techniques to address the need for content-based patent image search and retrieval. The proposed framework involves the application of document image pre-processing, image feature and textual metadata extraction in order to support effectively content-based image retrieval in the patent domain. To evaluate the capabilities of our proposal, we implemented a patent image search engine. Results based on a series of interaction modes, comparison with existing systems and a quantitative evaluation of our engine provide evidence that image processing and indexing technologies are currently sufficiently mature to be integrated in real-world patent retrieval applications.
... Some of the literature on design practices encourages collecting and seeking visual examples to inform the design process [18][19][20][21][22][23]. Some prior works have mined and analyzed patent images, but they only use geometric and local features to represent images without considering technology-related knowledge [24][25][26][27][28]. ...
... Also, the database used in this system is quite small. Vrochidis et al. [25] created the adaptive hierarchical density histogram (AHDH) to exploit the content of patent images. This method has been extended by Sidiropoulous et al. [65], who introduced the quantized relative density information. ...
... Specifically, we add an auxiliary task to classify each patent image into a specific type group, which can also be viewed as a multi-class classification task. Vrochidis et al. claimed that there are nine types for patent images, including drawing, flowchart, graph, chemical structure, table, DNA, math formula, computer program, and symbol [25]. The purpose of the auxiliary task is to embed type-related knowledge into design features. ...
The patent database is often used by designers to search for inspirational stimuli for innovative design opportunities because of the large size, extensive variety and the massive quantity of design information contained in patent documents. Growing work on design-by-analogy has adopted various vectorization approaches for associating design documents. However, they only focused on text analysis and ignored visual information. Research in engineering design and cognitive psychology has shown that visual stimuli may benefit design-by-analogy. In this study, we focus on visual design stimuli and automatically derive the vector space and the design feature vectors representing design images. The automatic vectorization approach uses a novel convolutional neural network architecture named Dual-VGG aiming to accomplish two tasks: visual material type prediction and international patent classification (IPC) section-label predictions. The derived feature vectors that embed both visual characteristics and technology-related knowledge can be potentially utilized to guide the retrieval and use of near-field and far-field design stimuli according to their vector distances. We report the accuracy of the training tasks and also use a case study to demonstrate the advantages of design image retrievals based on our model.
... Some prior works have mined and analyzed patent images [5][6][7][8][9]. Although it can provide many potential benefits for datadriven design, patent image retrieval faces some challenges. ...
... It is trained with two different tasks: visual material type prediction and IPC class label prediction. Specifically, visual material type prediction is to classify patent images into 9 pre-defined type categories [6], namely abstract drawing, flowchart, graph, chemical structure, table, DNA, math formula, computer program and symbol. Thus, the first task enables our model to learn visual characteristics of images. ...
... And the database used in this system is quite small. Vrochidis et al. [6] created the adaptive hierarchical density histogram (AHDH) to exploit the content of patent images. This method has been extended by Sidiropoulous et al. [37], who introduced the quantized relative density information. ...
The patent database is often used in searches of inspirational stimuli for innovative design opportunities because of its large size, extensive variety and rich design information in patent documents. However, most patent mining research only focuses on textual information and ignores visual information. Herein, we propose a convolutional neural network (CNN)-based patent image retrieval method. The core of this approach is a novel neural network architecture named Dual-VGG that is aimed to accomplish two tasks: visual material type prediction and international patent classification (IPC) class label prediction. In turn, the trained neural network provides the deep features in the image embedding vectors that can be utilized for patent image retrieval and visual mapping. The accuracy of both training tasks and patent image embedding space are evaluated to show the performance of our model. This approach is also illustrated in a case study of robot arm design retrieval. Compared to traditional keyword-based searching and Google image searching, the proposed method discovers more useful visual information for engineering design.
... Some prior works have mined and analyzed patent images [5][6][7][8][9]. Although it can provide many potential benefits for datadriven design, patent image retrieval faces some challenges. ...
... It is trained with two different tasks: visual material type prediction and IPC class label prediction. Specifically, visual material type prediction is to classify patent images into 9 pre-defined type categories [6], namely abstract drawing, flowchart, graph, chemical structure, table, DNA, math formula, computer program and symbol. Thus, the first task enables our model to learn visual characteristics of images. ...
... And the database used in this system is quite small. Vrochidis et al. [6] created the adaptive hierarchical density histogram (AHDH) to exploit the content of patent images. This method has been extended by Sidiropoulous et al. [37], who introduced the quantized relative density information. ...
The patent database is often used in searches of inspirational stimuli for innovative design opportunities because of its large size, extensive variety and rich design information in patent documents. However, most patent mining research only focuses on textual information and ignores visual information. Herein, we propose a convolutional neural network (CNN)- based patent image retrieval method. The core of this approach is a novel neural network architecture named Dual-VGG that is aimed to accomplish two tasks: visual material type prediction and International Patent Classification (IPC) class label prediction. In turn, the trained neural network provides the deep features in the image embedding vectors that can be utilized for patent image retrieval. The accuracy of both training tasks and patent image embedding space are evaluated to show the performance of our model. This approach is also illustrated in a case study of robot arm design retrieval. Compared to traditional keyword-based searching and Google image searching, the proposed method discovers more useful visual information for engineering design.
... Targeting patent image retrieval, a few attempts [11,22,46,48,52] have been made exploiting the contents of patent drawings in different ways. Besides its potential benefits, the development of a patent image retrieval system poses some research challenges. ...
... -We further investigate the potential of the contextual information by proposing a novel spatial area weighting scheme, which is designed to weight the local primitives and contextual local primitives histograms of the binary patent images. -For the evaluation, we perform similar image retrieval experiments using the Binary Patent Image Database [48] (BPID) containing 2000 images and another database introduced by the CLEF-IP evaluation campaign in 2011 containing 38,087 images [20] in Section 4. Both of these databases are described in Section 4.1. ...
... To exploit the content of patent images, Vrochidis et al. [48] proposed the adaptive hierarchical density histogram (AHDH) approach which has been extended by Sidiropoulos et al. [43] by introducing a quantized relative density feature. Both approaches calculate the distribution of black pixels on the white image plane in a patent image by performing image partitioning. ...
Local features and descriptors that perform well in the case of photographic images are often unable to capture the content of binary technical drawings due to their different characteristics. Motivated by this, a new local feature representation, the contextual local primitives, is proposed in this paper. It is based on the detection of the junction and end points, classification of the local primitives to local primitive words and establishment of the geodesic connections of the local primitives. We exploit the granulometric information of the binary patent images to set all the necessary parameters of the involved mathematical morphology operators and window size for the local primitive extraction, which makes the whole framework parameter free. The contextual local primitives and, their spatial areas as a histogram weighting factor are evaluated by performing binary patent image retrieval experiments. It is found that the proposed contextual local primitives perform better than the local primitives only, the SIFT description of the contextual Hessian points, the SIFT description of local primitives and state of the art local content capturing methods. Moreover, an analysis of the approach in the perspective of a general patent image retrieval system reveals of its being efficient in multiple aspects.
... [3], [4]), as well as with prototype systems and demos (e.g. [5], [6]). In addition, dedicated sessions and talks have been organised in relevant symposiums, workshops and conferences (e.g. ...
... Image-based patent search also allows for retrieving patents ranging in time, which could have been written using different terminologies. Motivated by the above, recent works in patent search [5] are directed towards the development of systems, which could automatically extract and retrieve patent images based on visual similarity ...
... More recently, the PatMedia image search engine was developed during the PATExpert project [19]. PatMedia is capable of retrieving patent images based on visual similarity using the Adaptive Hierarchical Density Histograms (AHDH) [20] and constitutes the retrieval engine of an integrated patent image extraction and retrieval framework [5]. ...
Nowadays most of the patent search systems still rely upon text to provide retrieval functionalities. Recently, the intellectual property and information retrieval communities have shown great interest in patent image retrieval, which could augment the current practices of patent search. In this chapter, we present a patent image extraction and retrieval framework, which deals with patent image extraction and multimodal (textual and visual) metadata generation from patent images with a view to provide content-based search and concept-based retrieval functionalities. Patent image extraction builds upon page orientation detection and segmentation, while metadata extraction from images is based on the generation of low level visual and textual features. The content-based retrieval functionality is based on visual low level features, which have been devised to deal with complex black and white drawings. Extraction of concepts builds upon on a supervised machine learning framework realised with Support Vector Machines and a combination of visual and textual features. We evaluate the different retrieval parts of the framework by using a dataset from the footwear and the lithography domain.
... A number of approaches to capture the content in binary images [5], [11], [14], [1], [12] have been developed. Working on patent image retrieval, Huet et al. [5] employed the relational skeletons approach to capture the geometric structure of the image. ...
... Yang et al. [13] exploit the invariant characteristics of the images at the local and global levels by partitioning the image in an adaptive hierarchical way and computing the geometric centroids vector of an image at each partition level. Vrochidis et al. [12] proposed the adaptive hierarchical density histogram (AHDH) which calculates the distribution of black pixels on a white plane. The density estimation is done at local as well as global level. ...
... At local level the image plane is divided into four regions based on the centroid of the image and the local density estimation is performed in each region. Sidiropoulos et al. [9] extended the work presented in [12] by introducing quantized relative density features. They obtain the hierarchical partitioning of an image following the technique presented in [13]. ...
Local primitives are useful in the analysis, recogni-tion and retrieval of document and patent images. In this paper, local primitives are classified in 4 and 8-directional spaces at optimally detected junction and end points by using a distance based approach. Local primitives are quantized by using a variant of Local Binary Patterns. Spatial relationships between local primitives are established by using a morphology based approach. Binary images are described by Local Primitive Histograms of the classified local primitives in 4 and 8-directional spaces capturing their occurrences and pair-wise co-occurrences. Performance evaluation of the proposed Local Primitive Histograms for patent binary image retrieval shows improvement in comparison with histograms obtained by SIFT description of Local Primitives.
... Vrochidis et al. [90] have presented some requirements for the architectural design of a PIR framework. Based on the objective of a PIR system (discussed above), a set of requirements and challenges defined for a generic PIR system is presented: ...
... Taking a review of the existing trademark search techniques, Schietse et al. [75] discuss the practice and challenges in trademark image retrieval. Treating only patent image search, few works [17,37,86,90,99] are available. We give a short overview of each of these in the following. ...
... In an attempt, to exploit the local as well as global content information in patent images, Vrochidis et al. [90] proposed a framework called PATMEDIA in 2009. The framework allows hybrid query submission as text based, content based, and concept based (retrieval of images that depict the concepts of technological domain of interest). ...
To verify the originality of an invention in a patent, the graphical description available in the form of patent drawings often plays a critical role. This paper introduces the importance, requirements, and challenges of a patent image retrieval system. We present a brief account of the work done in the specific and related areas of the patent image domain. We begin with a review of work done dealing specifically with retrieval and analysis of images in the patent domain. Although the literature found dealing with patent images is small, there is a significant amount of work that has been done in related areas that is useful and applicable to the patent image area. From a methodological point of view, we present an overview of the algorithms developed for the retrieval and analysis of CAD and technical drawings, diagrams, data flow diagrams, circuit diagrams, data charts, flowcharts, plots, and symbol recognition.
... An overview of the benefits, requirements and challenges involved in the development of a patent image retrieval framework is provided in [4]. Furthermore, a patent search engine called PatMedia was developed based on the proposed framework . ...
... The PATSEEK [5] application is a content-based image retrieval search engine for the US patent database. Just like PatMedia [4], PATSEEK [5] detects the figures from patent drawings and extracts a feature vector for each figure to be used for retrieval purposes. Both of them use slightly different techniques. ...
... The graph is then further reduced and segmented such that document layout constraints are not violated. The method presented in this article use similar techniques used in [4], [8] and [9] to extract figure captions and part labels. PatMedia used a commercial Optical Character Recognition (OCR) library where as it was not allowed for the USPTO challenge. ...
The US Patent and Trademark Office, together with the NASA Tournament Lab, launched a contest to develop spe-cialized algorithms to help bring the seven million patents presently in the patent archive into the digital age. The contest was hosted by TopCoder.com, the largest competitive online software developer community. The challenge was to detect, segment and recognize figures, captions and part labels from patent drawing images. The solution presented in this work was the winning submission.
... This reflects the importance of the graphical information and this also means graphics/images retrieval is very important [2]. Table, block diagram, flowchart, plot, time chart, line drawing and pictorial information are different classes of figures in patent documents and the categorisation of these figures is illustrated in Fig 1 [3]. Figures in patent documentation are usually binary images. ...
... Similarity is calculated based on the distance of the two images using their features vector. A common content-based image retrieval system is depicted in However, the literature review in Section 2 shows that there is very little published work in the field of patent image retrieval [3,5] and most of the published works are focused on trademark search [6][7][8][9]. Therefore, there is a need to improve the current patent image retrieval (PIR) techniques and to meet users requirements [3,4]. ...
... A common content-based image retrieval system is depicted in However, the literature review in Section 2 shows that there is very little published work in the field of patent image retrieval [3,5] and most of the published works are focused on trademark search [6][7][8][9]. Therefore, there is a need to improve the current patent image retrieval (PIR) techniques and to meet users requirements [3,4]. This remaining paper is organised as follows: Section 2 presents the related work and discuss on potential, benefits and challenges existing Patent Image Retrieval (PIR) approaches; Section 3 presents the architecture of the proposed approach, Section 4 presents the discussions and comparisons and Section 5 concludes this paper. ...
Patent information and images play important roles to describe the novelty of an invention. However, current patent collections do not support image retrieval and patent images are almost unsearchable. This paper presents a short review of the existing research work and challenges in patent image retrieval domain. From the review, the image feature extraction step is found to be an important step to match the query and database images successfully. In order to improve the current feature extraction step in image patent retrieval, we propose a patent image retrieval approach based on Affine-SIFT technique. Comparison discussions between the existing feature extraction techniques are presented to assess the potential of this proposed approach.
... Some recent studies have highlighted the importance of patent drawings. However, most have concentrated on retrieving similar patents based on the images ( Wu, Lam, Mehtre & Gao, 1996 ;Eakins, 2001 ;Tiwari & Bansal, 2004 ;Stefanos et al., 2010 ;Stefanos, Anastasia & Ioannis, 2012 ) and supporting experts in efficiently searching for patents ( Stefanos et al., 2010 ). In contrast, only a few studies have analyzed the content of patent drawings for driving technology intelligence. ...
... Some recent studies have highlighted the importance of patent drawings. However, most have concentrated on retrieving similar patents based on the images ( Wu, Lam, Mehtre & Gao, 1996 ;Eakins, 2001 ;Tiwari & Bansal, 2004 ;Stefanos et al., 2010 ;Stefanos, Anastasia & Ioannis, 2012 ) and supporting experts in efficiently searching for patents ( Stefanos et al., 2010 ). In contrast, only a few studies have analyzed the content of patent drawings for driving technology intelligence. ...
With recent advances in natural language processing and data analytics techniques, useful insights can be extracted not only from bibliographic data but also from the descriptive data of patents. Now, those advances have enabled the use of patent image data as a source of technology intelligence in addition to the two conventional types of patent data. Accordingly, this study focuses on the potential of patent image data and proposes an analysis method for investigating product/service/technology structures using block diagram images among the several types of images in patent documents. Using keywords extracted from patent block diagrams, the following four applications were introduced: (1) analysis of technology evolution, (2) in-depth investigation of technological elements, (3) comparative analysis with competitors, and (4) search for similar patents. The research findings of a case study on mobile earphone technology indicate that keywords are closely related to technological elements, and the four applications are found to be feasible. This study is among the first attempts to support technology intelligence using patent image data. It is also expected to be beneficial in subsequent studies and in practice, wherein patent image data convey valuable information regarding inventions.
... Keyword-based method is one of the most frequently used. However, patent documents always contain graphical data in the form of figures, technical drawings, data flow diagrams, flowcharts, graphs and plots, etc., all of which are critical in specifying the objects and ideas to be patented (Bhatti, Hanbury, & Stottinger, 2017;Vrochidis et al., 2010). An efficient patent search tool needs to maintain the function of image-based retrieval (Vrochidis et al., 2010). ...
... However, patent documents always contain graphical data in the form of figures, technical drawings, data flow diagrams, flowcharts, graphs and plots, etc., all of which are critical in specifying the objects and ideas to be patented (Bhatti, Hanbury, & Stottinger, 2017;Vrochidis et al., 2010). An efficient patent search tool needs to maintain the function of image-based retrieval (Vrochidis et al., 2010). Some researchers, albeit small in number, have explored the idea of an image-based patent search (e.g., Bhatti et al., 2017;Mogharrebi, Choo, & Satria, 2013;Rusiñol, de las Heras, & Terrades, 2015). ...
The relationship between intellectual property rights (IPRs) and the development of creativity is always a controversial topic. However, it has seldom been explored from the user-centered design (UCD) perspective. This paper describes how the UCD approach has been employed to develop Design Patent Retrieval Application (acronym: DsPLAi), a mobile app aimed to integrate IPRs related information into early design processes to enhance designers’ IP practice and to facilitate the creative process. Interview studies were first conducted to identify end-users’ understanding of IPRs and related practices. Next, participatory design workshops with designers and IP processionals were organized to understand the interaction between the two parties and their needs, thereby deriving requirements for DsPLAi. A prototype of the app was developed and evaluated with ten industrial designers. The prototype received positive feedback in the usability evaluation. The empirical results showed that the provision of IPRs related information at an early stage could be helpful to the design process and that the designers were positive about the use of DsPLAi in their daily design routines.
... Several authors have also proposed the utilisation of patent images and sketches in patent analysis, in order to determine similarities between patents (Bhatti and Handbury, 2013). In terms of pre-processing, image analysis challenges involve localisation of images and sub-images, categorisation of images and label recognition (Vrochidis et al., 2010). The primary sources of inter-information are cross-patent citations (Altuntas et al., 2015). ...
... Another task is automatic classification of patents using ML methods. Scenarios include patent quality analysis (Wu et al., 2016), patent categorisation (Vrochidis et al., 2010) and determining the impact of patents on other aspect of companies (Chen et al., 2013). Supervised learning methods, such as support vector machines (Wu et al., 2016) or artificial neural networks (Chen et al., 2013), are frequently used in such cases. ...
Purpose
The purpose of this paper is to evaluate innovations in intellectual property rights (IPR) databases, techniques and software tools, with an emphasis on selected new developments and their contribution towards achieving advantages for IPR management (IPRM) and wider social benefits. Several industry buzzwords are addressed, such as IPR-linked open data (IPR LOD) databases, blockchain and IPR-related techniques, acknowledged for their contribution in moving towards artificial intelligence (AI) in IPRM.
Design/methodology/approach
The evaluation, following an original framework developed by the authors, is based on a literature review, web analysis and interviews carried out with some of the top experts from IPR-savvy multinational companies.
Findings
The paper presents the patent databases landscape, classifying patent offices according to the format of data provided and depicting the state-of-art in the IPR LOD. An examination of existing IPR tools shows that they are not yet fully developed, with limited usability for IPRM. After reviewing the techniques, it is clear that the current state-of-the-art is insufficient to fully address AI in IPR. Uses of blockchain in IPR show that they are yet to be fully exploited on a larger scale.
Originality/value
A critical analysis of IPR tools, techniques and blockchain allows for the state-of-art to be assessed, and for their current and potential value with regard to the development of the economy and wider society to be considered. The paper also provides a novel classification of patent offices and an original IPR-linked open data landscape.
... Эта поисковая система поддерживает четыре поиска: визуальный, текстовый, гибридный и просмотр изображений патента. Система классифицирует каждое изображение патента типом [13]. Такая классификация основывается на текстах в заголовках рисунка, но заголовок не всегда содержит название класса изображения. ...
В работе проанализированы существующие автоматизированные системы поиска и классификации изображений в базе патентов, выделены их преимущества и недостатки. Все известные системы перед поиском предварительно анализируют загруженные изображения и изображения, хранящиеся в базе данных, после чего происходит поиск в патентной базе данных. С ростом времени рассмотрения заявки на регистрацию патента и с увеличением количества патентов растет. Эксперту патентного бюро необходимо установить уникальность патентуемой технологии.
... Explanations and ROIs in the recovered pictures can be arranged by the comment recognizer and starting their use re-audits the outcomes [7].The Annotation-put together recovery depends with respect to the picture metadata or watchwords that relate to the visual substance or the attributes of the picture document. A champion among the most well-known clustering calculations is Fuzzy C-Means (FCM); it is a solo learning system that is anything but difficult to execute and can keep up a greater number of information from the dataset than various techniques [8,9].The centroid enhancement in FCM using particular improvement calculations and the centroid esteems are taken. So as to beat the issues brought by using single element and upgrade the recovery precision, a picture recovery technique which circuits shading, surface and shape, these three essential highlights is proposed, clients can in conclusion get satisfied question results as demonstrated by the applicable criticism results [10]. ...
In late decades, Content-Based Image Retrieval (CBIR) has been one on the most distinctive research zones in the field of Computer applications. The critical goal of this examination is to improve the recovering presentation of CBIR framework by fusing advancement strategies to foresee suitable centroid in Fuzzy C-Means (FCM).The expectation of consolidating streamlining method to anticipate FCM centroids positively decrease intricacy and computational time. The outcomes clear that consolidation of ECSO with FCM uncovers better outcomes over challenge procedures when compared with existing procedures like PWO, SSO and CSO.
... Last but not least, achieving higher retrieval rates in less than a second is most critical. Several applications of CBIR are art collections [26,28], crime prevention [42,43,68], geographical information and remote sensing systems [24], intellectual property [39,77,82,97], medical imaging [3,13,33,67,69,96], military and defense [49], photograph archives and retail catalogs [14,20,23], nudity-detection filters [11,45,46], and face-finding systems [1,36,92]. ...
Content-based image retrieval (CBIR) retrieves visually similar images from a
dataset based on a specified query. A CBIR system measures the similarities
between a query and the image contents in a dataset and ranks the dataset
images. This work presents a novel framework for retrieving similar images
based on color and texture features. We have computed color features with an
improved color coherence vector (ICCV) and texture features with a gray-level
co-occurrence matrix (GLCM) along with DWT-MSLBP (which is derived from
applying a modified multi-scale local binary pattern [MS-LBP] over a discrete
wavelet transform [DWT], resulting in powerful textural features). The optimal
features are computed with the help of principal component analysis (PCA)
and linear discriminant analysis (LDA). The proposed work uses a variancebased
approach for choosing the number of principal components/eigenvectors
in PCA. PCA with a 99.99% variance preserves healthy features, and LDA
selects robust ones from the set of features. The proposed method was tested
on four benchmark datasets with Euclidean and city-block distances. The proposed
method outshines all of the identified state-of-the-art literature methods.
... The concept might have been expressed in other reports or scholarly research even years ago, but we want to highlight that the very first users of the concept perceived it as a "problem" and the "implied" need for cloud computing as "local disk" is incompetent. High volume, variety, and velocity are typical features of big data (Kaur & Singh, 2018), and technologies akin to optical image recognition which can retrieve a certain word in millions of scanned pages enable the usage of big data (Vrochidis et al., 2010). Similarly, data is produced in production facilities, workstations, machinery, maintenance units, vehicles, robots, power plants, supply chains, marketing, and after-sales efforts, etc. in millions of lines as a result of I4.0 technology, and the retrieval and utilization of the right data play an important role in big data. ...
Industry 4.0 (I4.0) marks a new era in manufacturing and has attracted notable attention from practitioners and researchers. Current production processes are being transformed towards interconnecting the elements of manufacturing systems as a result of digitization. Adopting new technologies is an indispensable practice to compete and sustain business concerns. In this paper, the Analytical Hierarchy Process (AHP), a multi-criteria decision-making methodology, is employed to evaluate and weigh the nine pillars that are the building blocks of an I4.0 system. The assessment model suggests three dimensions, nine pillars, and thirty-four sub-pillars which are evaluated by fourteen I4.0 professionals responding to a pairwise questionnaire. The results are important as they reflect the opinions of the professionals and can help define strategies for companies investing in I4.0 technologies by elucidating the relative impacts of factors in an I4.0 environment.
... Especially when dealing with design patents, images are the best cue to find relevant patents. This field is called content-based image retrieval [97]. ...
Patent document collections are an immense source of knowledge for research and innovation communities worldwide. The rapid growth of the number of patent documents poses an enormous challenge for retrieving and analyzing information from this source in an effective manner. Based on deep learning methods for natural language processing, novel approaches have been developed in the field of patent analysis. The goal of these approaches is to reduce costs by automating tasks that previously only domain experts could solve. In this article, we provide a comprehensive survey of the application of deep learning for patent analysis. We summarize the state-of-the-art techniques and describe how they are applied to various tasks in the patent domain. In a detailed discussion, we categorize 40 papers based on the dataset, the representation, and the deep learning architecture that were used, as well as the patent analysis task that was targeted. With our survey, we aim to foster future research at the intersection of patent analysis and deep learning and we conclude by listing promising paths for future work.
... Furthermore, achieving high precision and recall for large datasets is also a challenging task. CBIR applications: art collections [9,10], crime prevention [11][12][13], geographical information and remote sensing systems [14,15], intellectual property [16][17][18][19], medical imaging [20][21][22][23][24][25], military and defence [26], photograph archives and retail catalogs [27][28][29], nudity-detection filters [30][31][32], and face finding systems [33][34][35]. ...
The objective of the content-based image retrieval (CBIR) system is to retrieve the visually identical images from the database efficiently and effectively. It is a broad research realm with the availability of numerous applications. Performance dependence of CBIR focuses on the extraction, reduction, and selection of the features along with the practice of classification technique. In this work, we have proposed the hybrid approach of two different feature descriptors: global color histogram and multi-scale local binary pattern (MS-LBP); furthermore, the use of PCA for dimension reduction and LDA for the selection of features. The proposed method is evaluated concerning various benchmark datasets, viz., Corel-1k, Corel-5k, Corel-10k, and Ghim-10k together with result comparison based on the precision and recall values at different thresholds. The classification purposes are satisfied with Euclidean and City Block distance. The performance study of the proposed work displays it as outperformer than the identified state-of-the-art literature.
... However, text-based techniques are problematical [4] and generic patent image retrieval approaches (e.g. [19]) do not effectively capture the important working principle of the design [15]. Patent image retrieval systems such as Patseek and PatMedia [4] search for visual similarity of images, which is not necessarily a similar working principle, whereas a common working principle between designs suggests a potential conflict with the prior art. ...
Patent infringement detection usually implies research among documents in different forms, in both natural and unstructured language, often involving a lot of human resources and time. In order to ease this patent check process, we previously presented a visual tool to be used by designers themselves at any stage of the design process, providing them with useful and reliable information for deciding whether to steer their design away from potential patent infringements. In this work, we report on a usability study carried out on such a tool with 21 professional designers from industry in the field of mechanical engineering. The outcome of our study shows that our tool is very well accepted by designers, and felt useful and helpful even by legal experts.
... Commercial patent retrieval systems only employ text-based search methods and the need for advanced approaches is becoming more important as text-based techniques are increasingly challenging [8]. However, some possible approaches, such as content-based image retrieval techniques, are not well-suited to patent-images, because they mainly exploit colour images whereas patent images are mostly monochrome [9]. The requirements of a generic Patent Image Retrieval (PIR) system have been defined in the literature [10] and include a semantic-level interpretation of patents still to be developed in patent search systems [4]. ...
One of the main time and money consuming tasks in the design of industrial devices and parts is the checking of possible patent infringements. Indeed, the great number of documents to be mined and the wide variety of technical language used to describe inventions are reasons why considerable amounts of time may be needed. On the other hand, the early detection of a possible patent conflict, in addition to reducing the risk of legal disputes, could stimulate a designers' creativity to overcome similarities in overlapping patents. For this reason, there are a lot of existing patent analysis systems, each with its own features and access modes. We have designed a visual interface providing an intuitive access to such systems, freeing the designers from the specific knowledge of querying languages and providing them with visual clues. We tested the interface on a framework aimed at representing mechanical engineering patents; the framework is based on a semantic database and provides patent conflict analysis for early-stage designs. The interface supports a visual query composition to obtain a list of potentially overlapping designs.
... However, content-based image retrieval techniques are not well-suited to patent-images, for example they exploit colour of images whereas patent images are mostly black-and-white. The requirements of a generic patent image retrieval system have been defined (Vrochidis et al. 2010), which includes a semantic-level interpretation of images not present in contemporary patent search systems (Vrochidis et al. 2012;Bhatti and Hanbury 2013). However, semantic search has been limited to the image descriptive text (Abbas et al. 2014) and other patent image retrieval research has focused on image page orientation, segmentation and low-level feature-extraction (e.g. ...
In some mechanical engineering devices the novelty or inventive step of a patented design relies heavily upon how geometric features contribute to device functions. Communicating the functional interactions between geometric features in existing patented designs may increase a designer’s awareness of the prior art and thereby avoid conflict with their emerging design. This paper shows how functional representations of geometry interactions can be developed from patent claims to produce novel semantic graphical and text annotations of patent drawings. The approach provides a quick and accurate means for the designer to understand the patent that is well suited to the designer’s natural way of understanding the device. Through several example application cases we show the application of a detailed representation of functional geometry interactions that captures the working principle of familiar mechanical engineering devices described in patents. A computer tool that is being developed to assist the designer to understand prior art is also described.
... The textual similarity in a pair of media objects computed via Jaccard index to avoid sparsity problems connected to a vector-based representation. Acoustic and visual similarities in a pair of media objects computed via Euclidean distance, since it is simple to compute and used in well-known image and video retrieval applications [2,67,68]. Once the similarity is computed, appropriate thresholds (θ ti , θ ai , θ vi ) selected to establish in the graph the edges representing textual, acoustic, and visual relationships. The choice of these thresholds allows, for each considered couple of nodes, to decide if a textual, acoustic, and visual relationship must connect them in the graph or not. ...
On web information exists in the form of text, audio, image, and video objects often referred to multiple media objects. Vertical web search provides the search of multiple media information usually via keyword-based queries. The search results in different media formats usually presented in separate panels/tabs; integration is mostly non-blended. Therefore, results exploration via vertical web search engines require the selection of a source and scrolling of a linear ranked list of results. Relationships in the results presented in separate panels/tabs are mostly not considered. Search aggregations unify results from several vertical web sources via blended integration, but exploration still requires scrolling of a linear ranked list. Multimedia search frameworks provide the exploration of results in different media formats but more focused towards the retrieval issues. We proposed a multiple media information search framework to address issues, particularly in aggregated search. Our search framework provides a mechanism to explore results via non-linear ways. The search framework realized by suggesting a framework architecture design and instantiating a search tool. The effectiveness of blended integration and browsing is measured via precision and click through rate respectively. Search task support in results exploration mechanism measured via task-based evaluation. We also validated the conformance of various search/exploration attributes discussed in the state-of-the-art in our frameworks.
... CEDDs consider both color and edge details in matching, unlike most of the other existing complex descriptors do [8,9] . For extracting CEDDs and to compare them, we have still used MatLab and selected the Euclidean distance due to its simplicity, and since it is very used in well-known image and video retrieval applications [2,32,33] . ...
Nowadays, together with the increasing spread of the content available online, users’ information needs have become more complex. To fulfill them, users strongly rely on Web search engines, but traditional ways of presenting search results are often unsatisfactory. In fact, Web pages carry information that exists in multiple media formats, such as text, audio, image and video objects. Vertical search engines and medium-specific search services do not provide users with an integrated view of search results. Furthermore, multiple media objects are in most of the cases highly semantically interlinked, but the connections between them are not sufficiently exploited to provide a further exploration of the retrieved objects. To address these issues, in this paper we propose a graph-based approach aimed at providing users with the possibility to dynamically visualize and explore a search result space built over a repository of multimedia documents containing interconnected multiple media objects. To do this, we represent the search result space via a graph-based data model, where both the retrieved multimedia documents and connected relevant media objects are considered. Media objects, among them, are connected via different kinds of similarity relationships, which depend on the low-level features and metadata taken into consideration to access the media objects. The approach and the connected visualization and exploration interface have been implemented and tested on a publicly available dataset, and they have been evaluated by means of a usability test.
... Searching patent drawings is currently a labour-intensive, error-prone task which would be facilitated by automatic indexing methods. Generic methods for automatically indexing patent drawings have been reported Vrochidis et al., 2010;Codina et al., 2009) but the heterogeneity of patent drawings means that it is difficult for a one-size-fits-all approach to reach high levels of performance on all image classes. Flowcharts represent a large and useful subclass of patent images for which specially-adapted methods will improve indexing performance. ...
The CLEF-IP 2012 track included the Flowchart Recognition task, an image-based task where the goal was to process binary images of owcharts taken from patent drawings to produce summaries containing information about their structure. The textual summaries include information about the owchart title, the box-node shapes, the connecting edge types, text describing owchart content and the structural relationships between nodes and edges. An algorithm designed for this task and characterised by the following method steps is presented: Text-graphic segmentation based on connected-component clustering; Line segment bridging with an adaptive, oriented filter; Box shape classification using a stretch-invariant transform to extract features based on shape-specific symmetry; Text object recognition using a noisy channel model to enhance the results of a commercial OCR package. Performance evaluation results for the CLEF-IP 2012 Flowchart Recognition task are not yet available so the performance of the algorithm has been measured by comparing algorithm output with object-level ground-truth values. An average F-score was calculated by combining node classification and edge detection (ignoring edge directivity). Using this measure, a third of all drawings were recognized without error (average F-score=1.00) and 75% show an F-score of 0.78 or better. The most important failure modes of the algorithm have been identified as text-graphic segmentation, line-segment bridging and edge directivity classification. The text object recognition module of the algorithm has been independently evaluated. Two different state-of-the-art OCR software packages were compared and a post-correction method was applied to their output. Post-correction yields an improvement of 9% in OCR accuracy and a 26% reduction in the word error rate.
... This additional textual information is preprocessed, important semantic aspects are extracted and stored in semantic web format RDF 24 (see 3.1.5). Details on the extraction process, the execution of image similarity search, and a retrieval framework for patent image search are available from Vrochidis et al. [2010]. ...
Todays society generates and stores digital information in enormous amounts and at rapidly increasing rates. This trend affects all parts of modern society, such as commerce and economy, politics and governments, health and medicine, science in general, media and entertainment, the private sector, etc. The stored information comprises text documents, images, audio files, videos, structured data from a variety of sources, as well as multimodal combinations of them, and is available in a multitude of electronic formats and flavors. As a consequence, the need for automated and interactive tools supporting tasks, such as searching, exploring, monitoring, sorting, and making sense of this information at different levels of abstraction and within different but steadily converging domains, increases at the same pace as the data is generated and represents one of the biggest challenges for current computer science.
A relatively young approach to tackle these tasks by exploiting human analytic power in synergetic combination with advanced computerized techniques has emerged with the research field of visual analytics. Visual analytics aims at combining automated methods, visualization techniques, and approaches from the field of human computer interaction in order to equip analysts with more powerful tools, tailored to domains, where large amounts of data must be analyzed. In this work, visual analytics methods and concepts play a central role. They are used to search and analyze texts or multimodal documents containing a considerable amount of textual content. The presented approaches are primarily employed for analyzing a very special type of document from the intellectual property domain, namely patents. Since the retrieval and analysis tasks carried out in the patent domain differ greatly from standard search and analysis tasks regarding rigorous requirements, high costs, and the involved risks, new, more effective, efficient, and more reliable methods need to be developed.
Accordingly, this thesis focuses on researching the combination of automatic methods and information visualization by using advanced interaction techniques in order to improve upon the state of the art in patent literature retrieval. Such integration is achieved and exemplified through different visual analytics prototypes, aiming at creating support for real-world tasks and processes. The main contributions presented in this thesis encompass enhancements for all stages of patent literature analysis processes. This includes improving patent search by presenting techniques for interactive visual query building, which helps analysts to formulate complex information needs, the development of a technique that allows users to build their own precise search mechanism in the form of binary classifiers, and advanced approaches for making sense of a retrieved result set through visual analysis. The latter builds the base to let users generate insights needed for judging and improving previous query formulations. Interaction methods facilitating forward analysis as well as feedback loops, which constitute a critical part of visual analytics approaches, are discussed afterwards. These methods are the key to integrating all stages of the patent analysis process in a seamless visual manner. Another contribution is the discussion of scalability issues in context of the described visual analytics approaches. Especially interaction scalability, the recording of analytic provenance, insight management, the visualization of analytic reporting, and collaborative approaches are addressed.
Although the described approaches are exemplified by applying them to the field of intellectual property analysis, the developments regarding search and analysis have the potential to be adapted to complicated text document retrieval and analysis tasks in various domains. The general ideas regarding the facilitation of low-level feedback loops, user-steered machine classification, and technical solutions for diminishing negative scalability effects can be directly transferred to other visual analytics scenarios.
... For further details on the PATExpert ontology framework the reader is referred to Wanner et al. [275,274] and Giereth et al. [130]. The construction of the ORDO and AUTO domain ontologies is explained by Potrich and Pianta [198], whereas the application of the PDO for image retrieval is explained by Vrochidis et al. [249]. ...
Patents are of great importance for national and international economies, since they have a high impact on the development of products, trade, and research. Therefore, patents are analyzed by multiple interest groups for different purposes, for example for getting technical details, for observing competitors, or for detecting technological trends. Patent analysis typically includes the following steps: formulating a patent query, submitting it to one or more patent data services, merging and analyzing the results for getting an overview, selecting relevant patents for detailed analysis, continuing with refining the query based on the gained insights until the results are satisfying.
In general, patent analysis is a complex, time and knowledge intensive process, because of multiple application domains, data sources, and legal systems but also because of the intentionally used abstract expressions in patents and the huge amount of patent data. Therefore, this thesis investigates the research question of how patent analysis can be supported by the field of visual analytics which combines visualization, human-computer interaction, data analysis, and data management.
A contribution of this thesis is a general architecture for visual patent analysis which is based on a semantic representation model. The model can be accessed from analysis and presentation components, enriched with analysis results, published and reused. Analysis and presentation components are loosely coupled and exchangeable based a shared terminology defined by ontologies. A second contribution is an ontology for patent metadata. It provides an integrated semantic representation of the major patent metadata aspects and allows metadata restrictions to be added to a semantic search. As a third contribution, this thesis describes new visualization techniques, for example a treemap techniques for visualizing patent portfolios and co-classification relations and a graph overlay based technique for visualizing dependencies between text parts based on semantic relations. Additionally, this thesis contributes to the field of data security by describing a new encryption based method for secure sharing of semantic models to a known group of recipients. The main idea of this method is to only encrypt sensitive information while keeping all non-sensitive information publicly readable (partial encryption). This method allows the usage of open infrastructures for the exchange of sensitive semantic models. The adequacy of the proposed architecture has been validated by three prototypes for different use cases. It has been shown that the architecture is flexible and extensible. A first evaluation with patent experts suggested that visual analytics is a promising approach for improving patent analysis.
... The first attempts in patent image search were based on the extraction of visual low level features with a view of retrieving visually similar images based on the query by visual example paradigm. Within this context, several systems have been developed, including PATSEEK (Tiwari and Bansal, 2004) and PatMedia image search engine (Vrochidis et al., 2010). ...
Patent images are very important for patent examiners to understand the contents of an invention. Therefore there is a need for automatic labelling of patent images in order to support patent search tasks. Towards this goal, recent research works propose classification-based approaches for patent image annotation. However, one of the main drawbacks of these methods is that they rely upon large annotated patent image datasets, which require substantial manual effort to be obtained. In this context, the proposed work performs extraction of concepts from patent images building upon a supervised machine learning framework, which is trained with limited annotated data and automatically generated synthetic data. The classification is realised with Random Forests (RF) and a combination of visual and textual features. First, we make use of RF’s implicit ability to detect outliers to rid our data of unnecessary noise. Then, we generate new synthetic data cases by means of Synthetic Minority Over-sampling Technique (SMOTE). We evaluate the different retrieval parts of the framework by using a dataset from the footwear domain. The results of the experiments indicate the benefits of using the proposed methodology.
Content-Based Image Retrieval (CBIR) is a well-known research topic from the computer vision domain which helps retrieve similar images from a dataset as per the specified query image. The retrieval performance is inadequate for benchmark datasets viz., Corel-1k, Corel-5k, Corel-10k, and Ghim-10k. In this paper, we have encountered the problem of the low retrieval rates of the CBIR system and the high dimensionality of the feature vectors. We have proposed the hybrid framework consisting of three different feature descriptors for robust retrieval performance. We have propounded the use of modified Multi-Scale Local Binary Pattern (MS-LBP), Color Coherence Vector (CCV), and Global Color Histogram (GCH) for image retrieval. We have exerted the modified MS-LBP because of its ability to capture more texture detail than Local Binary Pattern (LBP) at multiple scales. This larger filter size of MS-LBP makes it less vulnerable to noise and illumination than the conventional LBP descriptor. The CCV captures color with location information well enough, but it’s vulnerable to the brightened images whereas, the GCH operator covers brightness (less sensitive to brightness than CCV), rotation, scale, translation, camera viewpoint invariant features, but lacks spatial information. The proposed framework improves the feature selection process by blending the strength of each of these descriptors. This paper also targets the high dimensionality of the feature vector of the MS-LBP and GCH descriptors by exerting Principal Component Analysis (PCA). Moreover, Linear Discriminant Analysis (LDA) selects robust and optimal features for retrieval. The proposed method is compared with state-of-the-art literature on four benchmark datasets in terms of Average Retrieval Precision (ARP), Average Retrieval Rate (ARR), and Retrieval Time (RT). Experimental results show that the proposed method excels the examined research practices.
In this paper, the specificity of patent images was studied, a method was developed that uses a neural network for image preprocessing (classification) and their subsequent comparison, the architectures of neural networks for working with images and deep learning libraries were analyzed, and a comparative analysis of existing methods for searching and classifying patent images was carried out. A number of practical tasks have been completed for the search and collection of patent images, the selection of the main classes of patent images, the formation of a training sample; the training of the neural network for the recognition of the selected classes of patent images was carried out, the analysis of the trained model was carried out; a software module was created based on the developed method.
In the twentieth century, the functions of written or spoken language were extensively studied. The functions of drawings were studied less. This was largely due to a kind of ‘verbal-centrism’ that dominated the general discussions on the mechanisms of interaction and communication. This article explores the various possible functions of drawings, focusing on architecture, urban design and planning. It initially attempts to build a typology of the different functions of drawing and, later, to discuss relevant aspects such as the relationship of each function with reality. The article concludes by dwelling on the theoretical and practical importance of this approach.
Improving the performance of the patented image retrieval system is of great significance in the intellectual property protection. The design patent image has a large amount of data, and how to quickly complete the retrieval is part of the main research issues for the design patent retrieval system. Classification is an effective way to improve the retrieval speed, so some methods of image classification have been proposed before. However, image classification cannot achieve high-level semantic classification. Thus the speed of improvement is very limited. In order to realize the classification effect of high-level semantics, in this paper, we propose a method that uses the image caption model-based to realize the automatic description generation of the design patent image. Experiments show that our method has better classification accuracy and better semantic classification performance than previous image classification methods.
Specialized patent retrieval systems mostly use textual information which is difficult enough because of the specialized characteristics of the text. However, in patents there also are drawings which show the invention. Empirical research has shown that patent experts can use these images to determine relevance very quickly. However, these drawings are binary and sometimes abstract and other times very specific; therefore there has not been an effective way to include the visual information into the information retrieval process. In addition, the number and the quality of drawings differs vastly even inside patent classes. This work focuses on the inclusion of images into the patent retrieval process using a combination of visual and textual information. With this multimodal approach it will hopefully be possible to achieve better results than just by using one modality individually. The goal is to develop a prototypical system using an iterative user-centered process.
Background
This paper renders a classification and retrieval of image achievements in the search area of image retrieval, especially content-based image retrieval, an area that has been very active and successful in the past few years.
Objective
Primarily the features extracted established on the bag of visual words (BOW) can be arranged by utilizing Scaling Invariant Feature Transform (SIFT) and developed K-Means clustering method.
Methods
The texture is extracted for a developed multi-texton method by our study. Our retrieval process consists of two stages such as retrieval and classification. The images will be classified established on the features by applying k- Nearest Neighbor (kNN) algorithm. This will separate the images into various classes in order to develop the precision and recall rate initially.
Results
After the classification of images, the similar images are retrieved from the relevant class as per the afforded query image.
Purpose
This paper aims to identify, evaluate and integrate the findings of all relevant and high-quality individual studies addressing one or more research questions about recommender systems and performing a comprehensive study of empirical research on recommender systems that have been divided into five main categories. To achieve this aim, the authors use systematic literature review (SLR) as a powerful method to collect and critically analyze the research papers. Also, the authors discuss the selected recommender systems and its main techniques, as well as their benefits and drawbacks in general.
Design/methodology/approach
In this paper, the SLR method is utilized with the aim of identifying, evaluating and integrating the findings of all relevant and high-quality individual studies addressing one or more research questions about recommender systems and performing a comprehensive study of empirical research on recommender systems that have been divided into five main categories. Also, the authors discussed recommender system and its techniques in general without a specific domain.
Findings
The major developments in categories of recommender systems are reviewed, and new challenges are outlined. Furthermore, insights on the identification of open issues and guidelines for future research are provided. Also, this paper presents the systematical analysis of the recommender system literature from 2005. The authors identified 536 papers, which were reduced to 51 primary studies through the paper selection process.
Originality/value
This survey will directly support academics and practical professionals in their understanding of developments in recommender systems and its techniques.
In this chapter, we will analyse the current technologies available that deal with graphical information in patent retrieval applications and, in particular, with the problem of recognising and understanding information carried by flowcharts. We will review some of the state-of-the-art techniques that have arisen from the graphics recognition community and their application in the intellectual property domain. We will present an overview of the different steps that compound a flowchart recognition system, looking also at the achievements and remaining challenges in such a domain.
In this paper, we present a novel mathematical tool, Structure Integral Transform (SIT), for invariant shape description and recognition. Different from the Radon Transform, which integrates the shape image function over a 1D line in the image plane, the proposed SIT builds upon two orthogonal integrals over a 2D K-cross dissecting structure spanning across all rotation angles by which the shape regions are bisected in each integral. This Structure Integral Transform brings the following advantages over the Radon Transform: (1) It has the extra function of describing the interior structural relationship within the shape which provides a more powerful discriminative ability for shape recognition; (2) The shape regions are dissected by the K-cross in a coarse to fine hierarchical order that can characterize the shape in a better spatial organization scanning from the center to the periphery; and (3) It is easier to build a completely invariant shape descriptor. The experimental results of applying SIT to shape recognition demonstrate its superior performance over the well-known Radon Transform, and the well-known shape contexts and the Polar Harmonic Transforms.
Intellectual property and the patent system in particular have been extremely present in research and discussion, even in the public media, in the last few years. Without going into any controversial issues regarding the patent system, we approach a very real and growing problem: searching for innovation. The target collection for this task does not consist of patent documents only, but it is in these documents that the main difference is found compared to web or news information retrieval. In addition, the issue of patent search implies a particular user model and search process model. This review is concerned with how research and technology in the field of Information Retrieval assists or even changes the processes of patent search. It is a survey of work done on patent data in relation to Information Retrieval in the last 20-25 years. It explains the sources of difficulty and the existing document processing and retrieval methods of the domain, and provides a motivation for further research in the area.
Relatively little research has been done on the topic of patent image retrieval and in general in most of the approaches the retrieval is performed in terms of a similarity measure between the query image and the images in the corpus. However, systems aimed at overcoming the semantic gap between the visual description of patent images and their conveyed concepts would be very helpful for patent professionals. In this paper we present a flowchart recognition method aimed at achieving a structured representation of flowchart images that can be further queried semantically. The proposed method was submitted to the CLEF-IP 2012 flowchart recognition task. We report the obtained results on this dataset.
Recently, the intellectual property and information retrieval communities have shown interest in patent image analysis, which could augment the current practices of patent search by image classification and concept extraction. This article presents an approach for concept extraction from patent images, which relies upon recursive hybrid (text and visual-based) classification. To evaluate this approach, we selected a dataset from the footwear domain.
Access to electronic books, electronic journals, and web portals, which may contain graphics (drawings or diagrams) and images, is now ubiquitous. However, users may have photographs that contain graphics or images and want to access an electronic database to retrieve this information. Hence, an effective photograph retrieval method is needed. Although many content-based retrieval methods have been developed for images and graphics, few are designed to retrieve graphics and images simultaneously. Moreover, existing graphics retrieval methods use contour-based rather than pixel-based approaches. Contour-based methods, which are concerned with lines or curves, are inappropriate for images. To retrieve graphics and images simultaneously, this work applies an adaptive retrieval method. The proposed method uses histograms of oriented gradient (HOG) as pixel-based features. However, the characteristics of graphics and images differ, and this affects feature extraction and retrieval accuracy. Thus, an adaptive method is proposed that selects different HOG-based features for retrieving graphics and images. Experimental results demonstrate the proposed method has high retrieval accuracy even under noisy conditions.
The paper discusses the problem of patent image retrieval. It describes the issues faced when extracting semantic data of images in patents, as well as an integration framework between the data thus extracted and semantic information extracted from text. Combining the two sources of knowledge is on the wish list of many patent information users, as current systems either search only the textual data, or have extremely limited image processing functionality. In practice in the patent domain, depictions of the product or method are often vital to the understanding of the invention. Yet they are almost completely unsearchable. They are tools enclosed in a glass case, at which we can look, but of which we cannot really make use. The IMPEx Project (Image Mining for Patent Exploration) cracks open this case with a new focus on processing this particular type of images. This paper presents the motivations, status and aims of the project.
New computing technologies, media acquisition/storage devices, and multimedia compression standards have increased the amount of digital data generated and stored by computer users. Nowadays, it is easy to access electronic books, electronic journals, and web portals, which contain tremendous graphics (drawings or diagrams) and images (pictures or scenery). Hence, it is imperative to develop an effective graphics/image retrieval method. In particular, when users have photos that may contains graphics or images, they want to access electronic database to retrieve related information. Although many content-based retrieval methods have been developed for images and graphics, few are specifically designed for graphics and images simultaneously. Moreover, most existing graphics retrieval methods use contour-based rather than pixel-based approaches. A contour-based method is concerned with a lot of lines or curves which is not proper for image retrieval. Thus, the objective of this study was to develop simple yet effective graphics/image retrieval using pixel-based features. The proposed method uses histograms of oriented gradient (HOG) as pixel-based features. However, the characteristics of graphics and images differ, and this affects feature extraction and retrieval accuracy. Thus, an adaptive method is proposed to select different HOG-based features for retrieving graphics and images with high retrieval accuracy. Experimental results confirm that the proposed method has high retrieval accuracy.
Drawings are an important component of patents, and many search tasks in the intellectual property domain rely on the comparison of patent drawings. In this paper, we begin with a review of algorithms developed for the automated retrieval of similar images in the patent domain. There is however a larger body of research dedicated to analysis and retrieval of images found in technical documents that is also applicable to the images found in patents. We present an overview of this research for technical drawings, diagrams, charts, plots and chemical structures. Finally, we discuss the evaluation of image retrieval for patents, including short descriptions of the new patent image evaluation tasks in the CLEF-IP and TREC-CHEM evaluation campaigns in 2011.
Recently, the intellectual property and information retrieval communities have shown increasing interest in image retrieval, which could augment the current practices of patent search. In this context, this article presents PatMedia search engine, which is capable of retrieving patent images in content-based manner. PatMedia is evaluated both by presenting results considering information retrieval metrics, as well as realistic patent search scenarios.
Using both text and image content features, a hybrid image retrieval system for Word Wide Web is developed in this paper. We first use a text-based image meta-search engine to retrieve images from the Web based on the text information on the image host pages to provide an initial image set. Because of the high-speed and low cost nature of the text-based approach, we can easily retrieve a broad coverage of images with a high recall rate and a relatively low precision. An image content based ordering is then performed on the initial image set. All the images are clustered into different folders based on the image content features. In addition, the images can be re-ranked by the content features according to the user feedback. Such a design makes it truly practical to use both text and image content for image retrieval over the Internet. Experimental results confirm the efficiency of the system.
This paper presents the evaluation of a number of algorithm
alternatives for content based retrieval from a database of technical
drawings representing patents. The objective is to help patent
evaluators in their quest for a possible patent bearing too much
similarity with the one under investigation. To achieve this, we have
devised a system where images (drawings) are represented using
attributed graphs based on the extracted line-patterns or histograms of
attributes computed from the graphs. Retrieval is either performed using
histogram comparison (see Huet, B. and Hancock, E.R., PAMI, vol.21,
no.12, p.1363-70, 1999) or thanks to a graph similarity measure (see
Huet and Hancock, CVPR, p.138-43, 1998). Promising results are presented
along with possible work extension
In this paper, we outline some of the main challenges facing trademark searchers today, and discuss the extent to which current automated systems are meeting those challenges.
A patent always contains some images along with the text. Many text based systems have been developed to search the patent database. In this paper, we describe PATSEEK that is an image based search system for US patent database. The objective is to let the user check the similarity of his query image with the images that exist in US patents. The user can specify a set of key words that must exist in the text of the patents whose images will be searched for similarity. PATSEEK automatically grabs images from the US patent database on the request of the user and represents them through an edge orientation autocorrelogram. L1 and L2 distance measures are used to compute the distance between the images. A recall rate of 100% for 61% of query images and an average 32% recall rate for rest of the images has been observed.
In this paper, we present a new feature extraction method that simultaneously captures the global and local characteristics of an image by adaptively computing hierarchical geometric centroids of the image. We show that these hierarchical centroids have some very interesting properties such as illumination invariant and insensitive to scaling. We have applied the method for near-duplicate image recognition and for content-based image retrieval. We present experimental results to show that our method works effectively in both applications.
The combination of different search techniques can improve the results given by each one. In the ongoing R&D projec t PATExpert1, four different search techniques are combined to perform a patent search. These techniques are: metadata search, keyword-based sear ch, semantic search and image search. In this paper we propose a general ar chitecture based on web services where each tool works in its own domain an d provides a set of basic functionalities to perform the retrieval. To be abl e to combine the results from the four search engines, these must be fuzzy (using a membership function or similarity grade). We focus on how the fuzzy result s can be obtained from each technique, and how they can then be combined. This combination must take into account the query, the similarity of the paten t to each part of the query, and the confidence on the technique
The paper presents a review of 200 references in content-based image retrieval. The paper starts with discussing the working conditions of content-based retrieval: patterns of use, types of pictures, the role of semantics, and the sensory gap. Subsequent sections discuss computational steps for image retrieval systems. Step one of the review is image processing for retrieval sorted by color, texture, and local geometry. Features for retrieval are discussed next, sorted by: accumulative and global features, salient points, object and shape features, signs, and structural combinations thereof. Similarity of pictures and objects in pictures is reviewed for each of the feature types, in close connection to the types and means of feedback the user of the systems is capable of giving by interaction. We briefly discuss aspects of system engineering: databases, system architecture, and evaluation. In the concluding section, we present our view on: the driving force of the field, the heritage from computer vision, the influence on computer vision, the role of similarity and of interaction, the need for databases, the problem of evaluation, and the role of the semantic gap.
Nowadays, an increasingly growing demand for advanced multimedia search engines is arising, as huge amounts of digital visual content are becoming available. The contribution of this paper is the introduction of a hybrid multimedia retrieval model accompanied by the presentation of a search engine that is capable of retrieving visual content from cultural heritage multimedia libraries as in three modes: (i) based on their semantic annotation with the help of an ontology; (ii) based on the visual features with a view to finding similar content; and (iii) based on the combination of these two strategies in order to produce recommendations. To achieve this, the retrieval model is composed of two different parts, a low-level visual feature analysis and retrieval and a high-level ontology infrastructure. The main novelty is the way in which these two co-operate transparently during the evaluation of a single query in a hybrid fashion, making recommendations to the user and retrieving content that is both visually and semantically similar. A search engine has been developed implementing this model which is capable of searching through digital libraries of cultural heritage collections, and indicative examples are discussed, along with insights into its performance.
The aim of the SCHEMA Network of Excellence is to bring together a critical mass of universities, research centers, industrial partners and end users, in order to design a reference system for content-based semantic scene analysis,
interpretation and understanding. Relevant research areas include: content-based multimedia analysis and automatic annotation of semantic multimedia content, combined textual and multimedia information retrieval, semantic -web, MPEG-7
and MPEG-21 standards, user interfaces and human factors. In this paper, recent advances in content-based analysis, indexing and retrieval of digital media within the SCHEMA Network are presented. These advances will be integrated in the SCHEMA module-based, expandable reference system.
Presents a review of 200 references in content-based image
retrieval. The paper starts with discussing the working conditions of
content-based retrieval: patterns of use, types of pictures, the role of
semantics, and the sensory gap. Subsequent sections discuss
computational steps for image retrieval systems. Step one of the review
is image processing for retrieval sorted by color, texture, and local
geometry. Features for retrieval are discussed next, sorted by:
accumulative and global features, salient points, object and shape
features, signs, and structural combinations thereof. Similarity of
pictures and objects in pictures is reviewed for each of the feature
types, in close connection to the types and means of feedback the user
of the systems is capable of giving by interaction. We briefly discuss
aspects of system engineering: databases, system architecture, and
evaluation. In the concluding section, we present our view on: the
driving force of the field, the heritage from computer vision, the
influence on computer vision, the role of similarity and of interaction,
the need for databases, the problem of evaluation, and the role of the
semantic gap
Extracting information from the ever growing stream of multimedia data is becoming increasingly difficult. One of the main reasons lies within the unstructured way multimedia data are usually presented. Audio-visual material represents a large part of current multimedia material and can be structured in meaningful ways due to the nature of visual communication. This paper surveys several approaches and algorithms that have been recently developed to help in automatically structuring audio-visual data, both for annotation and access 1 . Keywords: audio-visual data, audio segmentation, image databases, image indexing, image retrieval by pictorial content, video annotation, video databases, video segmentation. 1. Introduction The advent of digital technologies has enabled the emergence of a new type of database that supports an evolving collection of media elements [1, 7, 11, 23, 34, 38, 65, 67, 81, 82, 84, 97, 116, 143]. Multimedia database systems store large volumes of text, images...
Information retrieval (IR) has changed considerably in the last years with the expansion of the Web (World Wide Web) and the advent of modern and inexpensive graphical user interfaces and mass storage devices. As a result, traditional IR textbooks have become quite out-of-date which has led to the introduction of new IR books recently. Nevertheless, we believe that there is still great need of a book that approaches the field in a rigorous and complete way from a computer-science perspective (in opposition to a user-centered perspective). This book is an effort to partially fulfill this gap and should be useful for a first course on information retrieval as well as for a graduate course on the topic.
Trademark image searching is potentially one of the most important application areas for automated content-based image retrieval (CBIR) techniques. There are many large and growing collections of trademark images in electronic form. The task of maintaining manual indexes to these image collections is becoming increasingly onerous. And there is a paramount need for accurate and reliable searching, since the images can be of major commercial significance.
The ever-increasing number of registered trademarks has created greater demand for an automatic trademark retrieval system. In this paper, we present a method for such a system based on the image content, using a shape feature. Zernike moments of an image are used for a feature set. To retrieve similarly shaped trademarks quickly, we introduced the concept of a `visually salient feature' that dominantly affects the global shape of the trademarks. Experiments have been conducted on a database of 3000 trademark images. The retrieval speed was very fast and similar-shaped trademark retrieval results were very promising.
Taking as a starting point the actual and the perceived value of models of claimed inventions in the 19th century, the author explores the proposition that non-text disclosures of this type and, more significantly, their modern equivalents as 3D electronic images, as well as many other forms of electronic non-text material, will have an increasing place in fully communicating the inventions filed in 21st century patent applications. He argues that moving to a patent application process in which supplementary machine readable material can be supplied, and can form part of the total disclosure of an invention, without the need for it to be convertible to paper form, would increase intelligibility of invention disclosures of both existing problem areas such as genetic sequence patents, as well as forestalling increasing problems likely to arise in other areas as the 21st century unfolds. He notes that in this way the limitations inherent in having to describe 21st century inventions by using the 15th century technology of the printed page could be overcome. Analogies with the situation for certain types of trademark applications for registration, e.g. sound marks, are also noted.
Today a considerable amount of video data in multimedia databases requires sophisticated indices for its effective use. Manual indexing is the most effective method to do this, but it is also the slowest and the most expensive. Automated methods have then to be developed. This paper surveys several approaches and algorithms that have been recently proposed to automatically structure audiovisual data, both for annotation and access.Copyright 1999 Academic Press.
This paper describes an ongoing research project aimed at implementing a trademark retrieval system using an as-sociative memory neural network. The novel aspect presented in this paper is the proposed integrated framework for image retrieval using multiple representations of images based on gestalt principles. In this paper we summarise the methods we followed in extracting local perceptual features as well as features of the closed figures of images. In designing the search engine of the system we have adopted a novel similarity assessment criteria based on local features as well as features of the closed figures, which is being implemented using an associative memory neural network to achieve high performance in retrieval. Then we describe the strategy we followed in combining multiple similarity measures and present the results obtained from the first phase of evaluation of the system.
With ever increasing number of registered trademarks, the task of trademark office is becoming increasingly difficult to ensure the uniqueness of all trademarks registered. Trademarks are complex patterns consisting of various image and text patterns, called device-mark and word-in-mark respectively. Due to the diversity and complexity of image patterns occurring in trademarks, due to multi-lingual word-in-mark, there is no very successful computerized operating trademark registration system. We have tackled key technical issues: multiple feature extraction methods to capture the shape, similarity of multi-lingual word-in-mark, matching device mark interpretation using fuzzy thesaurus, and fusion of multiple feature measures for conflict trademark retrieval. A prototype System for Trademark Archival and Registration (STAR) has been developed. The initial test run has been conducted using 3000 trademarks, and the results have shown satisfaction to trademark officers and specialists.
Proper processing and efficient representation of the digitized images of printed documents require the separation of the various information types: text, graphics, and image elements. For most applications it is sufficient to separate text and nontext, because text contains the most information. This paper describes the implementation and performance of a robust algorithm for text extraction and segmentation that is completely independent of text orientation and can deal with text in various font styles and sizes. Text objects can be nested in nontext areas, and inverse printing can also be analyzed. It should be mentioned that the classification is based only on rough image features, and individual characters are not recognized. The three main processing steps of the system are the generation of connected components, neighborhood analysis, and generation of text lines and blocks. As output, connected components are classified as text or nontext. Text components are grouped as characters, words, lines, and blocks. Nontext objects are accumulated as a separate nontext block.
With ever increasing number of registered trademarks, the task of trademark office is becoming increasingly difficult to ensure the uniqueness of all trademarks registered. Trademarks are complex patterns consisting of various image and text patterns, called device-mark and word-in-mark respectively. Due to the diversity and complexity of image patterns occurring in trademarks, due to multi-lingual word-in-mark, there is no very successful computerized operating trademark registration system. We have tackled key technical issues: multiple feature extraction methods to capture the shape, similarity of multi-lingual word-in-mark, matching device mark interpretation using fuzzy thesaurus, and fusion of multiple feature measures for conflict trademark retrieval. A prototype System for Trademark Archival and Registration (STAR) has been developed. The initial test run has been conducted using 3000 trademarks, and the results have shown satisfaction to trademark officers and specialists.
In this article, we present ongoing work on an advanced patent processing service PATExpert. The central assumption underlying PATExpert is that in order to meet the needs of the users of patent processing services, recourse must be made to the content of patent material. We introduce a content representation schema for patent documentation and sketch the design of techniques that facilitate the integration of this schema into the patent processing cycle. Two types of techniques are discussed. Techniques of the first type facilitate the access to the content of patent documentation provided in a textual format - be it by the human reader or by the machine - in that they rephrase and summarize the documentation and map it onto a formal semantic representation. Techniques of the second type operate on the content representation. At this stage, PATExpert is explored in two technology areas - optical recording devices and machine tools. The work is being carried out in the framework of an R&D-project partially funded by the European Commission.
In focusing on the field of searching mechanical and device patents, the author explores the available resources, their uses and their limitations. In contrast to searches in the field of chemical patents, this field does not usually provide the benefits of consistent terminology, deep level classification and indexing or direct searching of drawings (cf structural compound search in chemistry). The many challenging problems that result for the searcher and the state of the art in progress on direct searching of drawings are reviewed. Illustrative examples in the fields of medical inhalers and pump actuators are provided. It is clear that there is still a long way to go before practicable, searchable databases can be created or even one of the key initial steps in this process - the general filing and manipulation of 3D drawings in patent applications - can be realised. Methods of searching patent images and tools for faster evaluation of drawings in the full patent specification would be welcomed.
Since there are mass of images in the outward-appearance patent-image database, a retrieval approach developed for this database is greatly demanded. The shape information of the patent image is the key feature for a patent image. Therefore, we present an outward-appearance patent images retrieval approach based on contour- description matrix. This matrix represents the 2D shape feature and shows the angular and radial distributing information. The proposed representation is invariant to scale, translation and rotation. Our experiments show that the contour-based technique performs significantly better compared to wavelet-based approach previously proposed in the literature. Key words: shape retrieval; patent image
An ever increasing number of registered trademarks has created greater demand for an automatic trademark retrieval system. We present a method for such a system based on the image content using shape features. Zernike or pseudo-Zernike moments of the image are employed as a feature set. To retrieve similar shapes, we take into account visually salient features that dominantly affect the global shape of the trademarks and ignore their minor detail. Experimental results on a database of 3,000 trademark images demonstrate that the proposed method retrieves visually similar trademarks which agree well with human perception
The Artisan system retrieves abstract trademark images by shape
similarity. It analyzes each image to characterize key shape components,
grouping image regions into families that potentially mirror human image
perception, and then derives characteristic indexing features from these
families and from the image as a whole. We have evaluated the retrieval
effectiveness of our prototype system on more than 10,000 images from
the UK Trade Marks Registry
Retrieval efficiency and accuracy are two important issues in designing a content-based database retrieval system. We propose a method for trademark image database retrieval based on object shape information that would supplement traditional text-based retrieval systems. This system achieves both the desired efficiency and accuracy using a two-stage hierarchy: in the first stage, simple and easily computable shape features are used to quickly browse through the database to generate a moderate number of plausible retrievals when a query is presented; in the second stage, the candidates from the first stage are screened using a deformable template matching process to discard spurious matches. We have tested the algorithm using hand drawn queries on a trademark database containing 1; 100 images. Each retrieval takes a reasonable amount of computation time (¸ 4-5 seconds on a Sun Sparc 20 workstation). The top most image retrieved by the system agrees with that obtained by human subjects, ...
Towardscontent-oriented patent
- L Wanner
- R Baeza-Yates
- S Brugmann
- J Codina
- B Diallo
- E Escorsa
Wanner L, Baeza-Yates R, Brugmann S, Codina J, Diallo B, Escorsa E, et al. Towardscontent-oriented patent Informat 2007;30(1):21–33.
Towards patent image retrieval, international patent information conference and exposition (IPI–ConfEx
- Vrochidis S In
Vrochidis S. In: Towards patent image retrieval, international patent information conference and exposition (IPI–ConfEx 2009
Content-based retrieval for trademark registration Available from
- Wu
- Jk
Wu JK et al. Content-based retrieval for trademark registration. Multimedia Tool Appl 1996;3:245–67. Available from: <http://www.springerlink.com/ content/u5t11g636m21t103/>.
Practice and challenges in trademark image retrieval. In: Proc 6th ACM int conf on image and video retrieval (CIVR)
- Eakins J Jp Schietse
- Veltkamp
Schietse J, Eakins JP, Veltkamp RC. Practice and challenges in trademark image retrieval. In: Proc 6th ACM int conf on image and video retrieval (CIVR); 2007. p. 518–24.
Patent image retrieval, information retrieval facility symposium
- Vrochidis S In
Vrochidis S. In: Patent image retrieval, information retrieval facility symposium (IRFS 2008), Vienna, Austria, 5–8 November 2008.
An outward-appearance patent-image retrieval approach based on the contour-description matrix. In: Proc 2007 Japan– China joint workshop on frontier of computer science technology
- Z Zeng
- J Zhao
- Xu
Zeng Z, Zhao J, Xu B. An outward-appearance patent-image retrieval approach based on the contour-description matrix. In: Proc 2007 Japan– China joint workshop on frontier of computer science technology; 2007. p. 86–9.
Advanced content-based semantic scene analysis and information retrieval: the schema project
- E Izquierdo
- J Casas
- R Leonardi
- P Migliorati
- O Connor
- N Kompatsiaris
Izquierdo E, Casas J, Leonardi R, Migliorati P, O'Connor N, Kompatsiaris I, et al.
Advanced content-based semantic scene analysis and information retrieval:
the schema project. In: Proc workshop on image analysis for multimedia
interactive services, London, UK; 2003. p. 519–28.
Towards patent image retrieval, international patent information conference and exposition
- S Vrochidis
- In