ArticlePDF Available
A preview of the PDF is not available
... The patent industry involves the management and tracking of an enormous amount of data, much of which takes the form of scientific drawings, technical diagrams and hand sketched models. The comparison of figures across this dataset and subsequent retrieval based on similarity in real-time is extremely challenging [15], [29], [30]. We aim to track the spread of technical information by finding copies and modified copies of technical diagrams in patent databases and academic journals. ...
... The dataset used to train and test our model is taken from a patent image search benchmark [29]. About 2000 sketchtype images are manually extracted from approximately 300 patents belonging to A43B and A63C IPC subclasses and contain types of foot-wear or portions thereof (henceforth termed "concepts"). ...
... The dataset consists of 8 concepts for this domain: cleat, ski boot, high heel, lacing closure, heel with spring, tongue, toe cap and roller blade. The details for the dataset can be found in [29]. The concepts dataset contains many dissimilarities within each class and is not suitable to train a classifier model to be used as a retrieval and matching framework. ...
... First, a survey paper [1] indicates drawings are an important component of patents and our study on a small set of figures randomly selected from US patents indicates that they represent approximately 95% in the design patents. Second, these descriptors can be especially useful to build search systems [2][3][4] that aid patent examiners to search for similar designs, which will speed up the patent examination and approval processes. ...
... To build the ground truth corpus, we randomly selected 3300 figure captions from 3300 patent figures in the 2020 dataset. Each caption is manually annotated by researchers in our lab using brat, a web-based annotation tool 3 . Two examples of annotated captions are shown in more specific descriptions of an object. ...
... 1. (Vrochidis et al., 2012) present an approach for automatically extracting concept information describing the patent image content to support searchers during patent retrieval tasks. The approach is based on a supervised machine learning framework, which relies upon image and text analysis techniques. ...
... querying and retrieving abstract, technical drawings [38,51,69], and so it is unclear whether the computer vision approaches that are successful with general images are similarly successful with technical drawings. Rather than evaluating computer vision algorithms directly, we used common search engines to evaluate whether these black-box systems (presumably incorporating state-of-the-art image retrieval algorithms) work as well in retrieving diagrams as they do with retrieving photographs. ...
Preprint
Much computer vision research has focused on natural images, but technical documents typically consist of abstract images, such as charts, drawings, diagrams, and schematics. How well do general web search engines discover abstract images? Recent advancements in computer vision and machine learning have led to the rise of reverse image search engines. Where conventional search engines accept a text query and return a set of document results, including images, a reverse image search accepts an image as a query and returns a set of images as results. This paper evaluates how well common reverse image search engines discover abstract images. We conducted an experiment leveraging images from Wikimedia Commons, a website known to be well indexed by Baidu, Bing, Google, and Yandex. We measure how difficult an image is to find again (retrievability), what percentage of images returned are relevant (precision), and the average number of results a visitor must review before finding the submitted image (mean reciprocal rank). When trying to discover the same image again among similar images, Yandex performs best. When searching for pages containing a specific image, Google and Yandex outperform the others when discovering photographs with precision scores ranging from 0.8191 to 0.8297, respectively. In both of these cases, Google and Yandex perform better with natural images than with abstract ones achieving a difference in retrievability as high as 54\% between images in these categories. These results affect anyone applying common web search engines to search for technical documents that use abstract images.
Article
With recent advances in natural language processing and data analytics techniques, useful insights can be extracted not only from bibliographic data but also from the descriptive data of patents. Now, those advances have enabled the use of patent image data as a source of technology intelligence in addition to the two conventional types of patent data. Accordingly, this study focuses on the potential of patent image data and proposes an analysis method for investigating product/service/technology structures using block diagram images among the several types of images in patent documents. Using keywords extracted from patent block diagrams, the following four applications were introduced: (1) analysis of technology evolution, (2) in-depth investigation of technological elements, (3) comparative analysis with competitors, and (4) search for similar patents. The research findings of a case study on mobile earphone technology indicate that keywords are closely related to technological elements, and the four applications are found to be feasible. This study is among the first attempts to support technology intelligence using patent image data. It is also expected to be beneficial in subsequent studies and in practice, wherein patent image data convey valuable information regarding inventions.
Chapter
In this paper, the specificity of patent images was studied, a method was developed that uses a neural network for image preprocessing (classification) and their subsequent comparison, the architectures of neural networks for working with images and deep learning libraries were analyzed, and a comparative analysis of existing methods for searching and classifying patent images was carried out. A number of practical tasks have been completed for the search and collection of patent images, the selection of the main classes of patent images, the formation of a training sample; the training of the neural network for the recognition of the selected classes of patent images was carried out, the analysis of the trained model was carried out; a software module was created based on the developed method.
Thesis
Full-text available
Big data is increasingly available in all areas of manufacturing, which presents value for enabling a competitive data-driven economy. Increased data availability presents an opportunity to introduce the next generation of innovative technologies. Firms invest in innovation and patents to increase, maintain and sustain competitive advantage. Consequently, the valuation of patents is a key determinant in economic growth since patents are an important innovation indicator. Given the surge in patenting throughout the world, the interest in the value of patents has grown significantly. Traditionally, studies on patent value have focused on limited data availability restricted to a specific technology area using methods such as regression, and mostly using numeric and binary categoric data types. We propose the definition for intellectual property intelligence (IPI) as the data science of analysing large amount of IP information, specifically patent data, with artificial intelligence (AI) methodologies to discover relationships and trends in the data for decision making. With the rise of AI and the ability to analyse larger datasets of patents, we develop an AI deep learning methodology for the valuation of patents. To do that, we build a large USPTO dataset consisting of all granted patents from 1976-2019: (i) we collect, clean, collate and pre-process all the data from the USPTO (and the OECD patent quality indicators database); (ii) we transform the data into numeric, categoric, and text features so that we are able to input them to the deep learning model. More specifically, we transform the text (abstract, claims, summary, title) into feature vectors using our developed Doc2Vec vector space model (VSM), that we assess using the t-distributed stochastic neighbour embedding (t-SNE) visualisation. The dataset is made publicly available for researchers to efficiently and effectively run fairly complex data analysis. We propose an AI deep learning methodology for the valuation of patents to identify valuable patents. Using our developed dataset, we build AI deep learning models, which are based on deep and wide feed-forward artificial neural networks (ANN), with dropout, L2 penalty and batch normalisation regularisation layers, to forecast the value of patents with 12 ex-post patent value output proxies. These include the grant_lag, generality, quality_index_4, and forward citations, generality_index and renewals in three time horizons (t4, t8, t12). We associate these patent value proxies to their respective patent value dimension (economic, strategic and technological). We forecast patent value using ex-ante patent value input determinants, for a wide range of technological areas (using the IPC classes), and time horizon domains (short term in t4, medium term in t8, and long term in t12). We evaluate all our models using a variety of strategies (out-of-time test, out-of-sample test, k-Fold and random split cross validation), and transparently report all metrics (accuracy, confusion matrix, F1-score, false negative rate, log loss, mean absolute error, precision, recall). Our models have higher accuracy and macro average F1-scores, with low values for the training and validation losses compared to prior art. With increasing prediction horizons, we observe an increase in the macro average F1-scores for several of the proxies. In addition, we find that the composite index that takes into consideration more than one value dimension, has the combined highest accuracy and macro average F1-score, relative to single value dimension patent proxies. Moreover, we find that firms seem to file widely at the short term time horizon and then focus their technological competencies to established opportunities. Patent owners seem to renew their patents in the fear of losing out. Our study has moved away from relatively small datasets, limited to specific technology field, and allowed for reproducibility in other fields. We can tailor models to different technology area, with different patent value proxies, with different time horizons. This study proposes an AI methodology, which is based on deep learning, using deep and wide feed forward artificial neural networks, to predict the value of patents, which has academic and industrial implications. We predict the value of patents with a variety of output proxies, including composite index proxies, for different technology areas (IPC classifications) and time horizons. Since we use all USPTO granted patents from 1976-2019 to train our models, we can apply this approach to patents in any technology field. Our approach enables researchers and industry professionals to value patents using a variety of patent value proxies, based on different value dimensions, tailored to specific technology areas. The proposed AI deep learning approach could effectively support expert decision making (technology, innovation and IP managers etc.) in their decision making by providing fast, low cost, data-driven intellectual property intelligence (IPI) from big patent data. Firms with limited resources, i.e. small-medium enterprises (SMEs) can choose representative proxies to forecast patent value estimates, saving resources. Consequently, the proposed approach could efficiently support experts in their patent value judgement, policy making in the government’s investments in technological sectors of the future to support the economy, and patent offices with the AI approaches to analyse efficiently and effectively big patent data. We anticipate this research would be interesting for future researchers to expand the emerging field of IPI research and the skills they will need to perform IPI data-driven research with a variety of data sources and AI deep learning ANN approaches.
Conference Paper
Full-text available
The aim of this document is to describe the methods we used in the Patent Image Classification and Image-based Patent Retrieval tasks of the Clef-IP 2011 track. The patent image classification task consisted in categorizing patent images into pre-defined categories such as abstract drawing, graph, flowchart, table, etc. Our main aim in participating in this sub-task was to test how our image categorizer performs on this type of categorization problem. Therefore, we used SIFT-like local orientation histograms as low level features and on the top of that we built a visual vocabularies specific to patent images using Gaussian mixture model (GMM). This allowed us to represent images with Fisher Vectors and to use linear classifiers to train one-versus-all classifiers. As the results show, we obtain very good classification performance. Concerning the Image-based Patent Retrieval task, we kept the same image repre-sentation as for the Image Classification task and used dot product as similarity measure. Nevertheless, in the case of patents the aim was to rank patents based on patent similarities, which in the case of pure image-based retrieval implies to be able to compare a set of images versus another set of images. Therefore, we investigated different strategies such as averaging Fisher Vector representation of an image set or considering the maximum similarity between pairs of images. Finally, we also built runs where the predicted image classes were considered in the retrieval process.
Article
Full-text available
In this paper a new approach to video event detection is presented, combining visual concept detection scores with a new dimensionality reduction technique. Specifically, a video is first decomposed to a sequence of shots, and trained visual concept detectors are used to represent video content with model vector sequences. Subsequently, an improved subclass discriminant analysis method is used to derive a concept subspace for detecting and recognizing high-level events. In this space, the median Hausdorff distance is used to implicitly align and compare event videos of different lengths, and the nearest neighbor rule is used for recog-nizing the event depicted in the video. Evaluation results obtained by our participation in the Multimedia Event De-tection Task of the TRECVID 2010 competition verify the effectiveness of the proposed approach for event detection and recognition in large scale video collections.
Article
Full-text available
Efficient and effective handling of video documents depends on the availability of indexes. Manual indexing is unfeasible for large video collections. In this paper we survey several methods aiming at automating this time and resource consuming process. Good reviews on single modality based video indexing have appeared in literature. Effective indexing, however, requires a multimodal approach in which either the most appropriate modality is selected or the different modalities are used in collaborative fashion. Therefore, instead of separately treating the different information sources involved, and their specific algorithms, we focus on the similarities and differences between the modalities. To that end we put forward a unifying and multimodal framework, which views a video document from the perspective of its author. This framework forms the guiding principle for identifying index types, for which automatic methods are found in literature. It furthermore forms the basis for categorizing these different methods.
Conference Paper
Full-text available
This paper proposes a novel binary image descriptor, namely the Adaptive Hierarchical Density Histogram, that can be utilized for complex binary image retrieval. This novel descriptor exploits the distribution of the image points on a two-dimensional area. To reflect effectively this distribution, we propose an adaptive pyramidal decomposition of the image into non-overlapping rectangular regions and the extraction of the density histogram of each region. This hierarchical decomposition algorithm is based on the recursive calculation of geometric centroids. The presented technique is experimentally shown to combine efficient performance, low computational cost and scalability. Comparison with other prevailing approaches demonstrates its high potential.
Article
Taking as a starting point the actual and the perceived value of models of claimed inventions in the 19th century, the author explores the proposition that non-text disclosures of this type and, more significantly, their modern equivalents as 3D electronic images, as well as many other forms of electronic non-text material, will have an increasing place in fully communicating the inventions filed in 21st century patent applications. He argues that moving to a patent application process in which supplementary machine readable material can be supplied, and can form part of the total disclosure of an invention, without the need for it to be convertible to paper form, would increase intelligibility of invention disclosures of both existing problem areas such as genetic sequence patents, as well as forestalling increasing problems likely to arise in other areas as the 21st century unfolds. He notes that in this way the limitations inherent in having to describe 21st century inventions by using the 15th century technology of the printed page could be overcome. Analogies with the situation for certain types of trademark applications for registration, e.g. sound marks, are also noted.
Article
The automatic removal of suffixes from words in English is of particular interest in the field of information retrieval. An algorithm for suffix stripping is described, which has been implemented as a short, fast program in BCPL. Although simple, it performs slightly better than a much more elaborate system with which it has been compared. It effectively works by treating complex suffixes as compounds made up of simple suffixes, and removing the simple suffixes in a number of steps. In each step the removal of the suffix is made to depend upon the form of the remaining stem, which usually involves a measure of its syllable length.
Article
LIBSVM is a library for support vector machines (SVM). Its goal is to help users to easily use SVM as a tool. In this document, we present all its imple-mentation details. For the use of LIBSVM, the README file included in the package and the LIBSVM FAQ provide the information.
Chapter
With ever increasing number of registered trademarks, the task of trademark office is becoming increasingly difficult to ensure the uniqueness of all trademarks registered. Trademarks are complex patterns consisting of various image and text patterns, called device-mark and word-in-mark respectively. Due to the diversity and complexity of image patterns occurring in trademarks, due to multi-lingual word-in-mark, there is no very successful computerized operating trademark registration system. We have tackled key technical issues: multiple feature extraction methods to capture the shape, similarity of multi-lingual word-in-mark, matching device mark interpretation using fuzzy thesaurus, and fusion of multiple feature measures for conflict trademark retrieval. A prototype System for Trademark Archival and Registration (STAR) has been developed. The initial test run has been conducted using 3000 trademarks, and the results have shown satisfaction to trademark officers and specialists.