Coarse-grained precision, recall and F-measure for person names in the blind test set. 

Context in source publication

Context 1

... by maximizing Like the dictionary-based extraction accuracy extractor, over the the positives plate per were dictionary, also taken with from dictionaries short lists including of mutually all development Regex extractor test also (validation) uses dictionaries set. to recognize exclusive the dictionaries categories: used by US the States previous (149), two street extractors. signs tokens that should be considered components of a (11) and school suffixes (6). person name. Matching of entries in these dic- Among the four base extractors, figures 3 and 4 tionaries is stage-wise case-sensitive. By this we show that the Regex extractor generally produces mean that the extractor first finds matching tokens the highest quality extractions overall. Much of the in a case-sensitive manner. Then for each page improvement exhibited by the Regex extractor over in which a dictionary entry is found, the extractor the simpler dictionary extractor comes from the reg- looks for case-insensitive matches of that word. The ular expression pattern matching which constrains Regex extractor then labels any token pattern as a possible matches to only the above patterns. The full name wherever one of the following regular ex- Regex extractor does less well on family and lo- pression patterns is found. Note that the patterns are cal histories (e.g. Libby and Fairfield) where the described in Perl5 regular expression syntax. given regular expressions do not consistently apply: there are many names that consist of only a single given name. This could be corrected with contex- tual clues. The MEMM extractor is a maximum entropy Markov model similar to that used in (Chieu and Ng, 2003) and trained on CoNLL NER training data (Sang and Meulder, 2003) in the newswire genre. Because of the training data, this MEMM was trained to recognize persons, places, dates and organizations in unstructured text, but we evaluated it only on the person names in the OCR corpus. The feature templates used in the MEMM follow. For dictionary features, there was one feature tem- plate per dictionary, with dictionaries including all the dictionaries used by the previous two extractors. The validation / development test set was used to select the most promising variation of the MEMM. Variations considered but rejected included the use of a character noise model in conjunction with an al- lowance for small edit distances (from zero to three) when matching dictionary entries, similar in spirit to, though less well developed than (Wang et al., 2009). Variations also included additional feature templates based on centered 5-grams. By way of comparison, this same MEMM was trained and tested on CoNLL data, where it achieved 83.1% F-measure using the same feature templates applied to the OCR data, as enumerated. This is not a state-of-the-art CoNLL NER system but it allows for more flexible experimentation. Figures 5 and 6 show the greatest quality differ- ence with respect to the other extractors in the two city directories (Birmingham and Portland). These directories essentially consist of lists of the names of people living in the respective cities, followed by terse information about them such as addresses and business names, one or two lines per person. Furthermore, the beginning of each entry is the name of the person, starting with the surname, which is less common in the data on which the MEMM was trained. The contrast between the newswire genre and most of the test data explains its relatively poor performance overall. Previous studies on domain mismatch in supervised learning but especially in NER (Vilain et al., 2007) document similar dramatic shortfalls in performance. The CRF extractor uses the conditional random field implementation in the Mallet toolkit (McCallum, 2002). It was trained and executed in the same way as the MEMM extractor described above, including the use of identical feature templates. Training and testing on the CoNLL data, as we did with the MEMM extractor, yielded a 87.0% F-measure. The CRF extractor is the only one of the four base extractors not included in the ensemble. Adding the CRF resulted in slightly lower scores on the development test set. We also ran the ensemble with the CRF but without the MEMM, resulting in a 2% lower score on the development test set, ruling it out. Separate experiments on CoNLL test data with arti- ficial noise introduced showed similarly worse be- havior by the CRF, relative to the MEMM. We combined the decisions of the first three base extractors described above using a simple voting-based ensemble. The ensemble interprets a full name in each base extractor’s output as one vote in favor of that entity as a person name. The general ensemble extractor is parameterized by a threshold, t , indi- cating how many of the base extractors must agree on a person name before it can be included in the ensemble’s output. By varying this parameter, we produced the three following ensemble extractors: Figure 3 shows that the Majority Ensemble out- performs each base extractor in terms of F-measure. A second set of ensembles was developed. They are identical to the three except that they allowed each base extractor to vote on individual tokens. This fine-grained ensemble did not produce accuracies as high as the coarse-grained approach when using the coarse-grained metrics, but when we use the fine-grained metrics it did better, achieving 68% F-measure over the entire corpus while the coarse- grained ensemble achieved only 60.7% F-measure. The highest-scoring base extractor (Regex) achieved 66.5% using the fine-grained metric. So, again, an ensemble did better than each base extractor regard- less of the metric (coarse or fine), as long as the matching version of the ensemble was applied. In conclusion, we answer the questions posed in the of We these would results like is to useful acknowledge for a different Ancestry.com application. and introduction. WER varies widely in this dataset: If Lee the Jensen intended of Ancestry.com application is for a person providing name the search OCR the average is much higher than the 20% reported engine, data from users their do free-text not want collection to manually and for sift financial through in other papers (Miller et al., 2000). In a plot of many support. false-positives; We would also with like a to sufficiently thank Lee Jensen large cor- for WER versus NER performance shown in figure 7, pus discussions containing regarding millions applications of book and of this newspaper work and ti- the linear fit is substantially poorer than for the data tles, the related a precision constraints. of 89.6% would be more desirable reported in the work of Miller et al. than a precision of 61.6%, even when only 14.1% Ranges of 0–64% or 28–89% F-measure for NER of the names available in the corpus can be recog- can be expected on noisy OCR data, depending on nized (low recall). Alternatively, if higher recall is the document and the metric. Figure 7 shows some necessary for an application in which no instances but not perfect correlation between NER quality should be missed, then the high-recall Union En- and WER. Among those errors that directly cause semble could be used as a filter of the candidates greater WER, different kinds of errors affect NER to be shown. Browsing and exploration of a data set quality to different degrees. for every case may be such an application. High- The Libby text’s WER was lower because of poor recall name browsing could facilitate manual label- character-level recognition (word order was actually ing or checking. good) while Inverness had more errors in word order This work is a starting point against which to where text from two columns has been incorrectly compare techniques which we hope will be more ef- interleaved by the OCR engine (its character-level fective in automatically adapting to new document recognition was good). From error analysis on such formats and genres in the noisy OCR setting. One examples, it seems likely that word order errors play way to adapt the supervised machine learning ap- a bigger role in extraction errors than do character proaches is in applying a more realistic noise model recognition errors. of OCR errors to the CoNLL data. Another is to We also conclude that combining basic methods use semi-supervised machine learning techniques to can produce higher quality NER. Each of the three take advantage of the large volume of unlabeled and ensembles maximizes a different metric. The Ma- previously unused data available in each of the ti- jority Ensemble achieves the highest F-measure over tles in this corpus. We plan to contrast this with the the entire corpus, compared to any of the base ex- more laborious method of producing labeled train- tractors and to the other ensembles. The Intersec- ing data from within the present corpus. Additional tion Ensemble achieves the highest precision and the feature engineering and additional labeled pages for Union Ensemble achieves the highest recall. Each evaluation are also in order. The rule-based Regex extractor could also be adapted automatically to dif- fering document or page formats by filtering a larger set of regular expressions in the first of two passes over each document. Finally, we plan to combine NER with work on OCR error correction (Lund and Ringger, 2009) to see if the combination can improve accuracies jointly in both OCR and information extraction. In conclusion, we answer the questions posed in the introduction. WER varies widely in this dataset: the average is much higher than the 20% reported in other papers (Miller et al., 2000). In a plot of WER versus NER performance shown in figure 7, the linear fit is substantially poorer than for the data reported in the work of Miller et al. Ranges of 0–64% or 28–89% F-measure for NER can be expected on noisy OCR data, depending on the document and the metric. Figure 7 shows some but not perfect ...