Pairs of image and corresponding OCR text from one page of Montclair. 

Contexts in source publication

Context 1

... our within knowledge, an What OCR this level engine collection of produced NER quality has by greater Kofax is achiev- vari- (Ko- dark annotated splotches for the in current the noisy project. image One and to the two fact pages that able ety fax, in 2009). in formatting a couple The of newspapers and months genre of were development than any OCRed other time, by image- an par- en- the from OCR each engine document failed were to recognize annotated column for each bound- of the ticularly and-text when corpus no used annotated in a published data is available NER experi- for the aries development during zoning. test and In blind figure 2, test letter-spacing sets. The annota- is in- corpus ment. for The training data includes or evaluation unstructured purposes? text How (full correctly tions consisted interpreted of marking by the OCR person engine, names, resulting including in well sentences), can we structured do on a (tabular) truly noisy text and including diverse long cor- the titles. introduction The number of superfluous of names annotated spaces within in the words. blind pus lists of of OCR names data? and end-of-book How do competing indexes, and extraction multi- This test set figure is given also in illustrates table 1 for the each common document problem in the of approaches column formatted compare text over from different the books document and newspa- types? words corpus. that Blind are split test and pages hyphenated were not inspected at line bound- dur- Can pers. improvements in extraction quality be gained aries ing the as development well as other of types the of extraction errors. systems. All by combining the strengths of different extractors? extractors, This particular including collection ensembles, of were OCR applied documents to the We provide answers to these questions in the fol- were originally intended to be indexed for keyword lowing sections. In § 2 we describe the data we used search. Because the search application requires as well as the names extracted. In § 3 we present no more than a bag-of-words representation, much each of the basic extraction methods and examine of the document structure and formatting, including their performance. In § 4 we present a straight- punctuation and line boundaries in many cases, were forward ensemble method for combining the basic discarded before the data was made available, which extraction methods and show an improvement in affects the quality of the documents with respect to performance over each of the component extractors. NER. Furthermore, in parts of some of the docu- Finally, we conclude and discuss future work ( § 5). ments, the original token ordering was not preserved consistently: in some cases this was caused by the OCR engine being unable to first separate columns into distinct sections of text, while in other cases this was more likely caused by the noisiness and poor thresholding (binarization) of the image. The quality of the original images on which OCR was performed varied greatly. Consequently, this corpus represents a very noisy and diverse setting for extracting information. competing The Three data OCR used NER engines as input techniques were to our used on named in the the same entity production OCR recogniz- cor- of gine Our task that is was the not extraction identified of by the the full corpus names owner. of people, pus. ers the is data the used OCR in output this study. for 12 titles PrimeOCR, spanning a commer- a diverse e.g., Examples “Mrs Herschel of images Williams”, and corresponding from OCRed OCR docu- out- range cial In voting starting of printed system such historical utilizing a project, documents six we OCR had engines, with several relevance selects ques- put ments. are given Since in the figures corpus 1 and was 2. not Figure 1 originally is an intended exam- tions: to the genealogy best What results variation and from family those of word history engines error research. (PrimeRecogni- rate (WER) These can ple as a of public one of benchmark the poorer quality for NER, images the and pages accom- used be documents tion, expected 2009). are over Abby described multiple is a version OCR in table 1. of engines Abby and To FineReader the types best of panying for development OCR output. test and Causes blind test of poor data were quality hand- are documents? of used our within knowledge, an What OCR this level engine collection of produced NER quality has by greater Kofax is achiev- vari- (Ko- dark annotated splotches for the in current the noisy project. image One and to the two fact pages that able ety fax, in 2009). in formatting a couple The of newspapers and months genre of were development than any OCRed other time, by image- an par- en- the from OCR each engine document failed were to recognize annotated column for each bound- of the ticularly and-text when corpus no used annotated in a published data is available NER experi- for the aries development during zoning. test and In blind figure 2, test letter-spacing sets. The annota- is in- corpus ment. for The training data includes or evaluation unstructured purposes? text How (full correctly tions consisted interpreted of marking by the OCR person engine, names, resulting including in well sentences), can we structured do on a (tabular) truly noisy text and including diverse long cor- the titles. introduction The number of superfluous of names annotated spaces within in the words. blind pus lists of of OCR names data? and end-of-book How do competing indexes, and extraction multi- This test set figure is given also in illustrates table 1 for the each common document problem in the of approaches column formatted compare text over from different the books document and newspa- types? words corpus. that Blind are split test and pages hyphenated were not inspected at line bound- dur- Can pers. improvements in extraction quality be gained aries ing the as development well as other of types the of extraction errors. systems. All by combining the strengths of different extractors? extractors, This particular including collection ensembles, of were OCR applied documents to the We provide answers to these questions in the fol- were originally intended to be indexed for keyword lowing sections. In § 2 we describe the data we used search. Because the search application requires as well as the names extracted. In § 3 we present no more than a bag-of-words representation, much each of the basic extraction methods and examine of the document structure and formatting, including their performance. In § 4 we present a straight- punctuation and line boundaries in many cases, were forward ensemble method for combining the basic discarded before the data was made available, which extraction methods and show an improvement in affects the quality of the documents with respect to performance over each of the component extractors. NER. Furthermore, in parts of some of the docu- Finally, we conclude and discuss future work ( § 5). ments, the original token ordering was not preserved consistently: in some cases this was caused by the OCR engine being unable to first separate columns into distinct sections of text, while in other cases this was more likely caused by the noisiness and poor thresholding (binarization) of the image. The quality of the original images on which OCR was performed varied greatly. Consequently, this corpus represents a very noisy and diverse setting for extracting information. Three OCR engines were used in the production of the data used in this study. PrimeOCR, a commer- cial voting system utilizing six OCR engines, selects the best results from those engines (PrimeRecognition, 2009). Abby is a version of Abby FineReader used within an OCR engine produced by Kofax (Kofax, 2009). The newspapers were OCRed by an en- Our task is the extraction of the full names of people, e.g., “Mrs Herschel Williams”, from OCRed documents. Since the corpus was not originally intended as a public benchmark for NER, the pages used for development test and blind test data were hand- annotated for the current project. One to two pages from each document were annotated for each of the development test and blind test sets. The annotations consisted of marking person names, including titles. The number of names annotated in the blind test set is given in table 1 for each document in the corpus. Blind test pages were not inspected during the development of the extraction systems. All extractors, including ensembles, were applied to the We built four person name recognizers while explor- ing possible adaptations of existing named entity recognition methods to the genre and especially the noisiness of our corpus. The work required a couple of months, with each extractor being built by a different researcher. The extractors are designated as dictionary-based, regular expression based, MEMM (maximum-entropy Markov model) and CRF (conditional random field). In these four extractors, we are comparing solutions from two competing disciplines for NER: the hand-written, rule-based approach and the supervised machine learning approach. We applied them individually and collectively (within the ensemble extractors) on the blind test data and report a summary of their results in figures 3 and 5 (coarse metrics), and 4 and 6 (fine metrics). Only the results for the coarse-grained ensembles are reported in these four figures. same Precision, pages. recall When and variations F-measure in scores individual were systems calcu- above, We The built dictionary the four extractor person extractor name that recognizes is recognizers a simple only extractor, while “William explor- in- were lated considered, for person names the options in both which a coarse performed and best fine Herschel” ing tended possible as as a baseline a adaptations full name and will of requiring have existing two about true named positives, 20 entity to 30 on manner. development The coarse-grained test ...

Context 2

... for the benefit of many ap- al. uses hand-written rules on two kinds of British plications. Perhaps most importantly, IE from un- parliamentary proceedings (2008). Earlier work structured documents allows us to go beyond now- by Miller et al. (2000) uses an HMM extractor on traditional keyword search and enables semantic matched conditions: for their OCR task, they printed search. Semantic search allows a user to search digital documents and scanned and OCRed the re- specifically for only those instances of an ambigu- sulting copy to produce the OCR data for both train- ous name that belong to a semantic type such as per- ing and test sets. To our knowledge, no published son and to exclude instances of other entity types. research targets the full extent of noisiness and di- By extracting information from noisy OCR data we versity present in some real corpora or compares aim to broaden the impact of IE technology to include printed documents that are otherwise inacces- sible to digital tools. In particular, we are interested in books, newspapers, typed manuscripts, printed records, and other printed documents important for genealogy, family history and other historical research. The specific task we target in the present study is the extraction of person names from a variety of types and formats of historical OCR documents. This task is an example of named entity recognition (NER) as described in (Nadeau and Sekine, 2007) and (Ratinov and Roth, 2009). Accurately and effi- ciently identifying names in noisy OCR documents containing many OCR errors presents a challenge beyond standard NER and requires adapting existing techniques or tools. Our applications of interest are search and machine-assisted browsing of document collections. Search requires names to be pre- identified and indexed. Machine-assisted browsing of document collections has greater tolerance for misidentified names. There has been little published research on named entity extraction from noisy OCR data, but interest in this field is growing. Recent work by Grover et al. uses hand-written rules on two kinds of British parliamentary proceedings (2008). Earlier work by Miller et al. (2000) uses an HMM extractor on matched conditions: for their OCR task, they printed digital documents and scanned and OCRed the resulting copy to produce the OCR data for both training and test sets. To our knowledge, no published research targets the full extent of noisiness and di- versity present in some real corpora or compares The data used as input to our named entity recognizers is the OCR output for 12 titles spanning a diverse range of printed historical documents with relevance to genealogy and family history research. These documents are described in table 1. To the best of our knowledge, this collection has greater variety in formatting and genre than any other image- and-text corpus used in a published NER experi- ment. The data includes unstructured text (full sentences), structured (tabular) text including long lists of names and end-of-book indexes, and multi- column formatted text from the books and newspapers. competing The Three data OCR used NER engines as input techniques were to our used on named in the the same entity production OCR recogniz- cor- of gine Our task that is was the not extraction identified of by the the full corpus names owner. of people, pus. ers the is data the used OCR in output this study. for 12 titles PrimeOCR, spanning a commer- a diverse e.g., Examples “Mrs Herschel of images Williams”, and corresponding from OCRed OCR docu- out- range cial In voting starting of printed system such historical utilizing a project, documents six we OCR had engines, with several relevance selects ques- put ments. are given Since in the figures corpus 1 and was 2. not Figure 1 originally is an intended exam- tions: to the genealogy best What results variation and from family those of word history engines error research. (PrimeRecogni- rate (WER) These can ple as a of public one of benchmark the poorer quality for NER, images the and pages accom- used be documents tion, expected 2009). are over Abby described multiple is a version OCR in table 1. of engines Abby and To FineReader the types best of panying for development OCR output. test and Causes blind test of poor data were quality hand- are documents? of used our within knowledge, an What OCR this level engine collection of produced NER quality has by greater Kofax is achiev- vari- (Ko- dark annotated splotches for the in current the noisy project. image One and to the two fact pages that able ety fax, in 2009). in formatting a couple The of newspapers and months genre of were development than any OCRed other time, by image- an par- en- the from OCR each engine document failed were to recognize annotated column for each bound- of the ticularly and-text when corpus no used annotated in a published data is available NER experi- for the aries development during zoning. test and In blind figure 2, test letter-spacing sets. The annota- is in- corpus ment. for The training data includes or evaluation unstructured purposes? text How (full correctly tions consisted interpreted of marking by the OCR person engine, names, resulting including in well sentences), can we structured do on a (tabular) truly noisy text and including diverse long cor- the titles. introduction The number of superfluous of names annotated spaces within in the words. blind pus lists of of OCR names data? and end-of-book How do competing indexes, and extraction multi- This test set figure is given also in illustrates table 1 for the each common document problem in the of approaches column formatted compare text over from different the books document and newspa- types? words corpus. that Blind are split test and pages hyphenated were not inspected at line bound- dur- Can pers. improvements in extraction quality be gained aries ing the as development well as other of types the of extraction errors. systems. All by combining the strengths of different extractors? extractors, This particular including collection ensembles, of were OCR applied documents to the We provide answers to these questions in the fol- were originally intended to be indexed for keyword lowing sections. In § 2 we describe the data we used search. Because the search application requires as well as the names extracted. In § 3 we present no more than a bag-of-words representation, much each of the basic extraction methods and examine of the document structure and formatting, including their performance. In § 4 we present a straight- punctuation and line boundaries in many cases, were forward ensemble method for combining the basic discarded before the data was made available, which extraction methods and show an improvement in affects the quality of the documents with respect to performance over each of the component extractors. NER. Furthermore, in parts of some of the docu- Finally, we conclude and discuss future work ( § 5). ments, the original token ordering was not preserved consistently: in some cases this was caused by the OCR engine being unable to first separate columns into distinct sections of text, while in other cases this was more likely caused by the noisiness and poor thresholding (binarization) of the image. The quality of the original images on which OCR was performed varied greatly. Consequently, this corpus represents a very noisy and diverse setting for extracting information. competing The Three data OCR used NER engines as input techniques were to our used on named in the the same entity production OCR recogniz- cor- of gine Our task that is was the not extraction identified of by the the full corpus names owner. of people, pus. ers the is data the used OCR in output this study. for 12 titles PrimeOCR, spanning a commer- a diverse e.g., Examples “Mrs Herschel of images Williams”, and corresponding from OCRed OCR docu- out- range cial In voting starting of printed system such historical utilizing a project, documents six we OCR had engines, with several relevance selects ques- put ments. are given Since in the figures corpus 1 and was 2. not Figure 1 originally is an intended exam- tions: to the genealogy best What results variation and from family those of word history engines error research. (PrimeRecogni- rate (WER) These can ple as a of public one of benchmark the poorer quality for NER, images the and pages accom- used be documents tion, expected 2009). are over Abby described multiple is a version OCR in table 1. of engines Abby and To FineReader the types best of panying for development OCR output. test and Causes blind test of poor data were quality hand- are documents? of used our within knowledge, an What OCR this level engine collection of produced NER quality has by greater Kofax is achiev- vari- (Ko- dark annotated splotches for the in current the noisy project. image One and to the two fact pages that able ety fax, in 2009). in formatting a couple The of newspapers and months genre of were development than any OCRed other time, by image- an par- en- the from OCR each engine document failed were to recognize annotated column for each bound- of the ticularly and-text when corpus no used annotated in a published data is available NER experi- for the aries development during zoning. test and In blind figure 2, test letter-spacing sets. The annota- is in- corpus ment. for The training data includes or evaluation unstructured purposes? text How (full correctly tions consisted interpreted of marking by the OCR person engine, names, resulting including in well sentences), can we structured do on a (tabular) truly noisy text and including diverse long cor- the titles. introduction The number of superfluous of names annotated spaces within in the words. blind pus lists of of OCR names data? and end-of-book How do competing indexes, and extraction multi- This test set figure is given also in illustrates table 1 for the each common ...