Kosho Shudo’s research while affiliated with Fukuoka University and other places

Since Sag et al. (2002) highlighted a key problem that had been underappreciated in the past in natural language processing (NLP), namely idiosyncratic multiword expressions (MWEs) such as idioms, quasi-idioms, clichés, quasi-clichés, institutionalized phrases, proverbs and old sayings, and how to deal with them, many attempts have been made to extract these expressions from corpora and construct a lexicon of them. However, no extensive, reliable solution has yet been realized. This paper presents an overview of a comprehensive lexicon of Japanese multiword expressions (Japanese MWE Lexicon: JMWEL), which has been compiled in order to realize linguistically precise and wide-coverage natural Japanese processing systems. The JMWEL is characterized by significant notational, syntactic, and semantic diversity as well as a detailed description of the syntactic functions, structures, and flexibilities of MWEs. The lexicon contains about 111,000 header entries written in kana (phonetic characters) and their almost 820,000 variants written in kana and kanji (ideographic characters). The paper demonstrates the JMWEL's validity, supported mainly by comparing the lexicon with a large-scale Japanese N-gram frequency dataset, namely the LDC2009T08 generated by Google Inc. (Kudo and Kazawa, 2009). The present work is an attempt to provide a tentative answer for Japanese, from outside statistical empiricism, to the question posed by Church (2011): “How many multiword expressions do people know?”

It has been widely recognized that one of the most difficult and intriguing problems in natural language processing (NLP) is how to cope with idiosyncratic multiword expressions. This paper presents an overview of the comprehensive dictionary (JDMWE) of Japanese multiword expressions. The JDMWE is characterized by a large notational, syntactic, and semantic diversity of contained expressions as well as a detailed description of their syntactic functions, structures, and flexibilities. The dictionary contains about 104,000 expressions, potentially 750,000 expressions. This paper shows that the JDMWE's validity can be supported by comparing the dictionary with a large-scale Japanese N-gram frequency dataset, namely the LDC2009T08, generated by Google Inc. (Kudo et al. 2009).

Since (Sag et al. 2002) is presented, the NLP society has been aware that one of the most crucial problems in NLP is how to cope with idiosyncratic multiword expressions, which occur in authentic sentences with unexpectedly high frequency. Here, the idiosyncrasy of expression is twofold in principle; one is idiomaticity, i.e. non-compositionality of meaning and the other is the strong probabilistic boundness of word combination. Thus, many trials to extract those expressions from corpora by using mostly statistical method have been made in NLP field. However, presumably because of the difficulty with their correct extraction without human insight, no reliable, extensive resource has yet been available. Authors recognized the crucial importance of such irregular expressions in around 1970 and started to develop a machine dictionary which contains Japanese idioms, idiom-like expressions and other multiword expressions which consist of frequently co-occurring words. In this paper, we give an overview of the first version of the dictionary, namely JDMWE (Japanese Dictionary of Multi-Word Expressions). It has about 104,000 head entries and is characterized by; 1. the wide notational, syntactic and semantic variety of contained expressions, 2. the syntactic function and structure given for each entry expression and 3. the possibility of internal modification indicated for each component word of the entry expression.

We report that a proper employment of MWEs concerned enables us to put forth a tractable framework, which is based on a multiple nesting of semantic operations, for the processing of non-inferential, Non-propositional Contents (NPCs) of natural Japanese sentences. Our framework is characterized by its broad syntactic and semantic coverage, enabling us to deal with multiply composite modalities and their semantic/pragmatic similarity. Also, the relationship between indirect (Searle, 1975) and direct speech, and equations peculiar to modal logic and its family (Mally, 1926; Prior, 1967) are treated in the similarity paradigm.

Tills paper proI)oses two new methods to identify tile correct meaning of Japanese homonyms in text based on the noumvcrb co-occurrence in a SPillChOP which can be obtained easily from corpora. The first method uses the near co-occurrence data sets, which are strutted from tile above (:o-occurrence relation, to select the most femiblc word muong homonyms in the scope of r sen- renee. The second uses the fivr cooccurrence data sets, which arc strutted dynamically h'om the near co- occurrence data sets in the course of processing input sentences, to select the most feasible word among homonyms in the scope of a sequence of sentences. An experiment of kana-to-kanji(t)honogramto -ideograph) conversion has shown that the conversion is carried out at the curimy rate of 79.6% per word by the first method. This accm'acy rate of our method is 7.4% higher than that of the ordinary method based on the word currenee h'equency.

this paper, the above mentioned grammatical model for the first phase of parsing, which may be called "pseudo-morphological" phase, is shown and the experimental system developed for the verification of its validity is outlined. After showing some operational examples and the result of the experiment, we conclude that our model is quite effective for Japanese language processing from the standpoints mentioned above

this paper, we report the results of our Kana-to-Kanji conversion experiments which embody the homophone processing based on large scale collocation data. It is shown that approximately 135,000 collocations yield 9.1% raise of the conversion accuracy compared with the prototype system which has no collocation data

It is important for future NLP systems to formulate the semantic equivalence (and more generally, the semantic similarity) of natural language expressions. In particular, paraphrasing, full text information retrieval, example-based MT and document compression technology require the effective equivalence criterion for linguistic expressions. In this paper, first, we discuss the meaning of Japanese sentence-final modality expressions (ME) and second, present equivalence rules for paraphrasing a string of MEs while preserving its meaning.

Word processors or computers used in Japan employ Japanese input method through keyboard stroke combined with Kana (phonetic) character to Kanji (ideographic, Chinese) character conversion technology. The key factor of Kana-to-Kanji conversion technology is how to raise the accuracy of the conversion through the homophone processing, since we have so many homophonic Kanjis. In this paper, we report the results of our Kana-to-Kanji conversion experiments which embody the homophone processing based on large scale collocation data. It is shown that approximately 135,000 collocations yield 9.1 % raise of the conversion accuracy compared with the prototype system which has no collocation data.

This paper proposes two new methods to identify the correct meaning of Japanese homonyms in text based on the noun-verb co-occurrence in a sentence which can be obtained easily from corpora. The first method uses the (phonogram-to-ideograph) conversion has shown that the conversion is carried out at the accuracy rate of 79.6% per word by the first method. This accuracy rate of our method is 7.4% higher than that of the ordinary method based on the word occurrence frequency.