Corpus Linguistics - Science topic

Probably, such tools like GATE (General Architecture for Text Engineering) and NLTK (Natural Language Toolkit) can be used to annotate and analyze text corpora, allowing for deeper semantic analysis. They can be adapted to work across languages depending on the data available. Or FrameNet , a linguistic resource that provides a structured inventory of meaning or semantics for words and sentences in different languages. EuroWordNet for European languages also may be of use.

Rafal Rzepka Hi there! This link is down, do you know how to access it now? Thank you so much!

Hello, I also curious to know about corpora in the sketch engine. I think we can add data from our language to that machine. If other persons who create the data on that machine by using our language, we can use the data to make comparisons with other languages.

Thank you very much, Illia, for sending me the figure. I guess the frequency of the collocate is given via the saturation of the colour of the spot, whereas the distance from the node indicates the real distance of the collocate from the node in a text (your span is -5/+5).

If I were you, I would go for collocation analysis, choosing such lemmas as “Christian”, “faith”, etc., as collocation heads. It is also useful to utilize a large corpus, which usually yields more interesting results.

I intend to use open-ended questions and target linguistics students, although I haven't yet begun to formulate the questions.

Kerry Sluchinski You might try the following user-friendly POS taggers/segmenters for Korean language data:

1. KoNLPy: KoNLPy is a Python module for Korean natural language processing. It features a POS tagger as well as numerous tools for Korean language processing. KoNLPy is straightforward and well-documented.

2. KOMORAN: KOMORAN is a Korean morphological analyzer and POS tagger that is free source. It is available as a command-line utility and as a Java library. For testing reasons, KOMORAN offers a user-friendly online interface.

3. Hannanum is a Korean morphological analyzer and POS tagger. It is a Java library that is built on a dictionary-based approach. Hannanum is simple to use and provides a user-friendly online interface for testing.

4. Korean Parser: Korean Parser is a dependency parser and part-of-speech tagger for Korean. It is written in Python and may be used as either a command-line utility or a Python library. Korean Parser is straightforward and well-documented.

5. Lingua-STS: Lingua-STS is a web-based tool for processing Korean language. It features a POS tagger as well as numerous tools for Korean language processing. Lingua-STS is simple to use and features an intuitive online interface.

These tools are all simple to use and may be used to separate Korean text data and conduct POS tagging.

Sajal Das Take a look:

1.

2. https://ieeexplore.ieee.org/document/9225565

3. https://github.com/shaoncsecu/BanglaEmotion

kindly refer this link:

There is No need to thin the bigger sample. The test can be conducted on samples of different size.

Check my paper here on creative writing assisted by corpora.

I am building an app to help with some of these problems, for example we added a cleaner (it's called not very accurately 'remove custom stopwords') where you can input the words you want removed. You can try the app here, it's still in beta, would love your feedback: https://sagetextipocapp.azurewebsites.net/

Did you try Python word count?

Of course, the way how an issue is presented by the media powerfully reacts on the cultural value people attribute to the self-analysis of its particular people section. Pre-labeled political parties or groups may assume a diametrical opposite value whether classifified as "right" or "left" for instance – as the recent debate on 6th Jan "assault" to Capitol Hill has been presented.

Work of freaks/or politically organized right wing expression ?

Dear Yufeng,

Translating is not only problem solving. And definitely cognitive experience is not only problem solving. Some product analysis results may lead to some very modest conclusions that can be safely assumed to unequivocally point to cognitive processing and phenomena. Analogically, consider what you can learn about what goes on in the mind of a shoemaker or a luthier by studying his shoes or her violins.

You may check Staples (2015), the title is: The Discourse of Nurse-Patient Interactions: Contrasting the Communicative Styles of U.S. and International Nurses

Interesting question: would you please elaborate on your NLP task so I can provide more suggestions?

In general, you need to think about (1) corpus size should be large enough (2) representativenss of the corpus design (3) use machine-readable format.

I would suggest using Standford CoreNLP to annotate your texts (corpus) with POS tags, and I believe this computational package can have different scripts for different languages. Then, extract the words with verb tags. Let me know if there is any question.

Abdelkader Mohamed Elsayed

Maybe Anthony Lawrence's Antconc. You can google and download easily.

You can read my 2 papers on citizenship education, I may help you with tools for a quantitative research, may something compared

Text to speech system, Shatha

In Corpus design, the "saturation corpus" is associated with the concept of "representativeness", developed by Douglas Biber: <http://otipl.philol.msu.ru/media/biber930.pdf>.

Here are some other sources from the University of Lancaster that might interest you:

1. Representativeness, balance and sampling:<https://www.lancaster.ac.uk/fass/projects/corpus/ZJU/xCBLS/chapters/A02.pdf>.

2. Corpus representativeness and balance:<https://www.lancaster.ac.uk/fass/projects/corpus/ZJU/xCBLS/chapters/B01.pdf>.

... Methods: e.g. a short paper from the University of Birmingham :

3. Exploring Methods for Evaluating Corpus Representativeness : <>https://www.birmingham.ac.uk/Documents/college-artslaw/corpus/conference-archives/2017/general/paper277.pdf.

4. A quantitative approach to corpus representativeness: <http://www.lexytrad.es/assets/cl24_0.pdf>

This is really interesting.

Applied linguistics includes teaching languages' the fist language & the second language', different kinds of written and spoken texts (corpus linguistics), style, sociolinguistics, compiling dictionaries, machine translation, language and brain 'neurolinguistics', language disorders.

Good luck

Hi,

you might want to take a look at the following link. It is a King James Version New Testament, parsed with the Charniak parser into Penn Treebank format:

Please note the caveats in the accompanying README. In particular, no cleanup or hand-correction has been made to the data.

Hope this helps.

Best wishes,

Dr. Ulrik Sandborg-Petersen

Denmark

Perhaps you are referring to the way that corpus linguistics tools enable us to track the "typical" ways in which words are combined? This sheds a lot of light on "priming", for example, which would probably link up with a (fairly) behaviouristic notion of human language, even though, as we all know, these "habits" are only part of the story. The language we produce is not just the outcome of habit!!

Hi Judit,

Thanks so much for your feedback. This was a conference paper that I worked on a bit and had published. Now, I am looking to do a larger project along the same lines.

MICASE (the Michigan Corpus of Academic Spoken English) has been open access since completion. No registration required!

In my opinion, analysing determiners in English corpora can be tricky.

One problem is that you have classes of determiners which have identical pronominal forms (possessive determiners/pronouns, demonstrative determiners/pronouns). You can minimize this problem by using part-of-speech tagged corpora. Many corpora today come in a part-of-speech tagged version already. By using part-of-speech tags, you can specify whether you are looking for a pronominal form, a determiner form, or both, when looking for a word form such as 'this'.

Second, English has noun phrases where an overt determiner is required, and others where the determiner is either optional or not permitted (as per the standard). NPs without an overt article form (sometimes called zero article) can only be found automatically when the corpus is annotated for syntactic structure. A corpus that is annotated for syntactic structure is referred to as a 'parsed' corpus, and they are quite rare (due to the high error rate involved in automatic parsing). However, using tools such as the UAM CorpusTool, you can parse a corpus yourself.

Finally, you mentioned that you are looking for the use of determiners in the context of emotions such as 'anger'. I fully agree with the suggestions above: You can look for determiners that contain certain words in their immediate context. In corpus linguistics, this is often called 'collocation'. Using a collocation approach, you could compile a list of nouns (or verbs, adjectives) that express the notion of anger, and then ask the corpus software to only give you those instances of 'this' which has one of the words from your lists in its immediate context (e.g. within a range of 9 words to the right or the left). Some corpora are even tagged for synonyms (e.g. the corpora on the popular corpus.byu.edu platform), so that you could search for 'this' and any synonym of 'anger' in its immediate context.

I apologize for the somewhat lengthy response, and hope it was helpful (I explained some corpus-linguistic terminology because I don't know your background). Feel free to ask a follow-up.

Dear Gina,

Here you have a comprehensive list of Speech analysis and transcription tools for several languages.

I hope this helps.

Kind regards,

Begoña

Dear Adrian,

That sounds quite interesting.  I've had a look at your link, and I'll definitely be contacting you soon with some questions about the corpus structure.

Thanks,

Daniel

I agree that you should go beyond saying that your results are "slightly different." In particular, I would recommend calculating an inter-rater reliability index such as Krippendorff"s alpha.

You might also consider whether doing this kind of re-rating is really necessary for your work. In some fields, such as communication studies, inter-rater reliability is almost a requirement in you are doing content analysis on media, but in other fields where qualitative research is more interpretive, it is not considered to be useful.

Dear Sadik,

Sketch Engine contains corpora for Arabic, Punjabi, Persian, Urdu and Malay.  You can prepare corpora for the remaining languages yourself using the WebBootCAT functionality in Sketch Engine. In case of any questions, just ask the support team of Sketch Engine at support@sketchengine.co.uk (they could also do that job for you, but there would be some costs involved).

Best regards,

Milos Jakubicek

Hi Rebeka, DAG is a variant ( form) of a syntax tree which gives direction to it. There are nothing such as other variants. This much I can suggest you as from your question it is not clear what exactly you are looking for.

Hi,

Hope the attached links will be able to assist you.

Regards

http://opus.lingfil.uu.se/ is a nice resource of parallel corpora for many language pairs.

You have 13.4M tokens for (en->hi).  Not much for Marathi though..(en->mr 6.2M)

See MADAMIRA :

Hi Sara,

You may need to consider using the "arabicFactored" as a Stanford parser model of "POStagger".

HTH.

Samer

(My answer assumes that you are referring to computer corpora of literary texts. I hope I have not misunderstood you.)

I am not familiar with specific literary corpora: I know they exist, but I do not know their size or composition. However, you might do well to bear in mind that a number of the main general corpora, including the OEC, the BNC and the COCA have a literature subcorpus.

Whether you would be better off with a smaller specific literature corpus or a larger general corpus which includes literary texts will depend on the nature of your research. If you are looking above all for a large amount of texts, then you might be better off with a mega corpus such as the OEC or the Coca: even though literature is not the mainstay of their composition, the sheer overall size of these two corpora (around 2 billion words in the case of the OEC and around half a billion in the case of Coca) will ensure that you are not short of literary texts. However, you would need to examine the composition of the literature subcorpus in these corpora, to see if it fits your purposes. If you require more information than that supplied with the corpus, you could always try contacting the corpus compilers.

The BNC is available on both SketchEngine and the Brigham Young University platform. You would need to check if it's the same version on both platforms, as it wasn't at one time. The OEC is also available on Sketch Engine, if you obtain prior permission to use it from the publishers (Oxford University Press.) Coca is available on the Brigham Young University platform.

no. no so definitely that it isn't worth trying to find one.

as for Ethnologue specifically, they are very strongly biased toward considering what many people would consider as dialects as distinct languages. Ethnologue is also part of SIL, so if you're looking for information on languages spoken in different places, SIL will give you the link to Ethnologue.

and the CIA Factbook lists Spanish as the only language spoken in Colombia. that's enough to render it unreliable. then there's the United States. even if you want to accept what it says, which is not a good idea, data at the level of "other Indo-European languages" and "Asian and Pacific Island" isn't what you seem to be looking for.

also, no one knows how many languages (and we are pretending that we have a definition of language right now) are spoken in the regions of the world that are geographically characterized with very dense vegetation or mountains, or both. these would include, at the minimum, the Amazon-Orinoco-Ucayali-Xingu river system in South America, the Congo River system with its tributaries and the mountainous area in the Great Lakes region adjacent to it, the inner region of southeast Asia, and New Guinea and the Solomon Islands.

Dear Amaal,

I would argue that "the bigger the better" is one of the key factors for the reference corpus when using it to extract keywords. In Sketch Engine you can try a number of big English corpora as reference corpora for keyword extraction, among others also the English Feed Corpus, which is a diachronic corpus of news feeds. You can also choose a part of this corpus (e.g. a single year) to be used as the reference data.

Best regards,

Milos Jakubicek

For COCA simply separate your list items by vertical bars as in 'this|is|my|list' and copy-paste it into the form.

Studying language shift through documented language use is obviously better than questionnaires etc, but it presupposes some language tools that may not be available for all languages. I am speaking of general, multipurpose language tools here.

Ideally, you need (1) a corpus covering the period you wish to study (from before to after), (2) a digitised speller, i.e. some kind of full form generator documenting the standard. You can then run the full form generator against different time sections of the corpus and look at what you get, specify etc.

There are a number of difficulties which will need to be dealt with, the first one being the size of the corpus. For Norwegian we have both the full form generator and a corpus of 100 million + tokens, covering the period 1866-2015. But the full form generator would be useless on the text from before 1940, because the orthography before 1940 is too variable, and the text mass too small.

Then you should ideally have comparable text selections from the different time periods. But our corpus has virtually no newspaper text from before 1998, while newspaper text dominates after 1998. This is because early newspaper text has to be keyed in manually, and keying costs money.

We have run the sort of comparison I talked about earlier on the post-1940 part of the corpus. We were faced with some results which I believe would turn up for most languages, i.e.

half the possible word forms from the full form generator were not found in the corpus. (Norwegian is a medium inflected, compounding language - eight possible noun forms)

half of the word forms from the full form generator that occurred in the corpus, occurred only once.

half the tokens of the corpus occurred once only.

ca 350  tokens equalled 50 % of the corpus token mass (prepositions, conjunctions etc)

After this possibly discouraging comment, I would still encourage collecting actual language as a corrective to questionnaires etc. Ideally, such materials should be formatted and saved as part of a larger text corpus.

 For UK politics you could try using Hansard. Also some articles by Professor Stef Slembrouck (early 1990s) might be of interest.

The really strange thing is that the word you are discussing is neither a proper noun (NNP) nor a cardinal number (CD). It's a closed class preposition, which neither the parser nor the tagger should be getting wrong.

Actually, I believe you have a different problem: From your image I see the model you have selected for the parser is arabicFactoredBuckwalter.ser.gz. This model is probably expecting Arabic in Buckwalter transliteration using Latin characters, see:

The input you are giving to the parser is in actual Arabic characters, which probably never appear in the data it is trained on. I think you need to convert your data to romanized Buckwalter style transliteration before you can use the parser. Alternatively I think there is also a model for native Arabic characters, called arabicFactored.ser.gz (see this guide: http://nlp.stanford.edu/software/parser-arabic-faq.shtml).

I would differentiate between grammar and semantics. From one side, I doubt that Infinitival Clauses or word order as syntactic elements play an important role in variation and change since grammatical rules are very specific for every concrete language.

From the other side, I would study their semantic meaning to get rid of language-dependence. For instance, in the emotion domain repetition of words must be studied to localize "hot spots". Below is an abstract from my PhD thesis (p.21):

Similarly, Infinitival Clauses or word order markers could be used as identification anchors to localize variation and change since they identify conspicuous points through the syntactic structure. I would assume there are also more grammatical elements in a language to be investigated comprehensively. Not only Infinitival Clauses that do not "distract" by some stylistic variation.

I agree with Joel Walters that using context is the best way to teach conjunctions besides asking learners to use in real dialogues inside class or outside to be sure that they use them appropriately and give feedback about that to their colleges

Corpus linguistics is a methodology/approach rather than a theory. A corpus can support or provide evidence for your theories on address forms.

Depending on the data in the corpus you are using, you can look for frequency patterns in the use of terms of address according to the variables of age, gender, institutional role, language proficiency, formality/informality of discourse situation, etc. However, the corpus you are using (or constructing) has to have this information available to you in the form of metadata, otherwise you will have no way of linking the tokens you find in the corpus to such factors.

Multimodality can be used to build inventories of the semiotic resources, organizing principles, and cultural references that modes make available to people in particular places and times: the actions, materials and artifacts people communicate with. This has included contributions to mapping the semiotic resources of visual communication and colour, gesture and movement, gaze, voice and music, to name a few.

Multimodal studies have also been conducted that set out to understand how semiotic resources are used to articulate discourses across a variety of contexts and media for instance school, workplaces, online environments, textbooks and advertisements. The relationships across and between modes in multimodal texts and interaction are a central area of multimodal research.

Multimodal research makes a significant contribution to research methods for the collection and analysis of digital data and environments within social research. It provides novel methods for the collection and analysis of types of visual data, video data and innovative methods of multimodal transcription and digital data management.

Four core concepts are common across multimodal research: mode, semiotic resource, modal affordance and inter-semiotic relations. Within social semiotics, a mode is understood as an outcome of the cultural shaping of a material through its use in the daily social interaction of people. The semiotic resources of a mode come to display regularities through the ways in which people use them and can be thought of as the connection between representational resources and what people do with them. The term modal affordance refers to the material and the cultural aspects of modes: what it is possible to express and represent easily with a mode. It is a concept connected to both the material as well as the cultural and social historical use of a mode. Modal affordance raises the question of what a mode is ‘best’ for what. This raises the concept of inter-semiotic relationships, and how modes are configured in particular contexts. These four concepts provide the starting point for multimodal analysis.

No; and one of the (big) problems is that the notion of 'semantics' in the context of formal systems does not have an unambiguous semantics [sic!].

Best,

Hans

have a look here: https://www.lt.informatik.tu-darmstadt.de/en/software/lexical-chains/

Biber, D. (1990). Methodological issues regarding corpus-based analyses of linguistic variation. Literary and linguistic computing, 5(4), 257-269.

For the third article, you may request full text from the author.  Here's the link.

I've started working on AntConc only recently; it helps you to view the files. However, it is not very good for extracting words or searching for collocations. Laurence Anthony (the designer) promised to develop a new version (4.0)  that can help you process Arabic texts without messing things up. Meanwhile, I'm working on Wordsmith, which is not free unfortunately. All the best!

I am quite conversant with content analysis and have been able to use it to analysis newspaper advertisements and Universities marketing communication materials ( websites and prospectus).

Content Analysis of UK Newspaper - http://www.emeraldinsight.com/action/doSearch?ContribStored=Mogaji%2C+E

Analysis of UK Universities websites and prospectus - https://www.researchgate.net/publication/269693306_Why_isn%27t_my_professor_a_black_woman_A_clue_to_gender_inequality_in_higher_education_from_websites_and_prospectuses

Analysis of Magazine Front Covers - https://www.researchgate.net/publication/275155323_Portraying_the_Female_Gender_in_Sports_Progression_or_Regression

Hope you find them useful, let me know if you want to discuss further.

 I think Chris Turner has given some excellent pointers above. You might also find the following article interesting: Bahns, J. and Eldaw, M. (1993). Should we teach EFL students collocations? System 21(1): 101-114.

Critical discourse analysis often focuses on gender-see works by Lazar or Wodak. For applications of corpus research within critical discourse analysis see works by Mautner. However, critical discourse analysts have sometimes been accused of adopting a priorist stances regarding power abuses in language, which may adulterate their linguistic analyses,  and blind them to interpretations which do not fit with their point of departure. For a discussion of this problem, related to CDA per se rather than to gender analysis within this school, see Section 3 of Controversies in Applied Linguistics, edited by Seidlhofer.

Moving outside discourse analysis per se, there may be some mileage in using sentiment analysis to discover differences in the attitudes portrayed by male and female writers of fiction, and/or in the way that men and women are portrayed in fiction. You could explore the literature available on the use of corpora within sentiment analysis. I am not familiar with this area, but a quick Google search suggests that there are a number of studies available.

It might also be worth your while to carry out an analysis of fictional dialogues within Conversational Analysis. For one possible framework of Conversation Analysis, see  English Conversation by Amy Tsui.

As you are no doubt aware, many large corpora such as the Oxford English Corpus or the BNC allow you to carry out searches within the specific fiction subcorpus (called "imaginative" in the version of the BNC available on Sketch Engine). Both these corpora will tell you if the text has been written by a man or a woman. With this information, you could compile one specific fiction subcorpus containing texts written by male authors and another by female writers. Use the following link to help you create subcorpora: https://www.sketchengine.co.uk/xdocumentation/wiki/SkE/Help/HowTo/CreateSubcorpus.

Once you have your subcorpora, you can use many functions to establish possible differences between male and female writers, such as collocations, word sketches and key words. It might also be worth your while to explore possible differences in the way that men and women writers realize key speech functions (requesting, ordering etc) in fictional dialogues. This could be done by searching for specific strings which you feel are likely to occur in dialogues written by one particular sex or by establishing the frequency of strings that you have observed during corpus searches.

Hi Birgit,

If you're interested in L2 German and comparable native data, I can recommend the Falko corpus:

I think this is the largest freely available learner corpus of German, and it contains essays and summaries from German learners of a variety of backgrounds, as well as comparable native German texts using exactly the same prompts (same essay topics etc.). There are also extensive annotations, including Target Hypotheses giving different versions of what native annotators would have written in cases where errors occur. These can be very helpful for studying specific types of errors.

Hope this helps,

Amir

As far as I can find out only Max Planck Institute for Evolutionary Anthropology provides some information. You find it in OLAC (but I guess you have already been there):

Psycholinguist Rainer Dietrich of the Humboldt University (Berlin) has worked on the language of ATC. You might find references to corpora in his work.

Perhaps you may use AntConc to search for and to identify the pattern in your randomized samples and whole corpus, and compare the results to see whether the randomized samples are representative or not~

Hi Sorina,

I am using SALT for Spanish and English (http://www.saltsoftware.com/). I don't know what languages you need to manage . It is a very user-friendly tool. You can transcript and redefine your own lists of words (concordances) and declare your own tags ([tag]).

You can check also the CHILDES project  tools (http://childes.psy.cmu.edu/).

Hope it helps.

Good luck!

Hi Bram,

I'm an experienced Python user but don't really know much about that specific scientific discipline. In any case you have a Python library (Natural Language Toolkit) that should be able to help you a lot:

Besides this for numpy based data (arrays) you may want to look at (besides panda) scikit-learn for Machine Learning algorithms:

Matplotlib for plotting:

http://matplotlib.org/ (although there are plenty of others if you need more performance in plots)

I'm not sure what in what formats your raw data comes from but there's a lot of libraries for dealing with CSV, EXCEL formats, among others...

I would advise you to install everything with 64 bits version. Python has official 64 bit releases but some libraries don't so this link might help you in that regard:

Besides I would advise you to install a more complete package set instead one library at a time. WinPython (for windows) has an excellent installation manager (besides already coming with a lot of stuff) :

Anaconda is another distribution (64 bits) also good for Mac or Linux:

Hope it helps.

I absolutely agree with Amy! I once used AntConc 3.2.4. It's convenient and easy to use!

Larisa,

Thanks so much for your reply. I have downloaded the book and look forward to reading it.

All the best,

Erik

It is free, only that I don't know how versatile it is and what are other alternatives.

Dear Veslava,

As to the construction of a thesaurus of social humanity terms, all you need to do is delimit the semantic field that includes these terms first, and then look for their paradigmatic relations in context. Tom do so, you'd better start with the nucleus of the field. As semantic fields are gnoseological rather than ontological categorries, it is up to you and your team what criteria and indicators you'll be applying in determinig the focal or nuclear elements. Afterwords, you need to excerpt all the lexical items that match these criteria and analyze them at the level of word cobinations, sentences and sentence sequences. The delicacy of analysis depends on what your goals and objectives in constructing the thesaurus are. After you've delimited the zones of semantic density within the field, you need to start looking for less clearly related terms which can be found in other semantic fields as well. For example discrete system refers to language but also to mathematics. Such terms require great care and deliberation because you need to go through a variety of contexts in order to decide how salient, or rather, how un-salient these are with respect to the other terms in the field. Also, in constructing the thesaurus you need to decide whether you'll be including terms from neighbouring fields that have overlapping bounderioes with social humanity terms. You might find it useful to choeck on the Visual Thesaurus of the English Language. I think parts of it are available on-line. It can also give you interesting clues. Once the semantic field with all the criteria ase been designed, you can proceed with the associative relations between the terms - connotation, stylistic variants, etc.. Then the time comes for paradigmatic relations. These are best identified if based on syntagmatic ones, e.g. Which terms can replace/are opposite to/include/exlude, etc. the term X in the sentence XeP.

I'll be more than happy to share all my experience with your team. I've done the same for therms from the conceptual doamain WATER/LIQUIDS and it was quite an adventure.

I suggest head/tail breaks, if your data are heavy tailed.

Jiang B. (2015), Head/tail breaks for visualization of city structure and dynamics, Cities, 43, 69-77.

Jiang B. (2013), Head/tail breaks: A new classification scheme for data with a heavy-tailed distribution, The Professional Geographer, 65 (3), 482 – 494.

http://www.ahds.ac.uk/creating/guides/linguistic-corpora/index.htm 

You may like to have a look at the above link. The book is written by a few defining figures of corpus linguistics including Sinclair. Size is now not the paramount consideration, esp. in building of a special corpus, while representativeness (if the text collection can represent the domains/fields/registers/genres of language use you are investigating) is crucial. 

A good question. I think the best book to read about this is The Linguistics Wars by Randy Allen Harris (1993). He gives an enormous amount of detail about the arguments for and against generative semantics (GS) and comes to the conclusion on p. 241 that GS promised too much and failed to deliver: it claimed not just to handle semantics and syntax, but also pragmatics, fuzziness, logic, ... . As a result, representations were becoming more and more unwieldy and the main practitioners just seemed to abandon the ideas they had put forward, very often turning their back on generative grammar and founding cognitive grammar/linguistics, various functional approaches, etc. I don't think that their conception of syntax/semantics was ever really shown to be wrong.

I'm not sure if this is imagery or not but on the podcast Lexicon Valley, they talked about the term "fiscal cliff" (I think the episode was called "down is up" or "up is good" or something like that). They mostly talked about metaphor and language, but there were some comments about how the word sounded, with the repeated "ih" vowel and all the fricatives. 

EMILLE (Enabling Minority Language Engineering)  Beta version of the corpus consists of: 30 million words of monolingual written data (Gujarati, Tamil, Hindi, Punjabi); 600,000 words of monolingual spoken data (Hindi, Urdu, Punjabi, Bengali, Gujarati); 120,000 words of parallel data in each of (English, Hindi, Urdu, Punjabi, Bengali, Gujarati).  Website: http://www.emille.lancs.ac.uk/

Thanks for your answers. I want to compile my own corpus and study a syntactic phenomenon so I need to annotate it first. Thanks again!

I don't have a specific answer to your question, but a paragraph in the OpenNLP wiki documentation may be an indication. The Instructions to train and run a simple PoS tagger for English specify a list of 5 training sets:

I'm not into linguistics, so can give specific advice on methods, just some general observation based on general statistical principles.

How many predictors are you trying out?

What validation steps are you following?

With a binomial response you have a very high risk of overfitting and what you are describing sounds like a classic case of overfitting. Do you do any data reduction methods to reduce the number of input variables while retaining maximal explanatory information? In datasets with properties similar to what you describe I would use a multi-layered validation procedure - start off with a subset of data (75-80%) doing cross validation to identify the most promising predictors, then a validation set (the remainder) to test the performance of those predictors. Ideally you would then test that model on a new set of collected data.

hello Matej,

You can use Wordnet to get the all synonyms of a word. R, Python, and Gate tools have packages that call Wordnet services.

see http://cran.r-project.org/web/packages/wordnet/wordnet.pdf

Do you know the PROIEL treebank? http://foni.uio.no:3000/ It has a considerable amount of Greek text from several periods (the core is the New Testament, but there is also Herodotus and some Byzantine chronicles). You can download fully lemmatised and tagged texts there and use them to train a morphological tagger. 

Hi Manuel,

there are indeed some options you can choose to lemmatize German. In case you are already happy with a stemmer you might want to have a look at this part of NLTK: http://www.nltk.org/api/nltk.stem.html

If you need lemmatization you probably find something useful here if you are familir with python:

If you prefer java you might want to look at Stanfords parser:

That is to my knowledge also able to parse and lemmatize German.

Python as well as Java have API's that allow you to scrape Facebook just google for it. It is easy to find. I hope this helps you.

Cheers, Markus

There are lots of papers on this which will be a better way to inform your opinion than a small number of brief responses. Maybe if we list examples of these. I'll start with Efron at http://statweb.stanford.edu/~ckirby/brad/papers/2005BayesFreqSci.pdf which I think provides a fairly direct answer to your question from someone whose opinions about statistics are much better to listen to than mine!

Depending on what you're looking for, you might also consider PhonBank, which contains orthography and IPA transcriptions of a child's target and actual utterances. You can find a link to this on the CHILDES website (http://childes.psy.cmu.edu/phon/). These can be used with Phon (https://www.phon.ca/phontrac/wiki/Downloads), a software created for child language research and particularly useful for phonological analysis.

Thank you for sharing these references!

I'm rather from the speech area than text, so I don't know in much detail. Google might help you more. If you don't find anything, I'd suggest you take your text corpus with tagged anxiety/non-anxiety sentences and do some simple statistics/clustering of parts-of-speech that tend to appear in those two classes.

I recall this paper - http://www.communicationcache.com/uploads/1/0/8/8/10887248/automating_linguistics-based_cues_for_detecting_deception_in_text-based_asynchronous_computer-mediated_communications.pdf

It's automatic linguistic-based detection of deception attempts in text, but might be helpful for detection of anxiety as well.