Machine Translation - Science topic

I think that teaching foreign languages translation in the era of neural machine translation requires training teachers on modern trends that have entered the field of teaching foreign languages, where we do not and we lack development in this field

Machine translation (MT) has undergone quite interesting developments. However, it still has some linguistic, pragmatic, and affective issues, which require human post-editing to shun any occurrence of “lost in translation” or misinterpretation that could lead to harm. For instance, the study by Taira et al. (2021) concluded that Google Translate is not fully ready for use in clinical settings because of inaccuracies that could result in harm to patients.

Nonetheless, neural machine translation (NMT) seems promising despite being expensive and having issues in translating vague expressions. NMT is powered by artificial intelligence, i.e., improvements are constantly made to ameliorate the quality of NMT output. Here is the full citation to the article by Taira et al. (2021).

Taira, B. R., Kreger, V., Orue, A., & Diamond, L. C. (2021). A pragmatic assessment of Google Translate for emergency department instructions. Journal of General Internal Medicine. https://doi.org/10.1007/s11606-021-06666-z

Good luck,

The profession of a technical translator in the future will involve a combination of the following skills: a linguist, as well as a kind of IT analyst, constantly looking for flaws in the machine translation system in order to improve it, and a machine translation specialist, bringing to the attention of the customer the found flaws. The translators of the future will not sell words that the machine already generates faster and in a larger volume, but the authenticity of these words, the certification of the essence of a document, the confirmation that the meaning of a document is transmitted correctly, that is, the responsibility that machine translation will not be able to assume under any circumstances. Just the ability to translate from scratch, for example, a standard contract is paid less and less every day, but those who are ready to master the possibilities offered by automation systems, and become at once operators of such systems and a kind of notaries-certifiers of the results issued by such systems, will definitely not remain unclaimed.

The post-editor should be aware of the problematic elements that may be present in the translated text, i.e. he should be familiar with the features of different types of machine translation, in terms of their inherent different types of errors. Understanding the" behavior " of the machine translation system directly affects the speed of the post-editor, when the latter knows in advance what errors to expect.

I use this tool https://www.systransoft.com/ and I start trying https://www.deepl.com/translator that recommended Wolfgang R. and works great, check this out and compare. Translation by computer still not precise 100%

Sincere condolences to his family members, friends, and colleagues.

May his soul rest in peace.

One of the helpful example for underfitting/overfitting model is discussed here https://towardsdatascience.com/overfitting-vs-underfitting-a-complete-example-d05dd7e19765 and its conceptual understanding is here https://towardsdatascience.com/overfitting-vs-underfitting-a-conceptual-explanation-d94ee20ca7f9 . A comparison for its diagnosing is here : https://machinelearningmastery.com/diagnose-overfitting-underfitting-lstm-models/

Hi,

Sorry for my late update. I used LSTM that is said if it is trained on my data the model cannot be used on other domains. Otherwise, I have link in the problem description. Have you checked the link?

Dear Sir,

You asked a very relevant question. In a recent development, everything is changing by adopting AI techniques but now the time to adopt AI in Learning-Teaching to go one step ahead from traditional Teaching-Learning. For example collaborative learning(CL), personalized learning(PL) and Flip Learning (F-L) can be applied for Teaching Translation(or Transformation) by adopting AI-based learning tools.

How latent variable is defined for Non auto-regenerative model?

Any suggestion is appreciated.

I don't think so because of the polysemous words. One English word may have so many meanings. The machine is still unable to choose the right meaning in the right context.

Which library do you use for the deep-learning?

tensorflow? or pytorch?

If you use tensorflow which is the one of the common libraries, just reinstall the tensorflow which is based on GPU.

(command: "pip install tensorflow-gpu")

Of course, there are few more install steps for using GPU.

Please, check the referred video: "https://www.youtube.com/watch?v=HExRhnO5Mqs"

If you correctly install the programs, you do not need to change your python source code for using GPU.

I hope that my answer help to your works.

Sorry, I have not been clear.

The translation were not done with the idea of creating a new sentence for each original sentence. For this reason, one sentence in English can correspond to multiple sentences in another language or vice-versa.

newspaper articles

I agree with Aida that sounds produced by infants are nonlinguistic ones. Therefore, they are meaningless. Please share us your reserach objective to be more helpful.

You are not giving enough information. Tell us what language(s) you are translating from and what language(s) you are translating them into.

To some degree, it depends on the target language (the language you are translating into). BLEU has the structure of English and similar languages in its DNA. "Similar" means languages where word order (rather than word morphology) is central. In particular, BLEU is built around n-gram co-occurence statistics. For languages that are very different from English, METEOR may be more useful than BLEU. For non indo-european languages (Chinese, Arabic, etc. etc.), I would say that assessing translation quality is an open question and that BLEU will not necessarily be useful. One possiblility is to annotate the low-level concepts (nouns, verbs, adjectives, adverbs, quantifiers, and important prepositions) in the source language and then have bilingual judges tell you whether each such concept is present in the target-language translation.

Perhaps some of the papers that have me as an author will be helpful to you (in particular, look at the papers that they cite). You may also want to look into the MetricsMaTr evaluations.

Bhat, I. A., Bhat, R. A., Shrivastava, M., & Sharma, D. M. (2018). Universal Dependency Parsing for Hindi-English Code-switching. arXiv preprint arXiv:1804.05868.

A lot of this information (i.e., questions) is not contained within the manual signs, but rather in the facial expressions of the signer

Hi Musfiqur Rahman,

You can use "Human judges" method and Kappa measure for judging the quality of a parallel corpus.

Dinh

there are many journals, but I prefer for you

ACM Journal on Computing and Cultural Heritage

OR

journal of deaf studies and deaf education.

I don't think a perfect article exists. I would start with the following paper and wait for further comparisons (future publications which cite it) or start searching for reports comparing particular features. That would be a nice paper if you survey them all!

Hi,

as you are dealing with sign language, i.e. image data, I wonder if we are not dealing with two challenges here:

Afaik, lamtram does not accept images to this date, so maybe you should rather look into image classification frameworks.

P.S.: Just as I am writing this, I come to realize how this is making your project more complex (and more innovative, in a positive line), just as it is mire complex to analyze a Youtube video as opposed to a classic newspaper article.

Hope there are some experts who have already trained a computer to accommodate sign language.

Best regards,

Christiane

Actually, it depends on language, data and objectives of the study.

Dear Dr. Pakary,

Please try the following links and see which of them serves  your purpose.

Best regards,

R. Biria

Dear Asma Djaidri,

Thank you for the interesting question. As you have rightly observed, WSD plays a pivotal role in machine translation because words usually have multiple senses in a given context. Notably, in human translation, the translator enjoys reflective, neurologic mechanisms for creating a relevant translation. However, in machine translation, there is need for intelligent computational techniques. As an illustration, particular machine learning software must be programmed in which  a classifier is trained to choose a particular sense from among the existing senses. Naturally, this is possible by carefully preparing a corpus of  sense-annotated instances defining a specific word. For more details, I refer you to the following links, which can hopefully satisfy the question.

Best regards,

R,. Biria

Hi Ikram,

Yes, this can be correct; as it doesn't form any assumptions about characteristics of the concepts. In addition, KNN learning cost is very low especially when the data is small. Finally, complex aspects can be learned using simple procedures.

HTH.

Samer

i am working on sanskrit-hindi translation but i have no parallel data of sanskrit-hindi.

The dcase is indeed a good reference

depending on what you are looking for you can also have a look to the

sweet-home corpora : http://sweet-home-data.imag.fr/

and

both having sounds captured in a home

See TurboParser (Dependency Parser with Linear Programming)

Inflectional paradigms are useful when you know only one form of the word and what to construct the others. It is straightforward for MT (to construct surface forms) in POS tagging and lemmatization (to predict inflection of out-of-dictionary words) and so on

It happened with me when I needed to train the malt parser which only accepts CoNLL format. I had three corpora, only one of them was in the CoNLL format and I needed to write a script to convert between the different formats. I guees you need to do that. sometimes the formats are different in the information they hold and some could be richer than others but I hope it is not your case. I hope it helps.

SRILM is quite handy and well-documented. FAQ explains how to compute ppx with ngram tool http://www.speech.sri.com/projects/srilm/manpages/srilm-faq.7.html

Another really short and handy explanation about ppx with SRILM was described here  http://cmusphinx.sourceforge.net/wiki/tutoriallm

I recommend you another excellent handbook, namely "Logic, Programming and Prolog (2ed)" by U. Nilsson and J. Maluszynski:

With kind regards,

Adam

Hi,

This is an information retrieval task. I suggest you to read the book by Manning (http://nlp.stanford.edu/IR-book/).

One of the simplest (but also efficient and effective) way is, as Yusniel already pointed out, to express both the documents and the query in the vector space model (https://en.wikipedia.org/wiki/Vector_space_model)

1) Pre-process your documents: remove stopwords and do stemming (and other processes you can find in the book)

2) Extract the vocabulary from the documents (all the remaining unique words)

3) Index the documents using the extracted vocabulary to create a term-document matrix, where terms(words) are in rows and document in columns. The most easy is to count the frequency of each word in each document: https://en.wikipedia.org/wiki/Document-term_matrix

4) Process the query with the vocabulary you extracted from the documents to form a query vector, also counting the number of times a word inside the query (usually they appear only once)

5) Compute the cosine similarity between your query vector and each one of the document vectors.  https://en.wikipedia.org/wiki/Cosine_similarity

6) Sort the similarities in descending order and select the top k documents as the more relevant for your query.

You can perform several other steps between the steps described above to improve the retrieval performance, such as extract n-grams instead of individual words, weighting the words in a different way than the frequency (e.g. term-frequency-inverse-document-frequency), vector normalization, Latent Semantic Indexing, etc.

In Python you can use the nltk package to do most of this things. Here is a webpage where they show how to do it:

Finally, there are other models different from the vector space, such as the probabilistic retrieval.

Have a look at the Stanford Parser (http://nlp.stanford.edu/software/lex-parser.shtml) and Stanford CoreNLP (http://stanfordnlp.github.io/CoreNLP/index.html).

You might also have come across the Moses MT toolkit (http://www.statmt.org/moses/) ? 

Great to hear that you solved the problem. It seems the issue was on the 3rd step of the installation.

Thanks Santanu. It's a great suggestion. I'll give it a try with n-gram model

Cheers,

Thushari

This work can definitely help you 

How about translating Hindi and English advertisements - rich in semantics and language variation. You will explore more on the cross-cultural issues too. All the best

If you are using (Moses Decoder), you can tag and supertag your corpus accurately. 

The methodology for comparing n machine learning (ML) methods over m data sets is described in (Demšar, 2006). Please see the link.

I show you an example in the attached document.

our college has a bible in albanian.  that's always a good corpus.

Dear Muhammad ,

please find  the file in the attachment.

hope it helps .

see on Moses website Factored translation models.

Dear Sandeep,

Next you'ill find a link to a set of excellent resources on Machine Learning, where you surely will be able to explore new applications and algorithms created to solve problems in this rich field of ML:

tutorial

Consider the following code

Binding of variables for use in compiling patterns:

Use Env env = TokenSequencePattern.getNewEnv() to create a new environment for binding

Bind string to attribute key (Class) lookup: env.bind("numtype", CoreAnnotations.NumericTypeAnnotation.class);

Bind patterns / strings for compiling patterns

// Bind string for later compilation using: compile("/it/ /was/ $RELDAY");

env.bind("$RELDAY", "/today|yesterday|tomorrow|tonight|tonite/");

// Bind pre-compiled patter for later compilation using: compile("/it/ /was/ $RELDAY");

env.bind("$RELDAY", TokenSequencePattern.compile(env, "/today|yesterday|tomorrow|tonight|tonite/"));

Your question is very confusing.  First you say that MaxEnt > SVM for NLP but then you say that SVM > MaxEnt.

Is there a clarification you could add to your question?

Also, if you are using linear classifiers, I would strongly recommend checking out L1 regularized logistic regression.

If you are using kernel methods to get non-linear classifiers, check out a recent implementation of neural networks with drop-out regulariztion, rectified linear units and accelerated learning like Adagrad.  You should get superior result versus your SVM or MaxEnt result.

Despite the massive researches in MT, MT still immature yet, that is because a high quality translation is based on full understanding of speech/text, understanding speech is beyond machine's capability. however, for a simple check to get some confidence in the output of an MT software you may do:

1- use Mt to translate your text from source to destination language.

2- use the output text to be translated back to your source language

3- compare your original text with the output from step 2

4- the more similarity you get the higher confidence will be.

Actually, there is a Massive Open On-line Course on Machine Translation entitled: "Approaches to Machine Translation: rule-based, statistical and hybrid".

It could be interesting.

For any information about it, visit:

Probably is the lack of correct classification: the same word means different things for different fields, so machines fail. Mayby they should offer an option for declaring the field from the user.

A simplified definition of a token in NLP is as follows: A token is a string of contiguous characters between two spaces, or between a space and punctuation marks. A token can also be an integer, real, or a number with a colon (time, for example: 2:00). All other symbols are tokens themselves except apostrophes and quotation marks in a word (with no space), which in many cases symbolize acronyms or citations. A token can present a single word or a group of words (in morphologically rich languages such as Hebrew) as the following token "ולאחי" (VeLeAhi) that includes 4 words "And to my brother".

A stirng as written by one of the previous researchers who responded

is a oncept taken from programming languages.

The following are the translation services that support Albanian language :

1. http://translate.yandex.com/

Yandex even exposes a free API with the help of which you can build an automated translation tool ( unlike Google translate ).

2. http://www.bing.com/translator

Cheers!

My assumptions

1. The writing of the language should be ‘recognizable’ to the reader

2. There should be words in the text which look like words in languages known to the reader

3. The syntactic arrangements in the language should be identical to that of at least one of the languages known to the reader

4. Then, it should be possible to get information from such a text

Techniques

1. Isolate the transparent words and interrogate them

2. Refine your views by drawing on logical conclusions

3. Reconstruct your views

Practice Text (In German)

Werd sind die Gideons?

Diese Frage oft von den Lesern dieser Neuen Testamente gestellt. Die nachfolgenden Einzelheiten geben Ihnen darüber Auskunft. Im Herbst 1898 begegneten sich in einem Hotel in Wisconsin/ USA zwei fremde Handelsreisende. Sie erkannten, daß sie beide Chriten waren, deshalb hielten sie demeinsam ihre Abendandacht. Gott gab ihnen den Gedanken, eine Vereinigung christlicher Handelsreisender zu gründen. Dieses Vorhaben fünhrten sie im folgenden Jahr mit einem Dritten aus. Nach gemeinsamem Gebet wählten sie den Namen ‘‘Gideons’’ aus dem Buch der Richter, Kapital 6 und 7im Alten Testamente. Gideon war der Führer einer kleinen Gruppe von Männern, die bereit waren, Gott zu dienem. Durch sie konnte Gott viel für sein Volk Israel tun.

Transparent English words noted in the text

• The heading has ‘‘Gideons and a question mark’’

(The question mark makes the reader think the writer is interrogating the name ‘‘Gideons’’. Anything that may follow may be a history and/ or explanation of the name).

• Neuen Testamente # Alten Testamente

• Hotel Wisconsin/ USA

• Christen

• Im Herbit 1898 (the syntactic arrangement suggest ‘‘the phrase means’’ in the year 1898).

• The position of ‘‘den’’ in the text suggests that it means ‘‘the’’.

• (Kapitel) 6 und 7

• Namen Gideon Israel

These isolated words appear to explain why the name ‘‘Gideons’’ was chosen. The passage suggested that a meeting took place at a hotel in Wisconsin in USA in the year 1898 involving Christians. The meeting had to do with the New Testament. A man from Israel, whose name ‘‘Gideons” is being considered is found in chapter 6 and 7 of one of the books in the Old Testament.

I have applied the techniques and I want people who speak German to tell if the technique has helped somehow in getting some information in this language that is unknown to me. I must say that some of the information derived may be right while others may not. This is natural. In real life situation, not all pieces of information that one receives are true.

Verification

To ascertain the truth of the message it has to be verified. In this regard, the learner would identify and seek help from a German/ English bilingual for assistance. The learner can also use a reliable language translation device that may be available to him.

I really don't think you want to have this as PHP since its will be very inefficient. I would recommend you do the 'back-end' work on a Python/Java program and just pass the results to PHP. PHP scripting is designed for basic sever-side things, not something like what you want to do. There are many ways of connecting Python/Java to PHP, and I am sure you can find an MEMM implementation in those languages.

Hi Goraksh, I think this link will be quite useful for you:

The IndoWordNet and MarathiWordNet seem to have Marathi as a supported language.

As the others have said, to disambiguate words, it is important to build an ontology of every word with its word senses. Luckily for us, there are research groups who have done this for most languages, so all you have to do is use these resources for your purpose.

In corpus linguistics, “any language whose original presentation was in oral form” is considered as ‘spoken language’.

Hi Ian

Thanks for your answer - I see what you mean now. I thought you knew systems that could produce different text versions, maybe based on complexity or something else. That would really be a novelty.

I do find the Google queries very useful - they can give the kinds of interactions that are useful.

Thanks again.

I have used this language pair on Moses. Have you tested that Moses itself is fully installed by running on one of their sample corpora? If YES then you should run "clean-corpus-n.perl" script from your installed Moses folder. Then you should study the difference in your "original" and "cleaned" corpus. I had experienced to see only one special mark that was causing failure in the training process. When I removed / replaced that one character, it started working.

Hello,

There are such software asL AppTek, Asia Online, Moses, Anusaaraka, Apertium, Matxin, BingTranslator, Babylon, Systran, WorldLingo, OpenLogos, NiuTrans, but the most popular is the Goggle Translate. However, you have to remember about a few shortcoming of software in Statistical Machine translation:

1) The results are unexpected. Superficial fluency can be deceiving.

2) Statistical machine translation do not work well between languages that have significantly different word orders (e.g. Japanese and European languages).

3) The benefits are overemphasized for European languages.

As Radu mentioned, there is no such thing as the best translation of one word because it depends on the specific sense associated with the source text word. Further, I would even add that there is no reason to study one-word translations because we speak with many words closely associated syntactically. So the meaning of a word depends ultimately on the interaction of the meaning of each and other words it is used with.

They are good for translators who don't have certain dictionaries. It also helps in some ways but in general long texts need to be translated by a person. Nuances, cultural differences, and vocabulary that is very local need to be translated by a person. Yet,MT can help. .

I think twitter API could help in that. You can build large dialects corpus in fast way.

Thanks

Mohammed