Conference PaperPDF Available

When Google Translate is better than Some Human Colleagues, those People are no longer Colleagues

Authors:

Abstract and Figures

We analyse posts on social media (Facebook, LinkedIn, and Twitter) as a means to understand how translators feel about machine translation (MT). A quantitative analysis of more than 13,000 tweets shows that negative perceptions outweigh positive ones by a ratio of 3:1 overall, and 5:1 in tweets relating MT to human translation. Our study indicates a disconnect between translation and research communities, and we outline three suggestions to bridge this gap: (i) identifying and reporting patterns rather than isolated errors, (ii) participating in evaluation campaigns, and (iii) engaging in cross-disciplinary discourse. Rather than pointing out each other's deficiencies, we call for computer scientists, translation scholars, and professional translators to advance translation technology by acting in concert.
Content may be subject to copyright.
Proceedings of the 39th Conference Translating and the Computer, pages 59–69,
London, UK, November 16-17, 2017. c
2017 AsLing
When Google Translate is better than Some Human Colleagues, those
People are no longer Colleagues
Samuel Läubli1and David Orrego-Carmona2,3
1Institute of Computational Linguistics, University of Zurich, Switzerland
2School of Languages and Social Sciences, Aston University, United Kingdom
3Department of Linguistics and Language Practice, University of the Free State, South Africa
Abstract
We analyse posts on social media (Facebook, LinkedIn, and Twitter) as a means to understand how
translators feel about machine translation (MT). A quantitative analysis of more than 13,000 tweets shows
that negative perceptions outweigh positive ones by a ratio of 3:1 overall, and 5:1 in tweets relating MT to
human translation. Our study indicates a disconnect between translation and research communities, and
we outline three suggestions to bridge this gap: (i) identifying and reporting patterns rather than isolated
errors, (ii) participating in evaluation campaigns, and (iii) engaging in cross-disciplinary discourse. Rather
than pointing out each other’s deficiencies, we call for computer scientists, translation scholars, and
professional translators to advance translation technology by acting in concert.
1 Introduction
Mistranslations can be hilarious. In fact, social media have become ideal outlets to share
pictures of clumsy food menus and mislabelled street signs, as well as screenshots of translation
errors produced by machine translation (MT) engines such as Google Translate. People share
them, laugh at them, and criticise them openly. Professional translators also participate in
the debates about such translations. On a regular basis, translators on LinkedIn, Facebook
and Twitter engage in discussions about mistranslations and how they show that MT is not
comparable to human translation (see Figure 1). Translators use these spaces to voice their
frustration with MT and the implications it has on their profession.
Social media groups dedicated to translation and translators count their members in the
thousands. Communities of practice have emerged thanks to these spaces; however, researchers
Figure 1: Meme posted on a translators’ group on
Facebook, mocking the use of Google Translate.
have barely looked at them in order to
better understand translators and their
opinions. Considering the lack of
attention to translators’ activities on social
media and curious about how these could
be used to understand the translators’
perceptions of MT, we decided to conduct
a study into how translators’ interactions
on social media could help improving
translation technology.
Perceptions of MT among translators
have been explored using questionnaires
and interviews. We conjectured that
eliciting their opinions from online
59
interactions would provide us with data to
understand their attitudes towards MT, and propose ways in which their efforts and knowledge
could support the improvement of the technology.
We used qualitative and quantitative methods to analyse how translators’ feel about MT. We
first present the results of our initial qualitative exploration of groups on Facebook and LinkedIn.
The posts on these platforms gave us the impression that sentiment towards MT in translation
groups is predominantly negative. Aiming at quantifying this initial impression and providing
empirical grounding, we then employed automatic sentiment analysis on a larger data set. We
classified a collection of 13,150 tweets about MT using human annotation on a subsample of
150 tweets, and automatic annotation for the entire collection.1
Both our qualitative and quantitative analyses show that negative perceptions in social media
outnumber positives. To the best of our knowledge, these results provide the first empirical view
of how MT is portrayed on social media. Based on our findings, we make a call for improving
collaboration among professional translators and researchers, and propose possible avenues to
move towards that goal.
2 Background
Most of the literature on the perception of MT among translators, some of which we review in
this section, relies on data obtained through formal questionnaires and interviews. This paper
is motivated by our impression that translators might be more open and direct when expressing
opinions on social media, as well as the fact that there is a lot more data than could be collected
through direct interrogation.
Already in 1993, Meijer found that the MT was seen as a threat among translators, and
negative opinions seem to persist (see Guerberof Arenas, 2013; Gaspari et al., 2015; Cadwell
et al., 2017). Despite significant technological advancements in recent years, translators are
‘still strongly resistant to adopting MT as an aid, and have a considerable number of concerns
about the impact it might have on their long-term work practices and skills’ (Cadwell et al.,
2017). As a response to these concerns, in the last two years, the International Federation of
Translators (FIT) has published three position papers on MT, crowdsourcing, and the future for
professional translators. In their paper on MT, they state that ‘MT is unlikely to completely
replace human translators in the foreseeable future. Leaving aside the area where MT is a
feasible option, there will continue to be plenty of work for them. Professional translators, who
have the appropriate skills and qualifications, will still be needed for demanding, high-quality
products’ (Heard, 2017). Given that the Federation represents the interests of professional
translators, their paper can be seen as an indicator of the relevance to understand how translators
feel about MT.
In spite of the seemingly significant importance for the community,2the use of social media
among professional translators has been barely studied. Desjardins (2016) addresses the aspects
of professionals using social media but primarily as a strategy to increase their visibility, not as
a way of interacting among themselves. The research that is available in the field of translation
and social media has mainly explored the work of non-professionals and their translations
(e. g., Dombek, 2014; O’Hagan, 2017; Jiménez-Crespo, 2017). Although not on social media,
the online presence of translators and their attitudes in other outlets have previously been
1All data and annotations are released under the CC BY-SA 4.0 license, available at https://github.
com/laeubli/MTweet.
2Social media groups dedicated to translation and translators count their members in the thousands, and since
2013, proz.com has been running the Community Choice Awards to recognise, among others, translation and
interpreting professionals and companies who are active and influential on the internet.
60
explored: McDonough Dolmaya (2011) analysed translators’ blog entries to understand the
attitudes and practices of professional translators, while Flanagan (2016) used blogs to study
their opinions towards crowdsourcing. Along the same lines, researchers have also asked
professional translators about their attitude towards MT (Meijer, 1993), their opinion about
post-editing of MT output (Guerberof Arenas, 2013), and their reasons to use or not use MT
(Cadwell et al., 2017).
The research on the translators’ opinions about MT is still limited and, to the best of our
knowledge, no study has analysed interactions on social media as a way of understanding the
translators’ attitude towards MT. Interacting on social media requires less time and effort than
maintaining a website or writing a blog post so we assume a larger number of translators would
be involved in different types of exchanges in social media platforms.
3 Qualitative Analysis
Our aim to fill this gap started with a preliminary analysis of translators’ posts and comments
on Facebook and LinkedIn. We hand-picked 137 examples related directly or indirectly to
MT by browsing through public and invitation-only3groups: Professional Translators and
Interpreters (ProZ.com),4Translation Quality,5The League of Extraordinary Translators,6
Things Translators Never Say,7and Translation and Interpreting Group.8It is important to point
out here that this part of the study does not claim to be comprehensive; it serves to illustrate the
situation that unfolds in these groups rather than provide generalisable results.
In relation to the assumption that MT can be a threat to professional translators, one of
the recurrent topics in these groups is quality. Translators engage in discussions about the
mistranslations produced by MT engines as a way of reinforcing the need for human translators.
Photos and screenshots of translation errors are systematically posted to the groups. Translators
criticise them and comment on the shortcomings of MT (see Figure 2b). Some use the examples
to respond with sarcasm to the possibility that translators might be replaced by machines in the
near future: ‘Oh yes, I’m very worried about being replaced by a machine when it can’t tell the
difference between an extraordinary announcement and a declaration of emergency...’. In their
discussion, Google Translate is normally the main culprit, probably because of its accessibility
and the considerable number of languages in which it operates. There are direct references to
Google Translate in 66 of the 137 posts we collected for this qualitative analysis.
Translators also question the improvements announced by the companies that develop MT.
In response to an article comparing the quality of neural MT to phrase-based MT and human
translation, one translator indicates her doubts about the results commenting ‘I wonder how
the quality was measured if neural came so close to human. In this case, MT as such is not
the issue that is put on the spot but the concept of quality that is used to assess MT output.
Also, as pointed out by other researchers (see Doherty and Kenny, 2014; Cadwell et al., 2017),
translators feel they are not considered part of the development of MT: ‘Yes, AI people who
know nothing about our job, we totally agree that you will figure out how to replace us with
machines in the next ten years. Sure you will.’
In some cases, translators even use MT as an indicator of poor quality when judging other
translators and their translations. Figure 2d shows a comment by a translator who uses
Google Translate’s output quality as a point of comparison to argue that the translation she
3Access is usually granted within minutes.
4https://www.linkedin.com/groups/138763
5https://www.linkedin.com/groups/3877235
6https://www.facebook.com/groups/extraordinarytranslators
7https://www.facebook.com/groups/thingstranslatorsneversay
8https://www.facebook.com/groups/Interpreting.and.Translation
61
(a) Translation from German into English made by Google Translate used as an example
of how confusing MT outputs can be. In the comments, the translators discuss Google
Translate’s poor attempt at rendering the different meanings of the terms in German. In
the output in English, all the German terms are translated as ‘economics’, resulting in a
meaningless repetition of the same term.
(b) A translator posted a link to a list of
examples of mistranslations generated by
Google Translate App’s function that can
translate text in images. The translator
sarcastically comments on the fact that
the poor quality of the translations makes
it unlikely that human translators will be
replaced by computers in the near future.
(c) Translators engage in a discussion about
whether or not MT can be acceptable in
specific circumstances. Some of them argue
MT can be useful for small business without
the resources to pay for a professional
translation, while others stress the fact that
accepting MT as a valid option means
lowering the standards of the profession.
(d) A translator complaints about the
quality of the translation she is revising.
As a way of signalling the poor quality
of the translation made by her colleague,
she claims it would have been better to
proofread a machine-translated text.
Figure 2: Examples of translator interactions in Facebook groups.
62
is proofreading is of low quality. Translators also recognise that some of the mistakes present
in the translations that are posted in the group are such poor examples of translations that ‘not
even Google translate [sic] is that bad.
However, not all the posts and comments on social media discredit MT straight away.
Figure 2a presents an image that was shared in one of the groups showing Google Translate’s
translation of a German text into English. Interestingly, the translators who commented on this
post were genuinely curious about the veracity of the output. Some of them took the time to
retype the text into Google Translate and check whether the translations into English or their
own target languages made any sense.
Comments in the groups often also point at the use of MT as an aid for translating as an
indicator of poor quality or a poorly skilled professional. One of the commentators states
that ‘Machine translation, like Google Translate, can give you a false sense of competence’,
suggesting that non professionals could get the impression they can translate thanks to the
support of MT. Another translator comments on the fact that the fear of MT is, in a way, an
indicator of the competence of the translators. She says that ‘[m]achines will only replace those
translators who translate like machines. These opinions do not represent isolated cases. In
another thread when discussing the issues that MT could bring to the profession, a translator
states that ‘When Google Translate is better than some human colleagues, those people are no
longer colleagues. Using Google Translate or the risk of being replaced by a machine seem
then to be related to a translator’s lack of professionalism or skills.
One of the highlights of MT is affordability: automating the process of translating makes
it possible for people to access translations, even when they do not have the resources to pay
for them. The discussion depicted in Figure 2c serves as an example of this argument among
translators. Some of the translators recognise there are situations in which having access to an
automatic translation is better than having no translation at all, while others would not consider
it possible to accept a translation that only allows users to ‘get the idea’. For some of the
translators, it seems, accepting MT as a valid option would constitute lowering the standards of
the profession.
Discussions in the groups commonly go back to the assumption that human translators
approach translation as a creative task, while MT only looks at translation as the word-for-word
replacement of a string of text. Not all the discussions centre on the negative aspects of
MT. Some translators point out that MT, and Google Translate in particular, are good for
certain language combinations or specific fields, and can actually support the work of skilled
professionals. A translator summarises these two points when he states that ‘Translation and
interpreting are very demanding professions where talented human linguists will continue to
make the difference in value and quality. Nevertheless, it is hard to deny the benefits of
applied language technology CAT for translators and VRI for interpreters to name but a few
to support linguists and language service providers in their joint mission to meet customer
requirements in a very rapidly changing market of demanding end users and organizations who
pay the bill for these language services.
4 Quantitative Analysis
The initial exploration of how MT is discussed on social media reinforced our impression that
perceptions are predominantly negative among professional translators. We conducted a larger
study in order to ground this impression empirically. In this stage, we focused on Twitter data
as large numbers of posts are difficult to obtain from Facebook and LinkedIn (see Section 4.1).
Our goal was to quantify the extent of positive, neutral, and negative tweets on MT, for which
we employed independent human judges (Section 4.2) and an automatic sentiment classifier
63
positive neutral negative tie
12
28
63
47
(a) Assignments per class
Annotator B
n= 150 positive neutral negative
Annotator A
positive 12 11 6
neutral 11 28 2
negative 8 9 63
(b) Confusion matrix
Figure 3: Human Sentiment Analysis
(Section 4.3).
4.1 Data Collection
We collected tweets from twitter.com using a purpose-built, open source web crawler.9We
only kept tweets which (i) contain the terms ‘machine translation’ and/or ‘machine translated’,
(ii) are written in English, according to Twitter’s language identification, and (iii) were created
between 1 January 2015 and 31 July 2017. This method is not exhaustive in that authors may
refer to machine translation by use of synonyms or without mentioning it explicitly. However,
the data is representative of what a user would find searching for the terms mentioned in (i)
above through Twitter’s search interface. Our filtered collection contains 13,150 tweets.
4.2 Human Sentiment Analysis
We sampled 150 tweets from this collection for human sentiment analysis. The selection was
random, except that we required each tweet to contain at least one of the following terms:
‘human’, ‘professional’, ‘translator’. As discussed in Section 5.2, we used this heuristic to
focus on discussions comparing MT and human translation in this part of the study.
The sampled tweets formed the basis for an annotation job on a web-based crowdsourcing
platform.10 Annotators were asked to read each tweet, click all links found in the text for
additional context, and then determine if the tweet is positive, neutral, or negative with regards
to machine translation. Tweets were presented in random order. We included ten control items
as a means to filter out random contributions: each annotator saw ten tweets twice, and we
expected them to be consistent with their own judgements.
Human annotators were recruited through convenience sampling, the restriction being
that they have never been involved with translation, translation studies, or computational
linguistics. Five annotators completed the entire job, from which we excluded three due to
low inter-annotator agreement: they failed to reproduce their own judgements on three or more
out of ten control items.
Human annotation results are summarised in Figure 3. Inter-annotator agreement is 68.7 %
(Cohen’s =0.495). The two remaining annotators independently assigned the same label to
103 out of 150 tweets: 12 positive, 28 neutral, and 63 negative (Figure 3a). Two different labels
were assigned to 47 tweets, with most disagreement between positive and neutral (Figure 3b).
9https://github.com/jonbakerfish/TweetScraper
10https://www.crowdflower.com
64
positive neutral negative
1,396
7,461
4,293
(a) Assignments per class
Automatic
n= 103 positive neutral negative
Human
positive 24 6
neutral 3 21 4
negative 1 17 45
(b) Confusion matrix
Figure 4: Automatic Sentiment Analysis
4.3 Automatic Sentiment Analysis
To annotate our entire collection of tweets (see Section 4.1), we leveraged Baziotis et al.’s
(2017) automatic sentiment classifier.11 Their system scored first in the SemEval 2017 shared
task on classifying the overall sentiment of a tweet (task 4, subtask A), i. e., deciding whether
it expresses positive, neutral or negative sentiment (see Rosenthal et al., 2017). It uses a deep
LSTM network (Hochreiter and Schmidhuber, 1997) with attention (Rocktäschel et al., 2015)
and is completely data-driven: rather than relying on linguistic information and hand-crafted
rules, the system learns to classify from large collections of manually annotated tweets.
We trained the system on the original SemEval data, meaning it is not specifically geared to
tweets on machine translation. Using the 103 tweets that both of our annotators labelled with
the same class as a reference (see Section 4.2), it classifies 68 tweets correctly. This corresponds
to an overall classification accuracy of 66.0 %. In terms of average recall per class the primary
evaluation metric used in SemEval its performance in our domain (66.0 %) is similar to
the performance achieved in the shared task with varied topics (68.1 %; see Rosenthal et al.,
2017). However, precision (33.3 %) and recall (16.7 %) are low for positive tweets. The system
performs better with neutral (precision: 50.0 %, recall: 75.0 %) and negative tweets (precision:
81.8 %, recall: 71.4 %), with a tendency to classify negative tweets as neutral (Figure 4b).
Overall, the classifier labels 1,396 tweets as positive (10.6 %), 7,461 as neutral (56.7 %),
and 4,293 as negative (32.6%). Note that in contrast to the subset used for human annotation
(see Section 4.2), this includes tweets not comprising the terms ‘human’, ‘professional’, or
‘translator’.
5 Findings and Discussion
Our study provides evidence that MT is often portrayed negatively among translators on social
media outlets. The suspicions about a negative attitude towards MT that stemmed from our
qualitative analysis of Facebook and LinkedIn posts (Section 3) were supported by the results
of the sentiment analysis carried out on Twitter data (Section 4).
5.1 Recurrent Topics
Our exploration of Facebook and LinkedIn data (see Section 3) sheds light on recurrent
MT-related topics in social media. Firstly, we observed frequent reiteration of how professional
11source code available at https://github.com/cbaziotis/datastories-semeval2017-task4
65
translators are and will still be needed as MT improves, as shown by the example is provided in
Figure 2b.
Secondly, translators doubt if MT improves at all, for example, by calling into question the
methodology and/or veracity of evaluation campaigns. Referring to a study on productivity with
a web-based translation workbench, a Facebook user says ‘if only your productivity estimate
was correct! If I actually could do 2k words/hour while watching esports [sic], I’d actually take
on all those bottom feeders and still make good bank!’
Thirdly, many posts merely criticise MT for bad quality. Translators spend considerable
time and effort on discussing MT errors, but we were surprised to find little variance
in the discussions and errors reported. In one instance, a translator even made up a
meaningless sentence in Japanese and mocks Google Translate for producing meaningless
output, conceivably because of its sexual connotation (see Figure 5).
5.2 Sentiment Towards MT
In analysing 150 tweets relating MT to human translation, two independent judges found
negative tweets to be most common, outnumbering positive and neutral ones by a ratio of
5:1 and 2:1, respectively. However, sentiment classification in tweets is not trivial: human
judgements do not overlap in a third of all cases, resulting in 47 ties. As shown in Table 1c, there
are even tweets classified as positive by one annotator and negative by the other. Even if other
studies on human sentiment analysis in tweets report similar inter-annotator agreement (e. g.,
Cieliebak et al., 2017), a negotiation phase following the independent procedure of annotation
could have resolved some of the disagreement.
Moreover, sampling tweets based on the presence of keywords ‘human’, ‘professional’, or
‘translator’ is somewhat arbitrary. Still, we found this heuristic useful to get a sense of how
Twitter users contrast MT with human translation (see the examples in Table 1).
Without this restriction, neutral tweets on MT are most common in our collection. News
and product announcements, such as ‘Facebook posts its fast and accurate ConvNet models
for machine translation on GitHub’, often fall into this category. But even so, there are three
times more negative than positive tweets in the 13,150 examples we collected, hinting at the
predominance of negative perceptions about MT in general.
The caveat here is that sentiment was determined by means of an automatic classifier. The
classifier did not have access to contents such as websites and images linked in tweets, which
human annotators were explicitly asked to consider when making their judgement. It also was
not geared to tweets on MT specifically; while the system we leveraged would have allowed for
topic-based classification (see Baziotis et al., 2017), we lacked appropriate amounts of training
data. Despite these limitations, the system reproduced human judgements with an accuracy
of 66.0 % overall. This corresponds to state-of-the-art results (see Rosenthal et al., 2017), and
is similar to the degree of disagreement between human annotators (see above). This is good
enough to get a sense of the class distribution in our data, even if the classifier does make
mistakes (e. g., Table 1b). A clear advantage is speed: the 13,150 tweets are labelled in seconds.
Eliciting human annotations would have taken a lot longer and would have been expensive at
this scale.
5.3 A Case for Collaboration
Even after its emergence as a profession in the last century, translation still struggles with
recognition and undervaluation (see Tyulenev, 2015; Flanagan, 2016). Apart from the general
situation of the profession, translators also feel, and indeed in many cases are, left out in
the processes towards the development of translation technologies (see Doherty and Kenny,
66
Text Automatic Human HumanAHumanB
(a) Six reasons why machine translation can
never replace good human translation:
https://t.co/JzLYbXO6yJ #xl8 #t9n
negative negative negative negative
(b) When you solely rely on machine
translation... via @inspirobot
#wedoitthehumanway #htt
https://t.co/UpfnVd4k8W
neutral negative negative negative
(c) High-quality machine translation is
threatening to make translators ‘the
coffee-bean pickers of the future’
https://t.co/n8fGvIHBao
negative tie positive negative
(d) Difference between professional translation
and machine translation by @ChrisDurbanFR
#xl8 #ITIConf17 https://t.co/gFhgRrLtJq
neutral neutral neutral neutral
(e) Pretty incredible. For a few languages,
machine translation is near equal to human
translation. https://t.co/GsCejE0cUW
positive positive positive positive
Table 1: Example tweets. Sentiment was assessed by two human annotators as well as an
automatic sentiment classifier.
2014; Cadwell et al., 2017). They resist technology because they feel they need to protect
their profession and resort to the defence of quality as the main argument for their cause.
This behaviour is neither a new strategy, nor something restrictive of professional translators.
As Pym (2011) puts it: ‘Resistance to technological change is usually a defense of old
accrued power, dressed in the guise of quality. However, it is unlikely that the technological
development will stop. Together with its quality, the use of MT has increased significantly
in the last decades. Research has also provided evidence of increased productivity through
post-editing of MT, and companies are moving more and more towards a context in which this
practice is the norm rather than the exception (e. g., Green et al., 2013; Koponen, 2016). While
it can be assumed that translation will continue being an activity with human involvement,
it will (continue to) involve various degrees of automation as translation technologies evolve.
We believe that translators should be actively engaged in these developments, and that their
actions on social media could help inform and support research on translation technology. In
the following section, we propose a set of recommendations aimed at fostering collaboration
and promoting common goals among researchers and professional translators.
6 Recommendations
As hilarious as MT errors can be, laughing about them does neither improve translators’ lives
nor the technology. The study we present in this paper fills a gap in the exploration and
quantification of translators’ perceptions as it brings social media into the picture. Our findings
imply that translators and researchers have different understandings of the functioning and
purposes of MT, but at the same time show that translators are aware of the types of issues that
are problematic for it. Considering our findings, we believe that professional translators could
and should have more influence on future developments in MT and translation technology in
general, and propose three initial recommendations to bridge this gap:
67
Figure 5: Translation of a meaningless text in Japanese into English by Google Translate. The
translator posts the screenshot because Google Translate’s output has a sexual connotation; he
uses this argument as proof that machines will never replace human translators.
(i) Identify and report patterns rather than isolated errors. State-of-the-art MT systems
are not based on rules, so complaining about specific words that are mistranslated does
not help much. Reporting error patterns, in contrast, may help gearing systems to new
objectives, such as ensuring noun phrase agreement or controlling for negation particles
(see also Sennrich, 2017). Professional translators have the knowledge and expertise to
identify these patterns and, given the appropriate tools, pattern reporting could easily be
integrated into their regular workflows.
(ii) Participate in evaluation campaigns. Our study has shown that criticising findings of
MT quality or productivity as being unrealistic is a recurrent theme on social media. The
MT research community puts a lot of effort into evaluation campaigns. At the annual
Workshop on Machine Translation (WMT), for example, research teams from all over
the world compete for the best translation system in several language pairs. Human
judgements are the primary evaluation metric, but rather than from translators, they
stem from ‘researchers who contributed evaluations proportional to the number of [MT
systems] they entered’ into the competition (Bojar et al., 2017) although blindly, the MT
researchers evaluate their own systems. We believe that the involvement of professional
translators in scientific evaluation campaigns would not only improve the quality and
credibility of their outcome, but also improve the translators’ understanding of and
impact on the methodologies used to evaluate MT systems.
(iii) Engage in cross-disciplinary discourse. We need to talk to each other. The issues
presented above show that professional translators and researchers hold different positions,
not due to a lack of information or skills, but rather poor communication. Translators make
good points about the shortcomings of MT, but they comment on MT issues primarily
among themselves. The outcomes of academic research, on the other hand, are poorly
communicated to translators and seemingly give them the impression that MT has little to
offer to improve their conditions. For translators to have an active role in the development
of the technologies they (have to) use, it is necessary for both sides, professional translators
and researchers, to meet halfway and cooperate. Common rather than separate spaces on
social media may be a starting point.
68
References
Baziotis, Christos, Nikos Pelekis, and Christos Doulkeridis. 2017. Datastories at SemEval-2017 task 4: Deep
LSTM with attention for message-level and topic-based sentiment analysis. In Proceedings of the 11th
International Workshop on Semantic Evaluation. Vancouver, Canada, pages 747–754.
Bojar, Ondˇ
rej, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias
Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino,
Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translation (WMT17). In
Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers. Copenhagen,
Denmark, pages 169–214.
Cadwell, Patrick, Sharon O’Brien, and Carlos S. C. Teixeira. 2017. Resistance and accommodation: factors for
the (non-) adoption of machine translation among professional translators. To appear in Perspectives, available
online at http://www.tandfonline.com/doi/full/10.1080/0907676X.2017.1337210.
Cieliebak, Mark, Jan Milan Deriu, Dominic Egger, and Fatih Uzdilli. 2017. A Twitter corpus and benchmark
resources for German sentiment analysis. In Proceedings of the 5th International Workshop on Natural
Language Processing for Social Media. Valencia, Spain, pages 45–51.
Desjardins, Renée. 2016. Translation and Social Media: In Theory, in Training and in Professional Practice.
Palgrave Macmillan.
Doherty, Stephen and Dorothy Kenny. 2014. The design and evaluation of a statistical machine translation syllabus
for translation students. The Interpreter and Translator Trainer 8(2):295–315.
Dombek, Magdalena. 2014. A study into the motivations of internet users contributing to translation
crowdsourcing: the case of Polish Facebook user-translators. Ph.D. thesis, Dublin City University.
Flanagan, Marian. 2016. Cause for concern? Attitudes towards translation crowdsourcing in professional
translators’ blogs. The Journal of Specialised Translation 25:149–173.
Gaspari, Federico, Hala Almaghout, and Stephen Doherty. 2015. A survey of machine translation competences:
insights for translation technology educators and practitioners. Perspectives 23(3):333–358.
Green, Spence, Jeffrey Heer, and Christopher D. Manning. 2013. The efficacy of human post-editing for language
translation. In Proceedings of the 2013 Conference on Human Factors in Computing Systems (CHI). Paris,
France.
Guerberof Arenas, Ana. 2013. What do professional translators think about post-editing? The Journal of
Specialised Translation 19:75–95.
Heard, Reiner. 2017. FIT position paper on machine translation. Babel 63(1):130–135.
Hochreiter, Sepp and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9(8):1735–1780.
Jiménez-Crespo, Miguel A. 2017. Crowdsourcing and online collaborative translations: Expanding the limits of
Translation Studies, volume 131 of Benjamins Translation Library. John Benjamins.
Koponen, Maarit. 2016. Is machine translation post-editing worth the effort? A survey of research into post-editing
and effort. The Journal of Specialised Translation 25:131–148.
McDonough Dolmaya, Julie. 2011. A window into the profession: What translation blogs have to offer translation
studies. The Translator 17(1):77–104.
Meijer, Siety. 1993. Attitudes towards machine translation. Language International 5(6):11–13.
O’Hagan, Minako. 2017. Deconstructing translation crowdsourcing with the case of a Facebook initiative. In
Dorothy Kenny, editor, Human Issues in Translation Technology, Routledge, pages 25–44.
Pym, Anthony. 2011. What technology does to translating. Translation & Interpreting 3(1):1.
Rocktäschel, Tim, Edward Grefenstette, Karl Moritz Hermann, TomášKoˇ
ciský, and Phil Blunsom. 2015.
Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664.
Rosenthal, Sara, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: sentiment analysis in Twitter. In
Proceedings of the 11th International Workshop on Semantic Evaluation. Vancouver, Canada, pages 502–518.
Sennrich, Rico. 2017. How grammatical is character-level neural machine translation? Assessing MT quality with
contrastive translation pairs. In Proceedings of the 15th Conference of the European Chapter of the Association
for Computational Linguistics (EACL). Valencia, Spain, pages 376–382.
Tyulenev, Sergey. 2015. Towards theorising translation as an occupation. Asia Pacific Translation and Intercultural
Studies 2(1):15–29.
69
... In certain domains, MT is now promoted by LSPs as a productivity boost to translating from scratch, even though recent studies (e.g., Läubli and Orrego-Carmona, 2017) have suggested that professional translators generally have rather negative opinions about MT and PE. ...
... Translators mostly express negative opinions about MT, focusing on the important role of professional translators even in MTcentered workflows, on the low efficiency of post-editing and on the poor quality of the MT output, according to Läubli and Orrego-Carmona's (2017) study that analyzed posts on social media (Facebook, LinkedIn, and Twitter). But are these negative opinions based on a resistance to technology? ...
... The answer is not clear. While some studies suggest that it might be, such as Läubli and Orrego-Carmona's (2017), others suggest the opposite. In ...
Article
Full-text available
Intending to explore the experiences with and perceptions of the use of machine translation and post-editing in professional practice, this paper examines quantitative and qualitative data from two questionnaires targeted at professional translators and project managers with experience in working with post-editing assignments and based in the Netherlands. Results suggest that trust in the technology, in the translation company, and/or the project manager are at the center of translators’ decisions on whether or not to accept a post-editing task. The current study supports previous claims that more attention needs to be paid to the pivotal role project managers play in the translation workflow, especially in creating and maintaining relations of mutual appreciation and trust between stakeholders, and argues that future research should focus more specifically on determining project managers’ MT and PE training and machine translation literacy.
... Conversely, the meaning conveyed in the source text may be less important than its formal properties (Tanrikulu 2017, Odinye 2019). However, machine translations have been associated with negative perceptions, such as producing senseless texts (Läubli & Orrego-Carmona 2017, Latief et al. 2020. ...
... In this article, machine translations are compared to post-edited versions to investigate whether machine translations can be harnessed to develop a readability corpus for Sesotho. As such, although machine translations have been associated with negative perceptions, such as producing senseless texts (Läubli & Orrego-Carmona 2017, Latief et al. 2020, the machine translations are considered appropriate if they produce the same levels of readability as translations that employ human intervention such as our post-edited dataset. ...
Article
Full-text available
This article addresses the scarcity of gold-standard annotated corpora for readability assessment in Sesotho, a low-resource language. As a solution, we propose using translated texts to construct a readability-labelled corpus. Specifically, we investigate the feasibility of using Google Translate to translate texts from Sesotho to English and then manually post-editing the texts. We then evaluate the effectiveness of the Google translations by comparing them to the human-post-edited versions. We utilised the Ghent University readability demo to extract the readability levels of both the Google translations and the human-post-edited translations. The translations are then evaluated using three evaluation metrics, namely, BLEU, NIST, and RIBES scores. The translation evaluation results reveal substantial similarities between the machine translations and the corresponding human-post-edited texts. Moreover, the results of the readability assessment and the comparison of text properties demonstrate a high level of consistency between machine translations and human-post-edited texts. These findings suggest that Google Translations show promise in addressing challenges in developing readability-labelled parallel datasets in low-resource languages like Sesotho, highlighting the potential of leveraging machine translation techniques to develop translated corpora for such languages. The evaluation of Google Translations in the context of educational texts in Sesotho and the demonstration of the feasibility and potential of using machine translations for enhancing readability in Sesotho will aid in the quest for developing Sesotho text readability measures.
... While there is a rapidly growing body of research investigating how MT is used by professional translators (e.g. Läubli and Orrego-Carmona, 2017;Moorkens et al., 2018;Sánchez-Gijón et al., 2019) and translation students (e.g. Kenny and Doherty, 2014;Gaspari et al., 2015;Moorkens, 2018;Rossi, 2017 doing post-editing (e.g. ...
... Kenny and Doherty, 2014;Gaspari et al., 2015;Moorkens, 2018;Rossi, 2017 doing post-editing (e.g. Dorst et al., 2022;Guerberof-Arenas, 2013;Läubli and Orrego-Carmona, 2017;Loock et al., 2022), there is to our knowledge little to no research that focuses on the way people actually conceptualise MT and how they understand the processes involved in MT as compared to HT. ...
Article
Full-text available
This paper explores how students conceptualise the processes involved in human translation (HT) and machine translation (MT), and how they describe the similarities and differences between them. The paper presents the results of a survey involving university students (B.A. and M.A.) taking a course on translation who filled out an online questionnaire distributed in Finnish, Dutch and English. Our study finds that students often describe both HT and MT in similar terms, suggesting they do not sufficiently distinguish between them and do not fully understand how MT works. The current study suggests that training in Machine Translation Literacy may need to focus more on the conceptualisations involved and how conceptual and vernacular misconceptions may affect how translators understand human and machine translation.
... The increasing use of NMT in professional translation is not always coupled with positive attitudes by qualified practitioners and still poses a series of ethical, professional, and quality issues that can undermine their trust or support their critical stance (e.g. Läubli and Orrego-Carmona, 2017;Vieira, 2020). On the other hand, the general high quality and fluency of NMT raw outputs can earn or increase the trust in this technology, especially that of less-or non-experienced translators and non-expert or less-educated users -e.g. ...
... Studies have since investigated the motivation of volunteers (Cámara de la Fuente 2015; Deriemaeker 2014; Olohan 2014), the inclusion of volunteer translation in training programs (Desjardins 2011;Sánchez Ramos 2019), the attitudes of professionals to collaborative practices (Gough 2011;Läubli and Orrego-Carmona 2017), the final quality of translations performed by volunteers (Carreira Martínez and Pérez Jiménez 2011), and the work of volunteer translators in the third sector (Boéri and Maier 2010; Tesseur 2019). However, few studies have explored the identity of the volunteer translator using methodologies that determine their relationship and position within society, as occurs in other disciplines. ...
Article
Full-text available
The need for immediate communication in a world characterized by an ever-increasing flow of information has led to a rapid increase in volunteer translation in recent decades. While previous studies have focused on aspects such as volunteer translators’ motivation and the attendant ethical issues of voluntary translation, comparatively little research has explored the phenomenon from a discursive perspective. The research reported in this article explored the discursive construction of the volunteer translator in non-governmental organizations (NGOs) active in Spain using critical discourse analysis. Drawn from analysis of 25 semi-structured individual interviews with volunteer translators in five different NGOs, the findings reveal that volunteer translators are seen either as an agent and actor of social change who facilitates communication and possesses a strong sense of identity and social status, or as someone who is interfering in a sector traditionally limited to professional translators and who consequently has a weaker sense of identity and social status. This suggests that the role of the volunteer translator currently lacks definition, and that further research is needed to better understand the relationship between volunteer and professional translators in the 21st century.
... Prior research in machine translation has predominantly focused on professional translators' viewpoints, including their perception of MT as a potential industry threat (Vieira, 2020), the factors influencing their adoption or rejection of MT (Cadwell et al., 2018), and the impact of MT advancements on their practices, selfperception, and professional standing (Läubli & Orrego-Carmona, 2017;Sakamoto, 2019). These studies have consistently highlighted translators' concerns and their generally lower regard for MT regarding quality and utility. ...
Article
Full-text available
Machine translation (MT) is the automated process of translating text between different languages, encompassing a wide range of language pairs. This study focuses on non-professional bilingual speakers of Turkish and English, aiming to assess their ability to discern accuracy in machine translations and their preferences regarding MT. A particular emphasis is placed on the linguistically subtle yet semantically meaningful concept of evidentiality. In this experimental investigation, 36 Turkish–English bilinguals, comprising both early and late bilinguals, were presented with simple declarative sentences. These sentences varied in their evidential meaning, distinguishing between firsthand and non-firsthand evidence. The participants were then provided with MT of these sentences in both translation directions (Turkish to English and English to Turkish) and asked to identify the accuracy of these translations. Additionally, participants were queried about their preference for MT in four crucial domains: medical, legal, academic, and daily contexts. The findings of this study indicated that late bilinguals exhibited a superior ability to detect translation accuracy, particularly in the case of firsthand evidence translations, compared to their early bilingual counterparts. Concerning the preference for MT, age of acquisition and the accuracy detection of non-firsthand sentence translations emerged as significant predictors.
... Many research efforts have been devoted to understanding why and how the quality of MT's raw output compares to that of human translations (Läubli & Orrego-Carmona, 2017). To reach acceptable quality standards, machine-translated raw output has to undergo postediting, that is, human revision of MT output (British Standards Institute, 2015). ...
Article
Full-text available
In today's professional landscapes, new technologies have altered media localization workflows as much as practitioners' workstations and habits. A more comprehensive integration of automation tools, including (neural) machine translation systems, has been ushered in by the proliferation of cloud ecosystems. In a further technological drive in the production of subtitle projects, systems now integrate automatic speech recognition and can machine translate subtitles from pre-spotted templates. The rise of post-editors in media localization, specifically in subtitling, has been a reality for some time now, triggering the need for up-to-date training methods and academic curricula. It is against this backdrop that this article seeks to examine the perception of post-editing among trainees in subtitling. A total of four teaching experiences, conceived as practical experiments in interlingual subtitle post-editing (English into Spanish), involving postgraduate students from both Spain and the United Kingdom, are described here. The sample comprised 36 master's-level students enrolled in translator training programmes that have a focus on audiovisual translation. A mixed-methods approach was adopted for this study; after each experience, the feedback collated through online questionnaires has proved paramount to understanding the participants' opinions about post-editing in the subtitling classroom. Interestingly, most of the respondents believe that subtitle post-editing training should feature more prominently in translation curricula even though they have voiced their reluctance to undertake post-editing work professionally.
Article
In recent years, machine translation post-editing (MTPE or PE for short) has been steadily gaining ground in the language industry. However, studies that examine translators’ perceptions of, and attitudes towards, MTPE paint a somewhat negative picture, with PE pricing methods and rates being a major source of dissatisfaction. While the European Master’s in Translation Competence Framework stresses the importance of preparing translation graduates for market challenges, to date there have been no concrete suggestions for practical activities designed to introduce MTPE-pricing-related topics into the translation classroom. The present article aims to address this gap by describing a teaching unit developed for master’s students. The activity includes comparing three MTPE pricing methods commonly used in the industry: word-based, time-based and effort-based rates. Using authentic performance data from individual PE tasks carried out in MateCat, students were able to discover the different levels of remuneration they would receive for the same task, depending on the pricing method applied. The results, which show wide variation across both methods and students, proved useful in raising students’ awareness of the thorny issue of setting PE rates and sparking reflection on the financial implications of accepting PE assignments.
Article
Full-text available
As part of a larger research project on productivity and quality in the post-editing of machine-translated and translation-memory outputs, 24 translators and three reviewers were asked to complete an on-line questionnaire to gather information about their professional experience but also to obtain data on their opinions about post-editing and machine translation. The participants were also debriefed after finalising the assignment to triangulate the data with the quantitative results and the questionnaire. The results show that translators have mixed experiences and feelings towards machine-translated output and post-editing, not necessarily because they are misinformed or reluctant to accept its inclusion in the localisation process but due to their previous experience with various degrees of output quality and to the characteristics of this type of projects. The translators were quite satisfied in general with the work they do as translators, but not necessarily with the payment they receive for the work done, although this was highly dependent on different customers and type of tasks.
Conference Paper
Full-text available
In this paper we present SB10k, a newcorpus for sentiment analysis with approx.10,000 German tweets.We use this new corpus and two existingcorpora to provide state-of-the-art bench-marks for sentiment analysis in German:we implemented a CNN (based on thewinning system of SemEval-2016) anda feature-based SVM and compare theirperformance on all three corpora.For the CNN, we also created Germanword embeddings trained on 300Mtweets. These word embeddings werethen optimized for sentiment analysisusing distant-supervised learning.The new corpus, the German wordembeddings (plain and optimized), andsource code to re-run the benchmarks arepublicly available.
Article
Full-text available
Advances in the field of machine translation have recently generated new interest in the use of this technology in various scenarios. This development raises questions over the roles of humans and machines as machine translation appears to be moving from the peripheries of the translation field closer to the centre. The situation which influences the work of professional translators most involves post-editing machine translation, i.e. the use of machine translations as raw versions to be edited by translators. Such practice is increasingly commonplace for many language pairs and domains and is likely to form an even larger part of the work of translators in the future. While recent studies indicate that post-editing high-quality machine translations can indeed increase productivity in terms of translation speed, editing poor machine translations can be an unproductive task. The question of effort involved in post-editing therefore remains a central issue. The objective of this article is to present an overview of the use of post-editing of machine translation as an increasingly central practice in the translation field. Based on a literature review, the article presents a view of current knowledge concerning post-editing productivity and effort. Findings related to specific source text features, as well as machine translation errors that appear to be connected with increased post-editing effort, are also discussed. [Full text available at: http://www.jostrans.org/issue25/art_koponen.pdf]
Article
Despite considerable advances in machine translation (MT), adoption by professional translators still meets with resistance. Research on the human factors associated with MT (non-)adoption is required to understand this state of affairs. We investigate whether two specific groups of professional translators use MT, what reasons they advance for its use/non-use and what factors might explain the reasons given. Participants advanced an equally diverse set of reasons for using MT as for not using it, and this was strongly linked to text type, language pair, quality and trust. Using an agency theory lens, we found evidence of Pickering’s dialectics of resistance and accommodation in the focus group data. We also found that one group of translators is more open to the use of MT, and suggest that the socio-technical context of deployment might explain this finding.
Chapter
Chapter 5 addresses the relationships between online social media (OSM) and the professional translation market, and, more specifically, how professional translators are leveraging OSM in creative – and sometimes surprisingly lucrative and beneficial – ways. The chapter lists some of the ways in which translators self-describe their work and their self-perceived role(s) on various OSM platforms, with emphasis given to activity on LinkedIn. Research in translation studies has sought to ‘unveil’ the very people – the translators – who have helped disseminate knowledge and culture such that they be seen and recognized for their contributions. While this chapter explores the positive aspects associated with translators’ digital presence, it also calls into question the potential pitfalls of this ‘digital visibility’. Could the translation tweets, statuses and other forms of user-generated content not also paradoxically contribute to the translator’s invisibility?