Content uploaded by Joss Moorkens
Author content
All content in this area was uploaded by Joss Moorkens on Dec 03, 2015
Content may be subject to copyright.
Consistency in Translation Memory Corpora: A Mixed-
Methods Case Study
Abstract:
This study investigates the prevalence and causes of inconsistency in translation memories (TM)
using a sequential explanatory mixed methods design. The initial quantitative phase introduces a
novel method and typology for measuring and categorizing inconsistencies in TM data. The data is
the product of professional computer-aided translation of software documentation. In the follow-on
qualitative phase, interviewees compare the quantitative results with their professional experience
of TMs. Their confirmation of the quantitative results improves the validity of the study. The
interview data also increases the utility of the research, suggesting possible causes and solutions for
inconsistency. Results are presented interactively, followed by a short discussion of the findings and
their consequences.
Situated within the field of Translation Studies, Translation Technology focuses on
computer-aided human translation and machine translation, usually of texts that are
repetitive and functional. Much of the research in the area is in the form of quantitative
studies. This study instead uses a sequential explanatory mixed methods approach to
analyze consistency in Translation Memory (TM), a widely-used computer-based aid for
human translation that was commercially introduced in the early 1990s.
A TM is a repository of previously translated text that has been divided into segments. By
default, a segment is usually a sentence, a heading, or a list element. Segments in the source
language are aligned with those in the target language so that they can be recycled within a
TM tool. A TM tool manages the translation process, providing a user interface for the
translator to see both source and target texts and automatically creating a TM during
translation by saving a segment of source and target text together as a translation unit (TU).
In the case of reappearance of a previously translated segment the TM software will
propose the previous translation to the translator. Depending on the parameters set by the
translator, the TM system will also suggest partial or ‘fuzzy’ matches, based on a percentage
of similarity between a new source text (ST) segment and a source language segment (or
source-language segments) already in memory.
The axioms behind the use of TM tools are that they reduce the cost of translation, save
time, increase productivity, and remove inconsistency by allowing the user to leverage
legacy materials (Austermühl, 2001, p.140; Olohan, 2011, p.343). Costs are further reduced
as translators are often paid based on TM match analyses, with full payment offered for
translation from scratch, partial payment offered for editing fuzzy matches, and a small (or
sometimes no) fee paid for reviewing 100% matches. In theory TM tools should produce
consistent translations as previously translated work is recycled. This study aims to discover
whether this is true in practice. The presence of inconsistency in a TM has an associated cost
in maintenance of the database and in lowering the match percentage (with which the
translation price is associated), particularly in the case of inconsistent source text.
Inconsistency may also reduce clarity and ease of use, and may impact on safety if the
content is sufficiently confusing (Cronin, 2013, p.37).
Some prior research on quality and consistency in TM aided translations identified a
tendency toward inconsistency (Merkel, 1998; Rieche, 2004) and error propagation (Bowker,
2005; Ribas-Lopez, 2007). These were studies using a small number of translators or pilot
studies involving inexperienced translation students. Merkel (1998) surveyed 13 translators
and said that, as those translators were gradually beginning to use TM, the inconsistencies
were caused by a “clash between an established translation culture” and the recently
introduced technology (p.166). Bowker (2005), following a quantitative study with a small
number of student translators, noted that “although it is frequently claimed that TMs
improve consistency, this is not always the case” (p.18).
In endeavoring to analyze translation consistency in TM data, the current research began
with a pilot study, which found inconsistencies in the target text (TT) of two TMs, but was
limited by a focus on the TT and by a lack of explanatory data. The pilot study did, however,
help to identify the types of inconsistency that may occur in a TT. The dearth of full studies
of consistency in the prior research and the limitations identified in the pilot study led to a
choice of mixed methods for the main study. Rather than just conducting a quantitative
study, as in the pilot, it was considered that a follow-up qualitative study would both “shore
up weaknesses” and “provide confirmation and elaboration” of quantitative results
(Parmelee, Perkins, and Sayre, 2007, p.195). Frey, Botan, Freidman, and Kreps write that,
although quantitative studies are understood to have high measurement reliability, a
combination of qualitative and quantitative studies of the same concept enhances validity
and reliability of measurements along with enhancing the “credibility of the conclusions
they draw” (1991, p.124).
The pilot study led to the choice of a biphasic approach for the main study. In the first phase,
we analyzed TM data to create a typology of commonly occurring inconsistencies in source
and target texts. As a single case study is considered to be a poor basis for generalization, a
decision was made to carry out analyses of multiple TMs using the typology, as a replicated
study gives “considerable advantages over single-case studies in terms of the rigor of the
conclusions which can be derived from them” (Susam-Sarajeva, 2009, p.7). In the second
phase, we carried out a series of interviews with localization professionals to ask whether
they considered consistency a problem and to suggest possible causes of inconsistency.
Credible and applicable conclusions were a focus within our research center, and the
interviews added validity to our measures of inconsistency in TM data and to our
recommendations for the minimization of inconsistency in the translation process.
Method
The purpose of this study was to analyze and document in detail the type of inconsistencies
found in TM data, then to explain how common these inconsistencies are and how they
might come about. This led to the choice of a sequential, explanatory mixed methods design
for this study, in which the qualitative data is intended to expand upon the quantitative
results (Creswell and Plano Clark, 2007, p.72). As the study focused particularly on the
quantitative analysis, and as the quantitative phase provided results from which the
interview questions were drawn, the follow-up explanations model of the explanatory
design was used. In the first, quantitative phase of the study, four sets of TM data (two
English-to-German TMs and two English-to-Japanese TMs presented in the TMX or
Translation Memory eXchange standard interchange format) from software documentation,
donated by two world-renowned software companies, were measured for consistency. In
the second phase, qualitative interviews with translators and other translation professionals
who work with TMs were conducted to explore their experiences of consistency issues in
TM. It was also anticipated that these interviews would allow explanation of some of the
findings from the quantitative stage of the study. An independent level of interaction
between the quantitative and qualitative research data was chosen, so that data from the
two phases was initially analyzed separately.
Access to data for use in this study was subject to protracted discussion and, in one case,
the signing of a non-disclosure agreement. Sharing of data is a hot topic generally, but is
particularly so in the localization industry. This is due to financial value being attributed to
TMs and the lack of a clear precedent with regard to ownership of TM data (see Gow, 2007;
Smith, 2009). In addition, no language services provider wants to be identified as having
poor quality TM data. As a result, companies involved in localization are protective of the
TM data they possess and accessing a large TM corpus for research purposes presents a
challenge. The data used for this study was made available anonymously thanks to
connections between the Centre for Next Generation Localisation (CNGL) at Dublin City
University, and the companies that donated data. This factor has an effect on the reliability
and validity of the research – while this study uses real-world data, it was necessarily limited
by what could be accessed.
For two of the TMs, a sample of 50,000 translation units (TUs) was studied. In order to
confirm homogeneity between the sample and the full TM corpus in each case, a chi square
test was carried out using Excel. The test was based on comparative measurements of
corpus statistics as measured using Wordsmith Wordlist software and comparative
measurement of the frequency of category 3 inconsistencies. The corpus statistics used
were types (distinct words), standardized type-token ratio, and mean word length (in
characters). The chi square test found no significant difference between the sample and the
full TM.
Quantitative Phase
The quantitative phase of the study measures inconsistency at the segment and sub-
segment level. Segment level inconsistencies are observed where two segments that one
could reasonably expect to be formally identical differ from each other in some way. Source
segments are viewed as being formally different if their meanings differ, but the term
‘inconsistent source segments’ is used to refer to cases where there are very minor formal
differences between two source segments and such differences do not reflect any semantic
differences between the segments in question. Such minor formal differences include
differences in capitalization, tags, punctuation, spaces, character formatting, and spelling
(where a segment may be inconsistent with another segment simply because of a
misspelling, inconsistent use of British or US English spelling, or a typographical error in one
of the segments).
In the case of target segments, it appears reasonable to expect segments that are
translations of ‘the same’ source segment (i.e. segments that are translations of different
tokens of the same source type) to be formally identical, especially in a translation memory
scenario where the goal is to reuse existing translations for already encountered source
segments.
Where there are two different translations (and thus two different target segments) for a
single source segment type, this is considered a target segment-level inconsistency. As there
may be more than one inconsistency within these segments, each discrete inconsistency is
counted and categorized. The differences between the target segments in question can be
very minor formal differences (as defined above), but they can also be more substantial, in
extreme cases even leading to semantic differences between the two segments.
At segment level, the following four categories are possible:
1. inconsistent source segments are translated as inconsistent target segments
2. inconsistent source segments are translated as consistent target segments
3. consistent source segments are translated as inconsistent target segments
4. consistent source segments are translated as consistent target segments
This study is primarily interested in categories 1 and 3, but also in the possibility of
consistency being introduced during the process of computer-assisted translation (category
2). Category 4 may be seen as the ideal in specialized translation, whereby the TM has
provided the best possible leverage and thus saved the maximum possible amount of time
and money. An example of each category from our TM data is given in Table 1.
Source Text
Target Text
Category 1
Callouts window
Fenster "Callouts"
inconsistent source->
inconsistent target
Callouts Window
Callouts-Fenster
Category 2
Plane, perspective
Ebenen, perspektivische
inconsistent source->
consistent target
Planes, perspective
Ebenen, perspektivische
Category 3
Camera button
Kameraknopf
consistent source->
inconsistent target
Camera button
Kamera- Schaltfläche
Category 4
text background
Texthintergrund
consistent source->
text background
Texthintergrund
consistent target
Table 1: Example of TU Categories
Inconsistent segments are counted by identifying the number of types n. The number of
segment-level inconsistencies is the type count minus one (n-1). Thus in the case of a single
source segment (type) that has 4 tokens, if there are 3 separate translations (3 types; one of
which appears twice), then the number of target segment inconsistencies is 2 (or 3-1). We
give a special status (of ‘master’ or ‘reference’ segment) to one of the target segments, and
treat the other two segments as inconsistent with that reference segment. The reference
segment is the one which appears first in our sorted list, and which a translator could have,
but did not reuse in unchanged form. For example, the following four translations for 'Click
an empty part of the drawing area.' appear in the TM data:
a. Klicken Sie auf der freien Zeichenfläche.
b. Klicken Sie auf einen freien Bereich der Zeichenfläche.
c. Klicken Sie auf einen freien Bereich der Zeichenfläche.
d. Klicken Sie auf einen beliebigen freien Bereich auf der Zeichenfläche.
Although there are four tokens, there are only three types: a, b, and d. If segment a is
assigned the status of reference segment, the segments that are inconsistent with the
reference segment are b (repeated for c) and d: thus count two inconsistencies are counted.
When there are three types n=3, and since the count in this study is of type (n - 1), two
inconsistencies are counted.
At segment-level, source or target segments are either consistent or formally differ and are
thus inconsistent. However, there may be more than one inconsistency within these
segments. For this reason, sub-segment inconsistencies found within inconsistent target
text segments (those found in categories 1 and 3 above) are also counted and categorized.
These inconsistencies are categorized mostly per part of speech aside from those with
inconsistent punctuation or where word order has been changed. If there are more than
three inconsistencies within a target segment, that segment is considered to have been
wholly retranslated. These categorized inconsistencies may be further subcategorized; for
example nominal inconsistencies that differ lexically, or in number (singular/plural). Other
typical sub-segment inconsistency categories are verb, space, punctuation, preposition (for
German), and postposition (for Japanese).
These subsegment-level inconsistencies are counted in the same way as segment-level
inconsistencies: we identify the number of types n, assign one the status of master or
reference segment, then count the types that are inconsistent with the part-of-speech or
word order in the reference segment. Thus the count is n minus the reference segment (n-1).
Again, the reference segment is the one which appears first chronologically, and which a
translator could have reused, but chose to edit in some way.
The aim of the quantitative study is to identify translation units that fall into the four
categories as specified in table 1. This was done by using a Python script to extract the ST
and TT segments from the <seg> tags. The extracted, aligned segments were pasted into a
spreadsheet using Libreoffice software and sorted alphabetically so that repeated segments
would appear consecutively. When repeated ST segments were found (or those containing
the minor inconsistencies as specified), the corresponding TT was checked for consistency.
The TT was also checked for repeated segments and where they were found, the
corresponding aligned ST segments were checked for consistency, to see if consistency was
introduced via translation using TM as per category 2. TUs extracted in this way were copied
to a new spreadsheet and classified according to whether they belonged to category 1, 2, or
3.
TUs in each of these spreadsheets were then categorized using the categories of sub-
segment inconsistency from the pilot study: noun, verb, adverb, punctuation, preposition,
word order, tag inconsistency, typographical error, or complete retranslation. In the case of
category 1, TUs were further categorized by ST inconsistency: capitalization, tags,
punctuation, or typographical error. The topics chosen for the follow-up qualitative study
were based on these results.
Qualitative Phase
The second phase of this study is a series of qualitative interviews with translators and
others in the localization industry with experience of using TM tools in order to find the
causes of inconsistency and methods of minimizing inconsistency. These are in the form of
face-to-face personal interviews or, where this was not possible, telephone interviews,
seeking opinions on results and conclusions reached in the quantitative phase of the study.
Interviewees who are translators are usually native speakers of the target language who
may also review and edit draft translations done by others for quality assurance purposes.
Semi-structured interviews, usually allowing probes once the interviewee has begun to
answer, are the “most common qualitative strategy used in mixed method design” (Morse
and Niehaus, 2009, p.127). Questions were scripted and standardized as this means the
interview is highly focused and makes responses easy to compare (Quinn Patton, 2002,
p.346). It was hoped that this would minimize the effect of the interviewer, while allowing
for the interviewer to remain active, that is, to prompt or ask for further explanation if
necessary, without pursuing unanticipated topics (Quinn Patton, p347). Even a tightly
scripted interview cannot be devoid of input from both parties. As the discourse is “shaped
by prior exchanges between interviewer and respondent” (Mishler, 1991, p.53) all
participants are “inevitably involved in making meaning” (Gubrium and Holstein, 2003, p.78).
Ethics approval for the interviews was sought and received from the university Research
Ethics Committee several months prior to beginning the qualitative phase. The committee
confirmed that this research qualified as a low-risk social research project, and granted
approval. In doing so, they stipulated that interviewees should receive a Plain Language
Statement, explaining the background to the research, its intended purpose, the voluntary
nature of interviewees’ participation, and informing the participants that they may
withdraw at any time. Thereafter, interviewees signed an Informed Consent Form to
confirm that they understood the information contained in the Plain Language Statement,
that they were aware that interviews were to be recorded, and that any further questions
had been answered satisfactorily.
For interviewee selection, purposive sampling was used. This means that subjects who
would provide the most detailed information about the research questions were chosen,
emphasizing their depth of knowledge rather than seeking a large sample of respondents.
Purposive sampling is associated with qualitative research and provides narrative data.
Researchers using purposive techniques tend to minimize the sample size, selecting only
cases that might “best illuminate and test the hypothesis of the research team” (Kemper,
Stringfield, and Teddlie, 2003, p.279). The subtype of purposive sampling in this case was
homogeneous cases sampling, which aims to gather opinions from people who are
“demographically, educationally, or professionally similar” (Kemper, Stringfield, and Teddlie,
p.282). In this case, those sampled had worked as translators or with TM data for at least
five years and were considered professionally similar. Selection of subjects may influence
the validity of research, but in this case it was felt that translators (or those who work with
TM) would be most accustomed to searching for inconsistencies in translations that had
been overlooked by other translators, and best able to describe phenomena found in the
TM data.
Thirteen interviewees took part, having responded to calls for potential interviewees
circulated via email and Twitter, or having been approached at translation industry events.
Face-to-face interviews were recorded using the Voice Recorder application on a
smartphone. Where this was not possible, interviews were carried out remotely via Skype
(version 2.2 for Linux) and recorded using Skype-Recorder version 0.8.
Each interview was limited to one hour due to constraints on respondents' time.
Interviewees were told of potential uses of the research findings, how their identities and
that of their company would be anonymized, and assured of secure storage of the data. The
interviewees were asked questions based on the findings from the quantitative study.
The interviewer practiced active listening throughout the interviews (Kvale and Brinkmann,
2009, p.138), consciously analyzing replies and offering affirmation in order to create a
rapport with the interviewee. In the case of an ambiguous response from the interviewee,
the interviewer endeavored to interpret the statement to the interviewee's satisfaction, so
that incorrect interpretations could be ruled out. Remaining ambiguities were cleared up by
email contact with the interviewee at the analysis stage. At the end of the interviews,
interviewees were asked if they had any other comments or suggestions, then offered
transcripts of the interview, not only as a matter of courtesy, but also in case they wanted to
change or add to an answer. Only one interviewee asked for a transcript and no
interviewees requested changes. Interviewees were later sent a copy of the final study to
ensure that their input had been fairly represented.
Interviews took place between December 2011 and February 2012 and recordings were
transferred to a password-protected PC. Interview data was then transcribed to a document
following playback with VLC. Transcription is a significant stage in processing interview data,
transforming the narrative mode from oral to written discourse and de-contextualizing the
interview conversation (Kvale and Brinkmann, 2009, p.178). This step necessarily involves
interpretation of meaning and associated choices, such as where to place punctuation,
which can substantially change the content.
Following transcription, top-down or concept-driven coding was applied, in that the
responses were categorized by questions which were based on the five prescribed themes
identified prior to the interviews: general opinions on TM, opinions of inconsistency, ST
inconsistency, TT inconsistency, and the future of TM. For sections that digressed from the
initial themes, bottom-up or data-driven coding was applied, allowing the interview data to
set the theme. In such cases, sections were labeled according to their topics (Richards, 2005,
p.88).
Coding was done using NVivo 9 qualitative analysis software in several steps: Transcripts
were first imported into NVivo, these were then coded by interviewee, by question, and
finally by themes that emerged over the course of the interviews. Each interview was
assigned attributes signifying the interviewee’s job, gender, first language, and main TM tool
so that queries could be refined using these attributes.
In attempting to glean data-driven themes from the interview material, the method that
Kvale and Brinkmann term 'bricolage' was used. This involved reading through the
interviews to get an overall impression, to generate meaning, and to capture key
understanding (2009, p.234). As is typical when coding with Nvivo, emergent themes were
gathered as free or open codes. These open codes were then sorted into a hierarchy of
branching “tree nodes” to reflect the “structure of the data” (Bazeley, 2007, p.100). Aside
from adding organization to the open codes, the sorting stage is also said to prompt the
user to code thoroughly, to improve conceptual clarity, and to help to identify patterns and
connections (Bazeley, p.104).
Figure 1: Relationship of Interview Questions and Emerging Themes
The first two questions were about the interviewee's background. Questions three and four
were intended to be quite broad, seeking the interviewee’s opinion of the benefits and
disadvantages of TM, leading to the effect of TM on consistency and whether they felt
consistency was important, in addition to their own experiences of consistency issues. Broad
introductory questions may yield spontaneous and rich descriptions of the interviewee’s
experience of the phenomena investigated (Kvale and Brinkmann 2009, p135). Question five
related to ST inconsistencies such as those found in categories 1 and 2. Question six
concerned specific types of inconsistency found in the data (see the following section for
details of these findings):
• TT noun or term inconsistency - the largest category of sub-segment inconsistency
found in the study, also the prevalence of Anglicization in the target language, and
suggestions to improve terminological consistency
• Inconsistencies of ST format retention in the TT
• Alternation of whole phrases throughout the TM
• Explicitation in Japanese TT
Interviewees’ responses to the specific phenomena referred to in questions five and six
should confirm that the phase one results were accurately represented, further justifying
the choice of a mixed methods approach. Finally, question seven gave the interviewee an
opportunity to suggest ways of improving how TM tools deal with consistency issues in
future. Table 2 contains a profile of the interviewees.
Job Title
First Language
Interviewee A
Translator
Brazilian Portuguese
Interviewee B
Translator
German
Interviewee C
Translator
Spanish
Interviewee D
Translator
French
Interviewee E
Translator
Brazilian Portuguese
Interviewee F
QA Specialist
English
Interviewee G
QA Specialist
Spanish
Interviewee H
Language Technology Consultant
English
Interviewee I
Project Manager
German
Interviewee J
Project Manager
Japanese
Interviewee K
Chief Operating Officer
Spanish
Interviewee L
Workflow Manager
French
Interviewee M
Software Localization Engineer
Spanish
Table 2: Interviewees from qualitative phase
Results
Among the four TM corpora studied, between 2 and 5% of the TUs contained ST segments
that were repeated with minor inconsistencies, falling into categories 1 and 2. Figure 1
shows the ST inconsistencies found in the data.
Figure 2: Categories of ST inconsistency in all TMs
The number of category 1 and 2 TUs found in the four sets of TM data differed, but in all
four TMs the most prevalent type of ST inconsistency was letter case or capitalization of
words. All but one of the interviewees (E) said that they had seen many instances of ST
inconsistencies as shown in figure 1. Interviewee G, a QA specialist, said that every time he
checks a TM “I have the same translation for different source texts that should have been
the same”. He continued “the TM technology… the quality issue of technology is there, but
the source text is the problem”. Eight interviewees suggested the attitude of some of their
clients as a cause of ST inconsistency. They felt that the focus of clients is usually on time
and cost savings, which leads to source text that is hurriedly written, often by a non-native
source language speaker, and is thus ambiguous and inconsistent. They said that educating
their customers about the effect of inconsistent ST on the translation process is one way in
which they attempt to minimize inconsistency. Several interviewees said that this education
usually proves difficult. For example, K, a COO, commented that this is “one of the biggest
fights that everyone in the localization industry has to fight with clients” to make them
understand that “if you don't control your English… you can't possibly expect to have
savings and leverage over time with the TMs”. Four interviewees (G, H, L, and M) have
returned to customers with an assessment of the financial repercussions of ST inconsistency
in order to “show just how much money they're wasting”, and to tell clients that “they could
do things a hell of a lot cheaper and a lot better” (H; language technology consultant). Five
interviewees said that inconsistent ST segments were, in their opinion, a cause of further TT
inconsistency.
Category 1: Inconsistent source and target text segments
Table 3 shows the most commonly-occurring types of category 1 ST inconsistency.
TM A
TM B
TM C
TM D
Letter case
60
13
314
753
Punctuation
48
3
60
73
Tags
42
10
46
137
Space
20
8
37
68
Total Segments
370
65
995
1980
Total
Subsegment
Inconsistencies
174
35
475
1042
Table 3: Inconsistencies found in category 1 source text segments
Category 1 TUs contain minor inconsistencies (as specified in our typology) in the source
segment and other kinds of inconsistencies in the aligned target segment. In category 1 TUs,
none of the TT segments aligned with ST segments that contain minor inconsistencies
themselves contain instances of the same kind of inconsistency; rather the TT segments in
question evince other kinds of inconsistencies, as in example 1 (with differences highlighted
in bold) from TM A:
1.1s Add the Macro Created in Exercise
1 to a Toolbar
1.1t Makro aus Übung 1 in eine Symbolleiste
aufnehmen
1.2s Add the macro created in exercise
1 to a toolbar
1.2t In Übung 1 erzeugtes Makro in eine
Symbolleiste aufnehmen
Segments 1.1s and 1.2s contain inconsistent letter case. Although there has been no
semantic change in the ST, the word order has been changed in the TT from ‘add the macro
from exercise 1 to a toolbar’ in segment 1.1t, adding the genitive form in segment 1.2t: ‘add
the macro that was created in exercise 1 to a toolbar’.
We found a high incidence of inconsistent placing of the space character, especially in TM D.
These spaces were initially noticed by automatically comparing the ST segment and the
following, seemingly identical, ST segment, as 54 of the 68 space inconsistencies in TM D
were at the end of the segment following a full stop. Again, the aligned target text (TT)
segments contain other kinds of inconsistencies, as in the following example from TM A:
2.1s Help Center includes both web-based
HTML and PDF versions of the information
and is accessed from the {1}{2}Help{3} {4}Help
Center{5}{6} menu.
2.1t Das Help Center enthält webbasierte
HTML- und PDF-Versionen der Informationen
und kann über das Menü{1}{2}Hilfe{3} {4}Help
Center{5}{6} aufgerufen werden.
2.2s Help Center includes both web-based
HTML and PDF versions of the information
and is accessed from the {1}{2}Help{3}{4}Help
Center{5}{6} menu.
2.2t Das Hilfe-Center beinhaltet webbasierte
HTML- und PDF-Versionen der Informationen
und kann über das Menü{1}{2}Hilfe{3}{4}Help
Center{5}{6} aufgerufen werden.
The ST segment contains a space inconsistency between the tags numbered 3 and 4, while
the aligned TT segments contain an inconsistent noun and verb. The use of ‘Help Center’ in
2.1t shows some influence from the source language, which can be seen throughout the
TMs.
Table 4 shows the most commonly-occurring types of category 1 TT inconsistency.
TM A
TM B
TM C
TM D
Noun
84
15
323
616
Punctuation
3
5
57
129
Verb
45
3
66
75
Space
6
0
38
141
Word Order
41
1
81
3
Preposition
12
N/A
138
N/A
Particle
N/A
1
N/A
42
Total Segments
370
65
995
1980
Total
Subsegment
Inconsistencies
240
33
730
1291
Table 4: Inconsistencies found in category 1 target text segments
In Table 4, the number of subsegment inconsistencies may be seen to be larger than that in
Table 3. This is because a segment may contain up to three subsegment inconsistencies
before we consider it completely retranslated. Among category 1 TUs a large proportion of
noun inconsistencies were found, comprising between 36% and 48% of the total number.
For example, in TM C there were 323 noun inconsistencies of which 12 showed influence of
the source language in one instance but not in another, and 87 contained inconsistencies of
capitalization or case, as per the following example:
3.1s At the Command prompt, enter subtract.
3.1t Geben Sie in der Befehlszeile differenz
ein.
3.2s At the command prompt, enter subtract.
3.2t Geben Sie an der Eingabeaufforderung
DIFFERENZ ein.
Example 3 also displays a phenomenon that accounts for the high prevalence of preposition
inconsistencies in TM C. 138 preposition inconsistencies were found in category 1, just
under 20% of the total. 126 of these preposition inconsistencies (and thus 18% of the total)
are secondary changes as required by the change of noun, thus we see an alternation
between the phrases 'in der Befehlszeile' (in the command line) and 'an der
Eingabeaufforderung' (at the command prompt).
In the qualitative phase, 11 of the 13 interviewees said that they often find similar
inconsistent phrases propagated in TMs. K (COO) said that she sees these kinds of
inconsistencies “day in, day out”, where there is “a new member of the translation team
who will reckon he or she has a better solution”. If this new team member ignores the
suggested match and rewrites the TT segment, “it's just so easy to bring in or upload a new
version onto the TM and there's nothing that stops it”. F (QA specialist) also said that he
sees this “all the time” and suggested that it may be the result of translators working
independently without adequate terminological guidance. Also, with the software that his
company uses, for fuzzy matches “it’s up to the translator to identify where that fuzziness is
and correct it, and sometimes they just over-correct”. Despite the importance of
consistency in his domain of medical device translation, he says “we wind up with this all
the time”.
Category 2 TUs (Inconsistent ST segments with consistent TT segments)
TM A
TM B
TM C
TM D
Letter case
140
219
480
505
Space
95
96
287
247
Punctuation
67
4
89
153
Inconsistent
word (Noun)
11 (3)
98 (86)
146 (70)
194 (77)
Total Segments
613
914
2077
1801
Total
Subsegment
Inconsistencies
316
450
1043
1123
Table 5: Inconsistencies found in category 2 segments
Category 2 TUs contain ST inconsistency and thus introduce consistency in the TT. The
majority of these ST inconsistencies in all TMs analyzed were inconsistent letter case. The
following example from TM B is typical of this ST inconsistency.
3.1s Case sensitive
3t 大文字と小文字の区別
3.2s Case Sensitive
3t 大文字と小文字の区別
As Japanese characters do not vary by letter case, this ST inconsistency has been removed in
the TT. In order to convey the concept ‘case sensitive’ in Japanese, the translator has
chosen ‘the distinction between upper and lower case letters’ in the TT. Although strict rules
on capitalization in German means introduced consistency would also be expected, there
are instances of the ST letter case being retained in the TT in all of these TMs (roman
lettering is sometimes used in the Japanese TT), particularly if the ST segment is in upper
case. This means we have a mix of transposed ST grammar, punctuation or formatting and
native TT formatting in TT segments. In the following example from the same TM,
containing a ST space inconsistency similar to those found in all four sets of TM data, we see
a German noun written (incorrectly) in lower case:
4.1s {1}securityoptions {2}
4t {1}sicherheitsoptionen{2}
4.2s {1}securityoptions{2}
4t {1}sicherheitsoptionen{2}
Twelve of thirteen interviewees had seen this sort of inappropriate target language
formatting frequently. I (Project Manager) suggested that a formatting issue such as the one
in segment 4t occurs when no match is suggested, in which case the translator might copy
the ST to the target segment and overwrite it without considering whether the formatting is
“actually compliant with German rules”. She said that she would deal with this by compiling
a style guide specifying what formatting to use, and that “that's something you have to
clarify up front, even if you have the best TM ever”.
Inconsistent punctuation is usually to do with the presence or absence of commas or full
stops in the ST which may or may not be retained in the TT. The following example from TM
D contains a punctuation inconsistency, but also contains an example of a section that has
been marked out in the TT, followed by a comment by the translator, explaining that he
chose the term 塗り潰し色 'nuritsubushiiro' for filling in colors. This comment was
subsequently propagated within the TM.
5.1s {1} If None (Color) is selected as the Map
Type then the color needs to be selected.
5t {1} [マップの種類]として[###塗り潰
し色]を選択した場合は、色を選択する必
要があります。■3-(B037)「なし」とい
う選択肢はなく、「塗り潰し色」という
選択肢が表示されるので、このようにし
ました。(Koizumi 06/11/21)
5.2s {1} If None (Color)) is selected as the Map
Type then the color needs to be selected.
5t {1} [マップの種類]として[###塗り潰
し色]を選択した場合は、色を選択する必
要があります。■3-(B037)「なし」とい
う選択肢はなく、「塗り潰し色」という
選択肢が表示されるので、このようにし
ました。(Koizumi 06/11/21)
There are a number of reasons why ST inconsistencies may be ignored by a translator who
chooses instead to accept a fuzzy match. Foremost among these in Japanese is the presence
of plurals in the ST. In our study of English to German TMs, plurals did not register in our
categories, as we considered that the ST had formally changed and thus accepted that the
TT would be inconsistent. However, as there is no distinction between singular and plural in
Japanese – numbers are given explicitly or are implicit in context – we can expect to see
plural and singular nouns translated consistently in the Japanese TT, and this is indeed the
case. Of 76 cases of inconsistent nouns in the ST segments of category two TUs in TM B, 42
differ in number: singular in one case, plural in another, as in example 6:
6.1s Dimension
6t
寸法
6.2s Dimensions
6t 寸法
All twelve of the interviewees with experience of ST inconsistency said that they revert to
clients with problems and one even tends to suggest a third party, a technical writing
consultancy firm, to assist clients with their ST consistency. F, a QA specialist, agreed that ST
consistency is important: “If they (clients) can't control their source text, then we can't be
expected to control the target text for them”.
Category 3 TUs (Consistent ST segments with inconsistent TT segments)
In the four TMs in this study, all four contained TT inconsistency introduced at a rate of
approximately 5% where the ST segment was repeated exactly. The types of TT
inconsistency found in categories 1 and 3 may be seen in figure 2.
Figure 3: Categories of TT inconsistency in all TMs
TM A
TM B
TM C
TM D
Noun (SL
influence)
81 (18)
49 (9)
282 (31)
365 (77)
Verb
34
40
30
59
Punctuation
12
8
44
183
Space
1
1
61
272
Explicitation
1
6
3
32
Preposition
7
N/A
112
N/A
Particle
N/A
15
N/A
57
Completely
rewritten (Not
added to total)
3
7
9
35
Total Segments
390
239
826
1713
Total
Subsegment
Inconsistencies
174
129
570
1035
Table 6: Inconsistencies found in category 3 segments
Category 3 contains TUs with inconsistent TT segments, where inconsistency has been
introduced in the TM data. Again, the most prevalent category of TT inconsistency was noun
inconsistency, as may be seen in table 6 (and figure 3). In TM A we found 81 inconsistently
translated nouns (47% of the inconsistencies) of which 18 showed influence of the English
source language in one instance as in example 7:
7s All lines that have been converted using
the {1}Create surface borders{2} function
can be recognized easily since they are
drawn with the {3}Border{4} pen.
7.1t Alle Linien, die mit der Funktion
{1}Flächenränder anlegen{2} konvertiert
wurden, können Sie leicht erkennen, da sie mit
dem Stift mit der Bezeichnung {3}Border{4}
gezeichnet werden.
7s All lines that have been converted using
the {1}Create surface borders{2} function
can be recognized easily since they are
7.2t Alle Linien, die mit der Funktion {1}
Flächenränder anlegen{2} konvertiert wurden,
können Sie leicht erkennen, da sie mit dem Stift
drawn with the {3}Border{4} pen.
mit der Bezeichnung {3}Rand{4} gezeichnet
werden.
This alternation between 'Border' and 'Rand' occurred three times in the TT and was one of
several patterns that emerged within the data.
All 13 interviewees agreed that this was a phenomenon that they saw often. E (translator)
suggested that these inconsistencies may be caused by having several translators working
on the same material without a follow-up consistency check to “catch this before it is
propagated to the TM”. C (translator) suggested that this problem may be caused by
merging TMs from different sources. She often receives TMs that contain inconsistent
terminology, so that “when you look for a term in a translation memory, you do a
concordance search”, in which case she often finds “two, three, or four different
translations” for the same term.
L (workflow manager) suggested that inconsistencies such as in example 7 could emerge
even with terminology databases as a result of the conflict between approved terminology
and search engine optimization (SEO). In this situation, a German translation for ‘border’
may have been “approved and reviewed”. Despite this, the client may have realized that “in
German, people don't actually go to google.de and look for the German translation, but they
actually look for the English words”, so when they start optimizing their German website,
they might decide that “the approved term is actually not what's going to get them the
hits.” This is one explanation for the prevalence of noun inconsistencies such as in example
7, featuring a native German word in one instance and a borrowed English word in another.
Non-translator interviewees felt that new or inexperienced translators on a job tend to add
new translations to the memory. K (COO) said that translators may also choose to accept
inappropriate matches. If inconsistencies exist in the TM, she continued, one cannot expect
consistency to be increased in the TT via “human decisions”.
Nine of the thirteen interviewees highlighted the need for terminology management as a
method of minimizing noun inconsistencies. Terminology management is a function of most
current TM tools, although TT term inconsistencies were still found in all four TMs, which
our interviewees all said are commonplace in their experience of TMs. M (software
localization engineer), warned that the use of TM without glossaries means “you're going to
end up with inconsistencies, mistranslations, and some [other] issues”. F (QA specialist) uses
glossaries as his company “can't afford to care about what happens to the syntax and the
grammar and everything else”, rather they just “have to focus on terminology being
consistent because that's what the clients really care about”. H (language technology
consultant) said that her clients, on the other hand, pay insufficient heed to terminology,
and she considers it a “gift from heaven” if she can get “20, 50 words that … you guarantee
in your documentation. According to her, “customers are not interested, don't appreciate,
do not understand; they're not willing to pay for terminology work”.
The Japanese TT in TM B again showed detail being added in translation that was not in the
ST. Ten inconsistencies in example 8 were translations of the word 'selecting'.
8s Selecting
8.1t エレメントの選択
8s Selecting
8.2t コールアウトエレメントの選択
8s Selecting
8.3t アセンブリの選択
8s Selecting
8.4t 多角形の選択
8s Selecting
8.5t 線の選択
8s Selecting
8.6t 楕円の選択
8s Selecting
8.7t 選択
8s Selecting
8.8t 長方形の選択
8s Selecting
8.9t ベジエ曲線の選択
8s Selecting
8.10t めねじの選択
8s Selecting
8.11t おねじの選択
In the example above 8.1t is taken as the reference translation as it appeared first in the TM
data. Each TT segment contains the noun 選択 (sentaku) meaning 'selection' but most
involve further explicitation. 8.1t is エレメントの選択 or 'selection of elements'. Segment
8.2t is コールアウトエレメントの選択 or 'selection of callout elements'. While this
explicitation may make the TT segments clear and understandable, it has a negative effect
on leverage. It may be in this case that the first translation contained added detail that was
not appropriate for the subsequent translations.
Nine interviewees said that they saw this phenomenon frequently, not just in Japanese, but
in German (B and I), Brazilian Portuguese (A), Romanian (H), Spanish (K), and Malay (K). K
(COO) felt that the ST in segment 8s is unclear). She suggested that the translator needs
more details as “it's critical that they [ST segments] are taken in context”. She also
introduced the topic of client expectation, saying that customers need to be told that “they
might be 100% match, but they're not perfect matches all the time”. Her policy is not to lock
100% matches to prevent new translations, but she tells translators not to touch 100%
matches unless absolutely necessary. In the case of incorrect 100% matches it’s left to the
assiduousness of her translators as to whether they revert to her with problems or leave the
100% matches untouched. If the context has changed, then the introduction of some
inconsistency in TT may be necessary.
After noun inconsistency, the next most prevalent type of inconsistency in TMs A and B is
verb inconsistency. Of the 40 verb inconsistencies contained in TM B, 18 of them contained
another repeated pattern, alternating between using the verb 拘束する (kousoku suru)
meaning 'to bind or restrict' in one case, and the verb 関連付ける (kanren tsukeru)
meaning 'to relate' in another. For example:
9s You can bind {1} XML elements by their
corresponding ISO elements’ {2}Object
type{3}, object {4}ID{5}, or object {6}Name{7}.
9.1t {1} XML エレメントを拘束するには
、これらに対応する ISO エレメントの
{2}「オブジェクトタイプ」{3}、オブジ
ェクトの{4}「ID」{5}、またはオブジェ
クトの{6}「名前」{7}を使用します。
9s You can bind {1} XML elements by their
corresponding ISO elements’ {2}Object
type{3}, object {4}ID{5}, or object {6}Name{7}.
9.2t {1} XML エレメントを関連付けるに
は、これらに対応する ISO エレメントの
{2}「オブジェクトタイプ」{3}、オブジ
ェクトの{4}「ID」{5}、またはオブジェ
クトの{6}「名前」{7}を使用します。
Segment 9.1t translates as ‘You can bind {1} XML elements by their corresponding ISO
elements’, segment 9.2t as ‘You can relate {1} XML elements by their corresponding ISO
elements’. Looking through the metadata, each verb choice is not attributable to a single
user ID, but the translations using 拘束する were all saved to the TM at the same time on
April 22nd 2009. Two uses of 関連付ける were also saved then, but all others were dated
from the 7th of May in 2009. At that stage, one would presume, the TM tool used must have
suggested the previously translated TT segment as a 100% match.
The other Japanese data, TM D, also contain a repeated pattern, alternating between the
borrowed English word レイヤ and the native Japanese word 画層 'gasou' 41 times as per
example 10:
10s A new layer group filter can be nested
only under another group filter.
10.1t 新しいレイヤグループ フィルタは
、他のグループ フィルタに対してのみネ
ストできます。
10s A new layer group filter can be nested
only under another group filter.
10.2t 新しい画層グループ フィルタは、
他のグループ フィルタに対してのみネス
トできます。
Three interviewees said that they had experienced problems with inconsistencies such as in
example 10 in Japanese TMs, whereby in one case a kanji compound was used and the
phonetic katakana loan word was used in another. J (PM), a native Japanese speaker, felt
that “the trend is now phonetical translation, so that's inconsistency just depending on the
translator preference, so we have to give them guidelines, style guidelines”.
TM D contains a large number of punctuation inconsistencies. Many of these (23) are
marked out using the # symbol, others have inconsistently placed quotation marks, and
many show indecision as to whether or not to retain ST formatting for commas or full stops
as in the following example:
11s Accesses Dimensioning mode
11.1t 寸法記入モードにします
11s Accesses Dimensioning mode
11.2t 寸法記入モードにします。
The high rate of preposition inconsistency in TM C is again a secondary effect of noun
inconsistency as shown previously in example 3. The inconsistencies of particle in Japanese
are also often secondary to a change in verb or verb form from active to passive or, as in the
following example, required by verb choice with the 表現 (scale representation) taking the
particle 'ga' when the verb 'aru' (to exist) is used, and the direct object particle 'wo' with
'motsu' (to hold).
12s Annotative objects may have multiple
{1}scale representations{2}.
12.1t 異尺度対応オブジェクトには複数
の{1}尺度表現{2}がある場合があります
。
12s Annotative objects may have multiple
{1}scale representations{2}.
12.2t 異尺度対応オブジェクトには、複
数の{1}尺度表現{2}を持つものもありま
す。
Category 4 TUs (Consistent ST segments with consistent TT segments)
These TUs are those that we consider to have been translated consistently. By looking at the
number of repeated TUs that fall into this category, we can see the overall rate of
introduced TT inconsistency within a TM as per the following table.
TM A
TM B
TM C
TM D
Category 3 TUs
390
239
826
1713
Category 4 TUs
6674
4263
18343
25541
Total TUs with
repeated ST
segments
7064
4502
19169
27254
Percentage of
TUs with
introduced
inconsistency
5.5%
5.3%
4.3%
6.3%
Table 7: Introduced Inconsistency in all TMs
While interviewees in the qualitative phase of this study agree that the inconsistencies
exemplified here are common, in their experience, there are also instances where
inconsistency is required. The interviewees would like “maximum consistency” (K), yet have
problems with clients’ assumptions that all 100% matches can be automatically accepted.
Eight interviewees said that 100% matches may be erroneous, a point previously made by
Reinke (2004). Several interviewees (particularly non-translators) felt that some TT
inconsistency may be necessary. F is “guided by my translators” as to whether it might be
better to introduce inconsistency. He said that, for him, it is more important for a
translation to be “accurate and natural and fluent than it is for the resulting translation unit
to be recyclable infinitely in all other documents”, adding that this loss of leverage is “a
sacrifice we have to make”.
Conclusion
Figure 4: Computer-Aided Translation Workflow
Following a simple workflow for translation using TM (see figure 4), the study found
inconsistencies in TM that interviewees blamed largely on the client for failing to control ST.
At the end of the translation process, the level of quality or consistency is again dependent
on the client and whether they feel it more important to make short term savings, as
translation providers may find it necessary to “skip quality steps to offer lower rates,
especially if the client is not measuring quality” (Kelly, 2012, p.2). If insufficient heed is paid
to quality, then inconsistency will be propagated in the TM.
At the translation stage, the quantitative study showed that inconsistency is introduced.
Interviewees felt that this was due to insufficient terminological assistance within the tools,
but also that translators may make different decisions, especially when inconsistent source
text (ST) leaves them the option of editing the target text (TT). Conversely, not all 100%
matches may be appropriate for reuse, although the interviewees said that this is again
related to ambiguous ST. If the ST is clear, the TT should be further recyclable. The
inconsistencies of grammar and formatting in the TT showed the importance of a style guide
to tell the translator whether to follow ST or, more usually, TT-appropriate grammar rules.
At the end of the translation process, translations are usually reviewed, but the
interviewees claimed that these final edits are rarely added to the TM, leaving inconsistency
to be further propagated in the next translation project using the same TM.
In the introduction, we discussed the cost associated with inconsistency. We estimate that,
for an average-sized translation project of 50,000 segments, each translated into 20
languages, the introduction of category 3 inconsistencies at a rate of 5% will add a cost of
over $21,000 to the subsequent iteration of the project. In addition, inconsistent source text
will result in low match results, incurring an extra cost, and inconsistent content means
more time to be spent by engineers and reviewers in quality assurance (QA) at the end of a
project. Many companies, particularly in the fields of in life sciences or medical device
translation, spend a lot of time on measuring quality post-translation. Inconsistent content
will add time and expense to this QA process.
As there is no one correct way to write a translated text or segment, new target segments
are continually added by translators. This is referred to by the Business Development Officer
of a prominent translation software company (SDL) as “accidental content” (Cronin, 2013,
p.37). Efforts to maximize consistency thus constrain the translator so as to minimize this
accidental content. Our suggestions in this study involve standardization of source texts,
which will maximize leverage if and when those source texts are repeated. We suggest
employing style guides, which have been shown to be time-consuming to create, but are
beneficial tools to limit target texts. As most of the introduced inconsistencies in this study
were noun inconsistencies, maintaining a tight control on terminology should improve
overall consistency a great deal. The interview data also suggests that the use of small,
bespoke TMs, prioritizing precision over recall, will remove the chances of inappropriate
matches being suggested by the TM tool.
Although the focus in this study was on the quantitative phase, the guidelines for improved
translation processes are drawn largely from the qualitative phase, based on interviewees’
knowledge and experience. The descriptive analysis in phase one may be presented as a
standalone work, but has little impact without prescriptive measures, and as phase two
relies on the quantitative data, the potential effectiveness of this study is due to the choice
of mixed methods. The validity of research conducted within the pragmatic paradigm is
judged on the effectiveness of the work and the actions it engenders once completed and
disseminated (Kvale and Brinkmann, 2009). A challenge lies in dissemination of key findings,
which will need to reach the localization industry in order to be effective.
This study contributes to the mixed methods literature by applying mixed methods in a field
where studies have tended to be quantitative (such as those focused on incremental
improvements to machine translation systems) or qualitative (such as those exploring the
reception of translation tools). Although some prior work on TMs has yielded quantitative
and qualitative results from within a single study (Lagoudaki, 2008), this study connected
both, adding substantially to the external validity of the study and showing that quantitative
analysis of corpora may be successfully combined with qualitative interviews in the field of
translation studies. Subsequent mixed methods research (Doherty, 2012; Guerberof, 2012)
has appeared as the focus in translation studies has moved towards technology and applied
research. This increase in published mixed methods studies is in line with the increased
prevalence rates for applied disciplines as reported by Alise and Teddlie (2010, p121), and
reflects the need for user input as translation technology more widely applied commercially.
Strengths and Limitations of this Research
This study began in 2008, since when there has been a great deal of change in TM tool
development. As technology is in a state of constant change and development, the current
research is limited by focusing on an area that dates quickly. The study was carried out by
one person; it was limited by the researcher’s language competence, restricting the study to
English, German, and Japanese. As a result, it may be useful to replicate the methodology
herein with different language pairs to see whether the results are repeated. The people
interviewed in the second phase work predominantly in translation of IT documentation,
which may have impacted on their views. The sample was limited to those who received and
responded to our call for participants via email, social media, and at industry events.
The choice of a mixed methods design introduced a threat to validity at the stage between
the two main phases of research, when the researcher must decide what results from the
first phase to pursue in the second phase. In this study, topics chosen for the qualitative
phase were based on the types of inconsistency that were most prevalent in the
quantitative results, but also on phenomena that was seen to have occurred across
categories, such as whole phrase inconsistencies, in order to accurately represent the data.
Future Work
Findings from this study have been used to envisage realistic scenarios in which leverage is
lost due to inconsistencies in the TM applied in a hypothetical job. This necessitates making
a number of assumptions about the project, but the creation of such scenarios can help
make the financial case for ensuring consistency (where possible) in TMs, despite the cost
associated with doing so. If cost-benefit analyses can demonstrate that it is financially
worthwhile to implement measures that promote consistency, then the major obstacle to
doing so (clients’ perception of cost) may be removed.
As more machine translation is integrated into the localization process, this study also forms
part of a larger discussion of consistency in translation technology, suggesting that
consistency of bilingual resources will continue to be an issue into the future and that the
“quality control of TMs needs to become much more sophisticated” (Zetzsche, 2012, p.51).
One follow-on project attempted to automate the removal of inconsistency with mixed
results (Moorkens, Doherty, Kenny, and O’Brien, 2013). Future work may also involve
further interviews to include other stakeholders in the translation process, such as clients
and end-users of products, and translators of content from domains other than computing.
References
Alise, M. A. & Teddlie, C. (2010). A continuation of the paradigm wars? Prevalence rates of
methodological approaches across the social/behavioral sciences. Journal of Mixed Methods
Research, 4(2), 103-126.
Austermühl, F. (2001). Electronic tools for translators. Manchester: St Jerome.
Bazeley, P. (2007). Qualitative data analysis with NVivo. London /Thousand Oaks, CA: Sage
Publications.
Bowker, L. (2005). Productivity vs quality? A pilot study on the impact of translation memory
systems. Localisation Focus, 4(1), 13-20.
Creswell, J. W. & Plano Clark, V. (2007). Designing and Conducting Mixed Methods Research. London
/Thousand Oaks, CA: Sage Publications.
Cronin, M. (2013). Translation in the digital age. New York : Routledge.
Doherty, S. (2012). Investigating the effects of controlled language on the reading and
comprehension of machine translated texts: A mixed-methods approach. Unpublished doctoral
dissertation. Dublin City University.
Frey, L. R., Botan, C. H., Freidman, P. G. & Kreps, G. L. ( 1991). Investigating communication: an
introduction to research methods. Cornell University: Prentice Hall.
Gow, F. (2007). You must remember this: the copyright conundrum of "translation memory"
databases. Canadian Journal of Law and Technology, 6(3), 175-192.
Gubrium, J. F. & Holstein, J. A. (2003). Active interviewing. In Gubrium, J. F. & Holstein, J. A. (Eds.),
Postmodern Interviewing (pp67-80). London /Thousand Oaks, CA: Sage Publications.
Guerberof Arenas, A. (2012). Productivity and quality in the post-editing of outputs from translation
memories and machine translation. Unpublished doctoral dissertation. Universitat Rovira i Virgili.
Kelly, N. (2012). What to do when translation vendors raise prices. Common Sense Advisory Article.
Boston, MA: Common Sense Advisory.
Kvale, S. & Brinkmann, S. (2009). InterViews: Learning the craft of qualitative research interviewing.
London /Thousand Oaks, CA: Sage Publications.
Lagoudaki, E. (2008). Expanding the Possibilities of Translation Memory Systems: From the
Translator’s Wishlist to the Developer’s Design. Unpublished doctoral dissertation. Imperial College,
London.
Lindlof, T. R. & Taylor, B. C. (2002). Qualitative communication research methods. London /Thousand
Oaks, CA: Sage Publications.
Merkel, M. (1998). A study of consistency and variation in technical translations: A study of
translators’ attitudes. In Bowker, L., Cronin, M., Kenny D. & Pearson, J. (Eds.), Unity in diversity?
Current trends in translation studies (pp137-149). Manchester: St. Jerome Publishing.
Mishler, E. G. (1991). Research interviewing: context and narrative. Cambridge, MA: Harvard
University Press.
Moorkens, J., Doherty, S., Kenny, D. & O'Brien, S. (2013). A virtuous circle: Laundering translation
memory data using statistical machine translation. Perspectives: Studies in Translatology, 22:3.
Morse, J. M. & Niehaus, L. (2009). Mixed method design: Principles and procedures. Walnut Creek,
CA: Left Coast Press.
Olohan, M. (2011). Translators and translation technology: The dance of agency. Translation Studies
4(3), 342-357.
Onwuegbuzie, A. J. & Teddlie, C. (2003). A framework for analysing data in mised methods research.
In Tashakkori and Teddlie (eds.) Handbook of mixed methods in social and behavioral research (p
p351-384). London /Thousand Oaks, CA: Sage Publications.
Parmelee, J. H., Perkins, S. C. & Sayre, J. J. (2007). ’What about people our age?’ Applying qualitative
and quantitative methods to uncover how political ads alienate college students. Journal of Mixed
Methods Research, 1(2), 199.
Reinke, U. (2004). Translation memories: systeme – konzepte – linguistische optimierung. Frankfurt:
Peter Lang.
Ribas López, C. R. (2007). Translation memories as vehicles for error propagation: A pilot study.
Minor dissertation. Universitat Rovira i Virgili, Tarragona.
Richards, L. (2005). Handling qualitative data: A practical guide. London /
Thousand Oaks, CA: Sage Publications.
Rieche, A. C. (2004). Memória de tradução: auxílio ou empecilho? MA Thesis. Pontifícia Universidade
Católica do Rio de Janeiro.
Schofield, J. W. (2000). Increasing the generalizability of qualitative research. In Gomm, R.,
Hammersley, M. & Foster, P. (Eds.) Case study method: Key issues (pp69-97). London /Thousand
Oaks, CA: Sage Publications.
Smith, R. (2009). Copyright issues in translation memory ownership. In proceedings of ASLIB
Translating and the Computer 31. London, UK. 19-20 November 2009.
Somers, H. (2003). Computers and translation: A translator’s guide. Amsterdam / Philadelphia: John
Benjamin.
Stake, R. E. (1995). The art of case study research. London / Thousand Oaks, CA: Sage Publications.
Susam-Sarajeva, Ş. (2009). The case study research method in translation studies. In Ian Mason (Ed.)
Training for Doctoral Research, special issue of The Interpreter and Translator Trainer (ITT) (pp37-56),
Manchester: St Jerome.
Teddlie, C. B. & Tashakkori, A. (2009). Foundations of mixed methods research: Integrating
quantitative and qualitative approaches in the social and behavioral sciences. London /Thousand
Oaks, CA: Sage Publications.
Webb, L. E. (1998). Advantages and disadvantages of translation memory: A cost/benefit analysis.
MA Thesis. Monterey Institute of International Studies.
Zetzsche, J. (2012). Translation technology comes full circle. Multilingual Computing, 23(3), 50-51.
