Aljoscha Burchardt

• Dr. Phil.
• Project Manager at Deutsches Forschungszentrum für Künstliche Intelligenz

Introduction As healthcare is shifting from a paternalistic to a patient-centred approach, medical decision making becomes more collaborative involving patients, their support persons (SPs) and physicians. Implementing shared decision-making (SDM) into clinical practice can be challenging and becomes even more complex with the introduction of artif...

The growing adoption and deployment of Machine Learning (ML) systems came with its share of ethical incidents and societal concerns. It also unveiled the necessity to properly audit these systems in light of ethical principles. For such a novel type of algorithmic auditing to become standard practice, two main prerequisites need to be available: A...

Background Patients after kidney transplantation eventually face the risk of graft loss with the concomitant need for dialysis or retransplantation. Choosing the right kidney replacement therapy after graft loss is an important preference-sensitive decision for kidney transplant recipients. However, the rate of conversations about treatment options...

BACKGROUND Patients after kidney transplantation eventually face the risk of graft loss with the concomitant need for dialysis or retransplantation. Choosing the right kidney replacement therapy after graft loss is an important preference-sensitive decision for kidney transplant recipients. However, the rate of conversations about treatment options...

Zusammenfassung Unter Berücksichtigung der in den vorherigen Kapiteln erarbeiteten Rückschlüsse wird nun Bezug auf das Konzept der Meaningful Human Control (MHC) genommen. Grundprämisse dieses Konzeptes ist es, menschliche Kontrolle über Systeme auch dann zu ermöglichen, wenn diese einen gewissen Grad an Autonomie aufweisen. Das meint nicht immer n...

Zusammenfassung Im Folgenden werden Orientierungsmarker für den Einsatz von KI-Systemen zur Entscheidungsunterstützung im klinischen Alltag ausformuliert. Aus der Verbindung der Akteursperspektiven mit dem Konzept der Meaningful Human Control lassen sich Bedingungen für die Ausgestaltung eines soziotechnischen Phänomens herausarbeiten, denen durch...

Zusammenfassung In diesem Einleitungskapitel wird der Bedeutung nachgegangen, die dem Einsatz von Künstlicher Intelligenz (KI) in der Klinik zukommt. Der Einsatz von KI ist dort mit der Hoffnung verbunden, das Gesundheitssystem zu verbessern, bringt jedoch auch Herausforderungen mit sich. Sowohl die Leistungsfähigkeit als auch die Undurchsichtigkei...

Zusammenfassung Unter Einbezug der vorher dargestellten empirischen Ergebnisse und der ethischen Auseinandersetzung wird nun ein rechtlicher Rahmen für den Einsatz Künstlicher Intelligenz in der Klinik diskutiert. Dieser gliedert sich in verschiedene Akteursperspektiven und hat zum Ziel, beim Einsatz von KI-Systemen eine bedeutsame menschliche Kont...

Zusammenfassung In diesem Kapitel werden zwei Anwendungsfälle aus dem Bereich der Nephrologie vorgestellt. Die Anwendungsfälle bestehen aus zwei Experimenten, die unter der Begleitung qualitativer Erhebungen durchgeführt wurden. Aus der Interpretation der gewonnenen Daten ergeben sich Rückschlüsse darüber, welche Bedürfnisse Ärzt:innen und Patient:...

Scientific publications about the application of machine learning models in healthcare often focus on improving performance metrics. However, beyond often short-lived improvements, many additional aspects need to be taken into consideration to make sustainable progress. What does it take to implement a clinical decision support system, what makes i...

Artificial Intelligence-based decision support systems (AI-DSS) to improve hemodialysis therapy are currently under development. However, the influence of AI-DSS on shared decision making (SDM) in hemodialysis patients has not been studied so far. We performed a Wizard of Oz experiment, using a sham AI-DSS suggesting ultrafiltration volume at the b...

Introduction Artificial intelligence–driven decision support systems (AI–DSS) have the potential to help physicians analyze data and facilitate the search for a correct diagnosis or suitable intervention. The potential of such systems is often emphasized. However, implementation in clinical practice deserves continuous attention. This article aims...

Im Auftrag des Bundesministeriums für Wirtschaft und Klimaschutz haben DIN und DKE im Januar 2022 die Arbeiten an der zweiten Ausgabe der Deutschen Normungsroadmap Künstliche Intelligenz gestartet. In einem breiten Beteiligungsprozess und unter Mitwirkung von mehr als 570 Fachleuten aus Wirtschaft, Wissenschaft, öffentlicher Hand und Zivilgesellsch...

Patient care after kidney transplantation requires integration of complex information to make informed decisions on risk constellations. Many machine learning models have been developed for detecting patient outcomes in the past years. However, performance metrics alone do not determine practical utility. We present a newly developed clinical decis...

Clinical text processing has gained more and more attention in recent years. The access to sensitive patient data, on the other hand, is still a big challenge, as text cannot be shared without legal hurdles and without removing personal information. There are many techniques to modify or remove patient related information, each with different stren...

In recent years, machine learning for clinical decision support has gained more and more attention. In order to introduce such applications into clinical practice, a good performance might be essential, however, the aspect of trust should not be underestimated. For the treating physician using such a system and being (legally) responsible for the d...

This paper presents a fine-grained test suite for the language pair German-English. The test suite is based on a number of linguistically motivated categories and phenomena and the semi-automatic evaluation is carried out with regular expressions. We describe the creation and implementation of the test suite in detail, providing a full list of all...

Patient care after kidney transplantation requires integration of complex information to make informed decisions on risk constellations. Many machine learning models have been developed for detecting patient outcomes in the past years. However, performance metrics alone do not determine practical utility. Often, the actual performance of medical pr...

Scientific publications about machine learning in healthcare are often about implementing novel methods and boosting the performance - at least from a computer science perspective. However, beyond such often short-lived improvements, much more needs to be taken into consideration if we want to arrive at a sustainable progress in healthcare. What do...

Given the rich body of technical developments and a relatively long history of industrial use of Machine Translation (MT), it is astonishing of how little interest the topic of MT quality has received so far. In this paper, we present three ways of performing MT quality evaluation from our own research: (1) using TQ-AutoTest, a framework for semi-a...

Der zunehmende Einsatz von sogenannter „Künstlicher Intelligenz“ (KI) verspricht viele Verbesserungen, beispielsweise durch Bilderkennung in der Medizindiagnose. Er birgt aber auch das Risiko, dass Menschen durch irrtümliche Vorhersagen von KI-Systemen zu Schaden kommen können. In solchen Fällen wird es immer schwieriger zu bestimmen, wer die Veran...

Rund ein Jahr haben DIN und DKE in einem gemeinsamen Projekt mit dem Bun-desministerium für Wirtschaft und Energie und zusammen mit ca. 300 Fachleuten aus Wirtschaft, Wissenschaft, öffentlicher Hand und Zivilgesellschaft an der Nor-mungsroadmap Künstliche Intelligenz gearbeitet. Eine hochrangige Steuerungs-gruppe unter dem Vorsitz von Prof. Wolfgan...

UNSTRUCTURED The Covid-19 pandemic has put new demands on the medical systems worldwide. The pressure of taking far-reaching decisions within multiply limited resources under the constraint that personal contact must be minimized has evoked the question if technical support in the form of Artificial Intelligence (AI) could help leverage these chall...

This paper describes a test suite submission providing detailed statistics of linguistic performance for the state-of-the-art German-English systems of the Fifth Conference of Machine Translation (WMT20). The analysis covers 107 phenomena organized in 14 categories based on about 5,500 test items, including a manual annotation effort of 45 person h...

This paper describes a test suite submission providing detailed statistics of linguistic performance for the state-of-the-art German-English systems of the Fifth Conference of Machine Translation (WMT20). The analysis covers 107 phenomena organized in 14 categories based on about 5,500 test items, including a manual annotation effort of 45 person h...

We present an analysis of 16 state-of-the-art MT systems on German-English based on a linguistically-motivated test suite. The test suite has been devised manually by a team of language professionals in order to cover a broad variety of linguistic phenomena that MT often fails to translate properly. It contains 5,000 test sentences covering 106 lin...

We present an analysis of 16 state-of-the-art MT systems on German-English based on a linguistically-motivated test suite. The test suite has been devised manually by a team of language professionals in order to cover a broad variety of linguistic phenomena that MT often fails to translate properly. It contains 5,000 test sentences covering 106 lin...

Um MS-Erkrankten und ihren Angehörigen die Möglichkeit zu bieten, sich miteinander zu vernetzen, und das in einem geschutzten Rahmen, hat die DMSG 2015 die Initiative zur Entwicklung einer verbandsinternen Online-Plattform ergriffen und vorangetrieben. Hier haben Mitglieder die Moglichkeit, sich überregional kennenzulernen und zwar unter Berücksich...

The exponential growth of Audiovisual Translation (AVT) in the last three decades has consolidated its place as an area of study within Translation Studies (TS). However, AVT is still a young domain currently exploring a number of different lines of inquiry without a specific methodological and theoretical framework. This volume discusses the advan...

We present an analysis of 16 state-of-the-art MT systems on German-English based on a linguistically-motivated test suite. The test suite has been devised manually by a team of language professionals in order to cover a broad variety of linguistic phenomena that MT often fails to translate properly. It contains 5,000 test sentences covering 106 lin...

Several works show that predicate-argument structure is a level of analysis relevant for addressing Natural Language Processing problems, such as Textual Entailment (another study on Textual Entailment can be found in this volume). Although large resources like FrameNet are available (see also the chapter on FrameNet in this volume), attempts to in...

In this paper, we report an analysis of the strengths and weaknesses of several Machine Translation (MT) engines implementing the three most widely used paradigms. The analysis is based on a manually built test suite that comprises a large range of linguistic phenomena. Two main observations are on the one hand the striking improvement of an commer...

In this article we present a novel linguistically driven evaluation method and apply it to the main approaches of Machine Translation (Rule-based, Phrase-based, Neural) to gain insights into their strengths and weaknesses in much more detail than provided by current evaluation schemes. Translating between two languages requires substantial modellin...

This report on the web services and datasets for applications provides an update of the contents/plans described in D3.4 and D3.1 and of the ongoing report (D3.13 v0.91) submitted earlier. The first part of this document (Section 2) will be concerned with web services. The MT Pilots developed in the QTLeap project are documented in Deliverables D2....

The volume of Audiovisual Translation (AVT) is increasing to meet the rising demand for data that needs to be accessible around the world. Machine Translation (MT) is one of the most innovative technologies to be deployed in the field of translation, but it is still too early to predict how it can support the creativity and productivity of professi...

This work addresses the need to aid Machine Translation (MT) development cycles with a complete workflow of MT evaluation methods. Our aim is to assess, compare and improve MT system variants. We hereby report on novel tools and practices that support various measures, developed in order to support a principled and informed approach of MT developme...

The tool described in this article has been designed to help MT developers by implementing a web-based graphical user interface that allows to systematically compare and evaluate various MT engines/experiments using comparative analysis via automatic measures and statistics. The evaluation panel provides graphs, tests for statistical significance a...

DFKI participated in the shared translation task of WMT 2015 with the German-English language pair in each translation direction. The submissions were generated using an experimental hybrid system based on three systems: a statistical Moses system, a commercial rule-based system, and a serial coupling of the two where the output of the rule-based s...

The idea to improve MT quality by using deep linguistic and knowledge-driven information has frequently been expressed. If the goal is to use deep information for building an MT system, there are two extreme options: (1) to start from a purely knowledge-driven approach (RBMT) and try to arrive at the same recall found in current SMT systems; (2) to...

This paper demonstrates the possibility to make an existing automatic error classi-fier for machine translations independent from the requirement of lemmatisation. This makes it usable also for smaller and under-resourced languages and in situations where there is no lemmatiser at hand. It is shown that cutting all words into the first four letters...

In recent years translation quality evaluation has emerged as a major, and at times contentious, topic. Despite a focus on systematizing the evaluation of translation quality, the industry landscape is still highly fragmented, in part because different kinds of translation projects require very different evaluations of quality. In addition, human a...

Significant breakthroughs in machine translation (MT) only seem possible if human translators are taken into the loop. While automatic evaluation and scoring mechanisms such as BLEU have enabled the fast development of systems, it is not clear how systems can meet real-world (quality) requirements in industrial translation scenarios today. The tara...

In order to support the integration of the MT Pilots generated in WP2 in the real usage scenario (WP3), they will be made available as web services. The web services will provide the essential translation functionalities while encapsulating the details of the whole development process, except for the necessary parameters required by the application...

Human translators are the key to evaluating machine translation (MT) quality and also to addressing the so far unanswered question when and how to use MT in professional translation workflows. This paper describes the corpus developed as a result of a detailed large scale human evaluation consisting of three tightly connected tasks: ranking, error...

Despite the growing interest in and use of machine translation post-edited outputs, there is little research work exploring different types of post-editing operations, i.e. types of translation errors corrected by post-editing. This work investigates five types of post-edit operations and their relation with cognitive post-editing effort (quality l...

This work presents the new flexible Multidimensional Quality Metrics (MQM) framework and uses it to analyze the performance of state-of-the-art machine translation systems, focusing on "nearly acceptable" translated sentences. A selection of WMT news data and "customer" data provided by language service providers (LSPs) in four language pairs was a...

Machine translation (MT) is – not only historically – a prime application of language technology. After years of seeming stagnation, the price pressure on language service providers (LSPs) and the increased translation need have led to new momentum for the inclusion of MT in industrial translation workflows. On the research side, this trend is back...

Post-editing is an increasingly common form of human-machine cooperation for translation. One possible support for the post-editing task is offering several machine outputs to a human translator from which then can choose the most suitable one. This paper investigates the selection process for such method to get a better insight into it so that it...

Human translators are the key to evaluating machine translation (MT) quality and also to addressing the so far unanswered question when and how to use MT in professional translation workflows. Usually, human judgments come in the form of ranking outputs of different translation systems and recently, post-edits of MT output have come into focus. Thi...

This white paper is part of a series that promotes knowledge about language technology and its potential. It addresses educators, journalists, politicians, language communities and others. The availability and use of language technology in Europe varies between languages. Consequently, the actions that are required to further support research and d...

A breakthrough in Machine Translation is only possible if human translators are taken into the loop. While mechanisms for automatic evaluation and scoring such as BLEU have enabled fast development of systems, these systems have to be used in practice to get feedback for improvement and fine-tuning. However, it is not clear if and how systems can m...

Current metrics for evaluating machine translation quality have the huge drawback that they require human-quality reference translations. We propose a truly automatic evaluation metric based on ibm1 lexicon probabilities which does not need any reference translations. Several variants of ibm1 scores are systematically explored in order to find the...

We present a pilot study on an evaluation method which is able to rank translation outputs with no reference translation, given only their source sentence. The system employs a statistical classifier trained upon existing human rankings, using several features derived from analysis of both the source and the target sentences. Development experiment...

In der europäischen Informationsgesellschaft des 21. Jahrhunderts sollte niemand einen sozialen oder wirtschaftlichen Nachteil dadurch erleiden, lediglich seine Muttersprache zu sprechen – sei es Lettisch, Ungarisch oder Portugiesisch. Sprachtechnologie hat das Potential dieser Herausforderung entgegen zu treten, sofern sie robust und kosteneffekti...

In der europäischen Informationsgesellschaft des 21. Jahrhunderts sollte niemand einen sozialen oder wirtschaftlichen Nachteil dadurch erleiden, lediglich seine Muttersprache zu sprechen – sei es Lettisch, Ungarisch oder Portugiesisch. Sprachtechnologie hat das Potential dieser Herausforderung entgegen zu treten, sofern sie robust und kosteneffekti...

Future improvement of machine translation systems requires reliable automatic evaluation and error classification measures to avoid time and money consuming human classification. In this article, we propose a new method for automatic error classification and systematically compare its results to those obtained by humans. We show that the proposed a...

The increasing use of social software in combination with the rapid expansion of ubiquitous, networked computing technology has already started to change the way people interact, socialize, and live. This chapter focuses on changes in the way people acquire, update, and propagate knowledge. It describes how (institutional) learning and (informal) k...

In this article, we underpin the intuition that frame semantic information is a useful resource for modelling textual entailment. To this end, we provide a manual frame semantic annotation for the test set used in the second recognizing textual entailment (RTE) challenge – the FrameNet-annotated textual entailment (FATE) corpus – and discuss experi...

We present a general approach to formally modelling corpora with multi-layered annotation in a typed logical representation language, OWL DL. By defining abstractions over the corpus data, we can generalise from a large set of individual corpus annotations, thereby inducing a lexicon model. The resulting combined corpus and lexicon model can be int...

This paper discusses our contribution to the third RTE Challenge -- the SALSA RTE system. It builds on an earlier system based on a relatively deep linguistic analysis, which we complement with a shallow component based on word overlap. We evaluate their (combined) performance on various data sets. However, earlier observations that the combination...

This paper discusses our contribution to the third RTE Challenge -- the SALSA RTE system. It builds on an earlier system based on a relatively deep linguistic analysis, which we complement with a shallow component based on word overlap. We evaluate their (combined) performance on various data sets. However, earlier observations that the combination...

In this paper, we describe the SALTO tool. It was originally developed for the annotation of semantic roles in the frame semantics paradigm, but can be used for graphical annotation of treebanks with general relational information in a simple drag-and-drop fashion. The tool additionally supports corpus management and quality control.

This paper describes the SALSA corpus, a large German corpus manually annotated with role-semantic information, based on the syntactically annotated TIGER newspaper corpus (Brants et al., 2002). The first release, comprising about 20,000 annotated predicate instances (about half the TIGER corpus), is scheduled for mid-2006. In this paper we discuss...

We present a baseline system for model- ing textual entailment that combines deep syntactic analysis with structured lexi- cal meaning descriptions in the FrameNet paradigm. Textual entailment is approx- imated by degrees of structural and se- mantic overlap of text and hypothesis, which we measure in a match graph. The encoded measures of similari...

In this paper, we present a rule-based system for the assignment of FrameNet frames by way of a "detour via WordNet". The system can be used to overcome sparse-data problems of statistical systems trained on current FrameNet data. We devise a weighting scheme to select the best frame(s) out of a set of candidate frames, and present first figures of...

The growing size of electronically available text corpora like companies’ intranets or the WWW has made information access a hot topic within Computational Linguistics. Despite the success of statistical or keyword based methods, deeper Knowledge Representation (KR) techniques along with “inference” are often mentioned as mandatory, e.g. within the...

The most central fact about natural language is that it has meaning. Semantics is the study of meaning. In formal semantics, we conduct this study in a formal manner. In computational semantics, we’re additionally interested in using the results of our study when we implement programs that process natural language. In this course, we show how forma...

Introduction Basic hybrid logic is an extension of modal logic in which it is possible to name states and to assert that a formula is true at a named state. It does this with the help of nominals and satisfaction operators. Nominals are special propositional variables (typically written i, j, k, and l), that are true at exactly one point in any mod...

We present a general approach to formally modelling corpora with multi-layered anno-tation, thereby inducing a lexicon model in a typed logical representation language, OWL DL. This model can be interpreted as a graph structure that offers flexible querying func-tionality beyond current XML-based query languages and powerful methods for consis-tenc...

This user study reports on an ongoing pilot that aims at using machine translation on a large scale, for the translation of technical documentation for a globally acting auto-motive supplier. The pilot is conducted by a language service provider and a research institution. First results go beyond expec-tations.