Ivana Kruijff-Korbayova

• Senior Researcher at Deutsches Rettungsrobotik-Zentrum

Earthquakes, fire, and floods often cause structural collapses of buildings. However, the inspection of such damaged buildings poses a high risk for emergency forces or is even impossible. We present three recently selected missions of the Robotics Task Force of the German Rescue Robotics Center (DRZ), where both ground and aerial robots were used...

Despite recent advancements in speech recognition, there are still difficulties in accurately transcribing conversational and emotional speech in noisy and reverberant acoustic environments. This poses a particular challenge in the search and rescue (SAR) domain, where transcribing conversations among rescue team members is crucial to support real-...

Earthquakes, fire, and floods often cause structural collapses of buildings. The inspection of damaged buildings poses a high risk for emergency forces or is even impossible, though. We present three recent selected missions of the Robotics Task Force of the German Rescue Robotics Center, where both ground and aerial robots were used to explore des...

Artificial Intelligence (AI) systems are increasingly pervasive: Internet of Things, in-car intelligent devices, robots, and virtual assistants, and their large-scale adoption makes it necessary to explain their behaviour, for example to their users who are impacted by their decisions, or to their developers who need to ensure their functionality....

To meet the challenges involved in providing adequate robotic support to first responders, a holistic approach is needed. This requires close cooperation of first responders, researchers and companies for scenario-based needs analysis, iterative development of the corresponding system functionality and integrated robotic systems as well as human-ro...

With robotics rapidly advancing, more effective human–robot interaction is increasingly needed to realize the full potential of robots for society. While spoken language must be part of the solution, our ability to provide spoken language interaction capabilities is still very limited. In this article, based on the report of an interdisciplinary wo...

Lead partner: FDDO Revision: final Dissemination level: PU This document describes the work done in Year 3 in WP3: the development of persistent models for distributed joint situation awareness. This includes the ongoing TDS development itself from technical point of view as well as in conjunction with the ontology-based approach for an improved SA...

We report on a field exercise in which a team of human fire-fighters used robots to enact a realistic disaster response mission in an industrial environment. In this exercise we evaluated the technical working of an integrated robotic system and gained insights concerning the manner in which robots and information streams can be utilized effectivel...

As robots that share working and living environments with humans proliferate, human-robot teamwork (HRT) is becoming more relevant every day. By necessity, these HRT dynamics develop over time, as HRT can hardly happen only in the moment. What theories, algorithms, tools, computational models and design methodologies enable effective and safe longi...

We report on the latest large-scale disaster-response exercise conducted by our project, which involves a robotic system with both ground robots (UGVs) and aerial robots (UAVs). In particularly, we focus on aspects related to Human-Robot teaming, and the uptake of new technology by end-users.

As robots are increasingly used in Search and Rescue (SAR) missions, it becomes highly relevant to study how SAR robots can be developed and deployed in a responsible way. In contrast to some other robot application domains, e.g. military and healthcare, the ethics of robot-assisted SAR are relatively under examined. This paper aims to fill this ga...

We provide key facts about the TRADR project deployment of ground and aerial robots in Amatrice, Italy, after the major earthquake in August 2016. The robots were used to collect data for 3D textured models of the interior and exterior of two badly damaged churches of high national heritage value.

Establishing a positive relationship between a user and a system is considered important or even necessary in applications of social robots or other computational artifacts which require long-term engagement. We discuss several experiments investigating the effects of specific relational verbal behaviors within the broader context of developing a s...

Social robots have the potential to provide support in a number of practical domains, such as learning and behaviour change. This potential is particularly relevant for children, who have proven receptive to interactions with social robots. To reach learning and therapeutic goals, a number of issues need to be investigated, notably the design of an...

This paper describes the project TRADR: Long-Term Human-Robot Teaming for Robot Assisted Disaster Response. Experience shows that any incident serious enough to require robot involvement will most likely involve a sequence of sorties over several hours, days and even months. TRADR focuses on the challenges that thus arise for the persistence of env...

Developing collaborative robots that can productively and safely operate out of isolation in uninstrumented, human-populated environments is an important goal for the field of robotics. The development of such agents, those that handle the dynamics of human environments and the complexities of interpreting human interaction, is a strong focus withi...

The TRADR project aims at developing methods and models for human-robot teamwork, enabling robots to operate in search & rescue environments alongside humans as teammates, rather than as tools. Through a user-centered cognitive engineering method, human-robot teamwork is analyzed, modeled, implemented and evaluated in an iterative fashion. Importan...

This paper describes our experience in designing, developing and deploying systems for supporting human–robot teams during disaster response. It is based on R&D performed in the EU-funded project NIFTi. NIFTi aimed at building intelligent, collaborative robots that could work together with humans in exploring a disaster site, to make a situational...

Conversational systems and robots that use reinforcement learning for policy optimization in large domains often face the problem of limited scalability. This problem has been addressed either by using function approximation techniques that estimate the approximate true value function of a policy or by using a hierarchical decomposition of a learni...

This paper presents the results from an experiment with a conversational human-robot interaction system aimed at long-term support for diabetic children. The system offers a set of activities aimed to help a child to improve its capability to manage diabetes. There is a large body of literature on the techniques that artificial agents can use to es...

We present results concerning the timing of children’s verbal turn-taking behavior in quiz-game interactions with a humanoid robot, spread over three sessions on different days, in one of two conditions: the robot either gave explicit signals of being familiar with the user from previous interactions, or it did not. We found that communication prob...

For robots to interact effectively with human users they must be capable of coordinated, timely behavior in response to social context. The Adaptive Strategies for Sustainable Long-Term Social Interaction (ALIZ-E) project focuses on the design of long-term, adaptive social interaction between robots and child users in real-world settings. In this p...

A synthetic agent requires the coordinated use of multiple sensory and effector modalities in order to achieve a social human-robot interaction (HRI). While systems in which such a concatenation of multi-ple modalities exist, the issue of information coordination across modal-ities to identify relevant context information remains problematic. A sys...

This work presents preliminary observations from a study of children (N=19, age 5-12) interacting in multiple sessions with a humanoid robot in a scenario involving game activities. The main purpose of the study was to see how their perception of the robot, their engagement, and their enjoyment of the robot as a companion evolve across multiple int...

We demonstrate a conversational humanoid robot that allows users to follow their own dialogue structures. Our system uses a hierarchy of reinforcement learning dialogue agents, which support transitions across sub-dialogues in order to relax the strictness of hierarchical control and therefore support flexible interactions. We demonstrate our syste...

Conversational systems play an important role in scenarios without a keyboard, e.g., talking to a robot. Communication in human-robot interaction (HRI) ultimately involves a combination of verbal and non-verbal inputs and outputs. HRI systems must process verbal and non-verbal observations and execute verbal and non-verbal actions in parallel, to i...

We describe an approach for multi-modal dialogue strategy learning combining two sources of uncertainty: speech and gestures. Our approach represents the state-action space of a reinforcement learning dialogue agent with relational representations for fast learning, and extends it with belief state variables for dialogue control under uncertainty....

Assigning intonation to dialogue system output in a way that reflects relationships between entities in the discourse context can enhance the acceptability of system utterances. Previous research concentrated on the role of linguistic context; dialogue situatedness and the role of visual context in determining accent placement has not been studied....

Dora the Explorer is a mobile robot with a sense of curiosity and a drive to explore its world. Given an incomplete tour of an indoor environment, Dora is driven by internal motivations to probe the gaps in her spatial knowledge. She actively explores regions of space which she hasn't previously visited but which she expects will lead her to furthe...

Formal domains, such as mathematics, require exact language to communicate the intended content. Special symbolic notations are used to express the semantics precisely, compactly, and unambiguously. Mathematical textbooks offer plenty of examples of concise, accurate presentations. This effective communication is enabled by interleaved use of formu...

In CoSy, our robots were to be able to interact with human. These interactions served to help the robot learn more about its environment, or to plan and carry out actions. For a robot to make sense of such dialogues, it needs to understand how a dialogue can relate to, and refer to, “the world” – local visuo-spatial scenes, as in the Playmate scena...

Situated dialogue is usually tightly integrated with be-havior planning, physical action and perception. This paper presents an algorithmic framework, Continual Collaborative Planning (CCP), for modeling this kind of integrated behavior and shows how CCP agents nat-urally blend physical and communicative actions. For experiments with conversational...

The background for this paper is the aim to build robotic assistants that can "natu-rally" interact with humans. One prereq-uisite for this is that the robot can cor-rectly identify objects or places a user refers to, and produce comprehensible ref-erences itself. As robots typically act in environments that are larger than what is immediately perc...

In this paper we present an approach to the task of generating and resolving referring expressions (REs) for conversational mobile robots. It is based on a spatial knowledge base encompassing both robot- and human-centric representations. Existing algorithms for the generation of referring expres- sions (GRE) try to find a description that uniquely...

The use of the treebank as a resource for linguistic research has led us to look for an annotation scheme representing not only surface syntactic information (in ‘analytic trees’, ATS) but also the underlying syntactic structure of sentences and at least some aspects of intersentential links (in ‘tectogrammatical tree structures’, TGTS). We focus i...

A dialogue system can present itself and/or address the user as an active agent by means of linguistic constructions in personal style, or suppress agentivity by using impersonal style. We describe how we generate and control per-sonal and impersonal style variation in the out-put of SAMMIE, a multimodal in-car dialogue system for an MP3 player. We...

A dialogue system can present itself and/or address the user as an active agent by means of linguistic constructions in personal style, or suppress agentivity by using impersonal style. We compare system evaluation judgments and input style alignment of users interacting with an in-car dialogue system generating output in personal vs. impersonal st...

We present DiaWOz-II, a configurable software environment for Wizard-of-Oz studies in mathematics and engineering. Its interface is based on a structuralwysiwyg editor which allows the input of com- plex mathematical formulae. This allows the collection of dialog corpora consisting of natural language interleaved with non-trivial mathemati- cal exp...

We present SAMMIE, a laboratory demonstrator of an in-car show- case of a multimodal dialogue system developed in the TALK project5 in cooperation between DFKI/USAAR/BOSCH/BMW, to show nat- ural, intuitive mixed-initiative interaction, with parti cular emphasis on multimodal turn-planning and natural language generation. SAM- MIE currently supports...

The SAMMIE1 system is an in-car multi- modal dialogue system for an MP3 ap- plication. It is used as a testing environ- ment for our research in natural, intuitive mixed-initiative interaction, with particu- lar emphasis on multimodal output plan- ning and realization aimed to produce out- put adapted to the context, including the driver's attentio...

We demonstrate work in progress1 us- ing the Nite XML Toolkit on a cor- pus of multimodal dialogues with an MP3 player collected in a Wizard-of-Oz (WOZ) experiments and annotated with a rich feature set at several layers. We designed an NXT data model, converted experiment log file data and manual tran- scriptions into NXT, and are building an- not...

This article reports on our experience with developing multilingual grammar resources for natural language generation (NLG). We employ a strong notion of multilinguality: (i) Grammars for different languages share their overall organization, as well as those descriptions that reflect similarities between languages and (ii) a single realization engi...

Natural language interaction between a student and an assis-tance system for mathematics is a new multi-disciplinary challenge that requires the interaction of (i) advanced natural language processing, (ii) flexible tutorial dialog strategies including hints, and (iii) mathematical domain reasoning. Although each of these fields o ers substantial s...

This paper investigates automatic identi- fication of Information Structure (IS) in texts. The experiments use the Prague Dependency Treebank which is annotated with IS following the Praguian approach of Topic Focus Articulation. We auto- matically detect t(opic) and f(ocus), us- ing node attributes from the treebank as basic features and derived f...

Discourse in formal domains, such as mathe- matics, is characterized by a mixture of tele- graphic natural language and embedded formal expressions. Little is known about the suitabil- ity of input analysis methods for mathematical discourse in a dialog setting, due to the lack of empirical data. In this paper, we report on the development of a dep...

Discourse in formal domains, such as mathematics, is characterized by a mixture of telegraphic natu-ral language and embedded (semi-)formal symbolic mathematical expressions. Due to the lack of em-pirical data, little is known about the suitability of input analysis methods for mathematical discourse in a dialog setting. We present an input underst...

Discourse in formal domains, such as mathematics, is characterized by a mixture of telegraphic natural language and embedded (semi-)formal symbolic mathematical expressions. We present language phenomena observed in a corpus of dialogs with a simulated tutorial system for proving theorems as evidence for the need for deep syntactic and semantic ana...

The goal of the MULI (MUltiLingual Information structure) project is to empirically analyse information structure in German and English newspaper texts. In contrast to other projects in which information structure is annotated and investigated (e.g. in the Prague Dependency Treebank, which mirrors the basic information about the topic-focus articul...

We present discourse-level annotation of newspa- per texts in German and English, as part of an ongoing project aimed at investigating information structure from a cross-linguistic perspective. Rather than annotating some specific notion of information structure, we propose a theory-neutral annotation of basic features at the levels of syntax, pros...

We propose a multilingual approach to characterizing word order at the clause level as a means to realize information structure. We illustrate the problem with three languages which differ in the degree of word order freedom they exhibit: Czech, a free word order language in which word order variation is pragmatically determined; English, a fixed w...

This issue presents a collection of papers that address the interactions between discourse structure and information structure from a variety of perspectives. Their common goal is to improve our understanding of how utterance-internal semantic devices that make up information structure, and utterance-external semantic devices that make up discourse...

Our goal is to improve the contextual appropriateness of spoken output in a dialogue system. We explore the use of the information state to determine the information structure of system utterances.

We demonstrate the production of spoken output with contextually appropriate intonation in the information-state based dialogue system GoDiS. We exploit the context representation in the information state to determine the information structure of system utterances, which we use to control the intonation of synthesized spoken output. 1

Our goal is to improve the contextual appropriateness of spoken output in a dialogue system. We explore the use of the information state to determine the information structure of system utter-ances. We concentrate on the realiza-tion of information structure by into-nation. We present the results of eval-uating the contextual appropriateness of var...

We demonstrate the production of spo- ken output with contextually appropri- ate intonation in the information-state based dialogue system GoDiS. We ex- ploit the context representation in the in- formation state to determine the infor- mation structure of system utterances, which we use to control the intonation of synthesized spoken output.

Dialogs in formal domains, such as mathematics, are characterized by a mixture of tele- graphic natural language text and embedded formal expressions. Due to the lack of empirical data for such environments, we have collected a corpus of dialogs with a simulated tutoring system for teaching proofs in naive set theory. The analysis of this corpus en...

We aim at improving the collaborative character of the responses in the GoDiS dialogue system participating in information-seeking dialogues. In this paper we describe how the system currently deals with successful and unsuccessful database search and show the need for generating more collaborative responses. We consider conditional yes/no response...

We analyze naturally occurring collaborative responses of the form "Not (if) /Yes if ". We distinguish two cases: when is established in the con- text, the conditional response indicates a possible need to revise , and thus opens negotiation; otherwise, the conditional response raises the question whether . We discuss the contexts where such respon...

The paper deals with conditional responses of the form "Not if c/Yes if c" in reply to a question "?q" in the context of information-seeking dialogues. A conditional response is triggered if the obtainability of q depends on whether c holds: The response indicates a possible need to find alternative solutions, opening a negotiation in the dialogue....

We present an implementation of a discourse parsing system for a lexicalized Tree-Ajoining Grammar for discourse, specifying the integration of sentence and discourse level processing.

It is commonly agreed nowadays that information structure is a crucial aspect of the linguistic meaning of a sentence and its interpretation in discourse context. Here, we present an information-structure sensitive account of discourse interpretation, formalized as a hybrid modal logic, which can be smoothly related to a dependency grammar. This fo...

Without Abstract

The paper discusses the methods followed to re-use a large-scale, broad-coverage English grammar for constructing similar scale grammars for Bulgarian, Czech and Russian for the fast prototyping of a multilingual generation system. We present (1) the theoretical and methodological basis for resource sharing across languages, (2) the use of a corpus...

this paper, we only consider the text type of full procedural instructions,on which we illustrate the TSM's capabilities in planning dierent typesof aggregation and contextual reference to enhance coherence and cohesion

This paper describes a multilingual text generation system in the domain of CAD#CAM software instructions for Bulgarian, Czech and Russian. Starting from a language-independent semantic representation, the system drafts natural, continuous text as typically found in software manuals. The core modules for strategic and tactical generation are implem...

. The use of the treebank as a resource for linguistic research has led us to look for an annotation scheme representing not only surface syntactic information (in `analytic trees', ATS) but also the underlying syntactic structure of sentences and at least some aspects of intersentential links (in `tectogrammatical tree structures', TGTS). We focus...

The paper describes a multilingual textgeneration system for Bulgarian, Czech and Russian in the domain of CAD/CAM software instructions. Starting from a language-independent semantic representation, the system drafs continuous text as it is typically found in instructional texts in the three languages. The core modules for strategic and tactical g...

We describe an implemented approach to text and sentence planning in a system for automatic generation of instruction texts. The approach aims at planning good quality texts through maximising coherence and cohesion.

The approach presented in this paper has been developed in the context of the international project called AGILE (Automatic Generation of Instructions in Languages of Eastern Europe)1. The overall aim of the project is to develop a multilingual system for generating continuous instructional texts in Bulgarian, Czech and Russian [2]. The project is...

algorithm for word order Given: a list G of ordering constraints imposed by the grammar, a list L1 of constituents that need to be ordered, a list Delta giving ordering of CB constituents, create empty lists LC and LN % LC for CB items, LN for NB items repeat for each element E in L1 if E is CB, then add E into LC, else add E into LN.

Slavic languages are characteristic by their relatively high degree of word order freedom. In the process of automatic generation from an underlying representation of the content, we have to ensure that a semantically and contextually appropriate word order is chosen. In this paper, we elucidate information structure as the main factor determining...