About
76
Publications
7,675
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
660
Citations
Introduction
Hannes Rieser currently works at the Faculty of Linguistics and Literary Studies, Bielefeld University. Hannes does research in Formal Semanics/Pragmatics, Concurrency, Dialogue Theory, and Gesture. Their current project is 'Process algebra accounts of linguistic processes in particular speech gesture coordination'.
Current institution
Publications
Publications (76)
This paper deals with three interrelated topics, linguistic anaphora, multi-modal anaphora and the top-down broadcasting of information using gestural post-holds in multimodal dialogue. Initially, a new solution for definite, pronominal and pro-adverbial anaphora is given based on the idea that an existentially quantified general term may output a...
We offer an account of multi-modal meaning coordination, taking speech-gesture meaning coordination as a prototypical case. We argue that temporal synchrony (plus prosody) does not determine how to coordinate speech meaning and gesture meaning. Challenging
cases are asynchrony and broadcasting cases, which are illustrated with empirical
data. We pr...
Humans communicate with different modalities. We offer an account of multi-modal meaning coordination, taking speech-gesture meaning coordination as a prototypical case. We argue that temporal synchrony (plus prosody) does not determine how to coordinate speech meaning and gesture meaning. Challenging cases are asynchrony and broadcasting cases, wh...
![Fig. 8 lf-tree [: red' ∧ airplane' [Adj: red'] [: [N: airplane']]]](profile/Hannes-Rieser/publication/334342719/figure/fig4/AS:937039419691010@1600419015372/lf-tree-red-airplane-Adj-red-N-airplane_Q320.jpg)
The paper is based on extensive corpus work dealing with the interaction of gesture and speech in natural route-description dialogues. The issue discussed is how non-regimented gesture and speech processes can be modelled in a formal system. The main argument is that this cannot be achieved in structural paradigms currently in use. The proposal is...
We deal with a yet untreated issue in debates about linguistic interaction, namely a particular multi-modal dimension of meaning-dependence. We argue that the shape interpretation of speech-accompanying iconic gestures is dependent on its co-occurrent speech. Since there is no prototypical solution for mod-eling such a dependence, we offer an appro...
The talk is based on extensive corpus work dealing with the interaction of gesture and speech in natural settings. The problem handled is how gesture and speech processes taken as semiotic systems can be modelled in a formal language. The main argument in the talk is that this cannot be achieved in paradigms currently under use such as DRT, SDRT or...
The implementation of speech-gesture interfaces
is one of the vital problems in
formal research on multi-modal discourse.
This paper provides empirical evidence
that, due to asynchronous occurrences of
gesture and speech, speech-gesture interfaces
cannot be expressed in purely
static structural terms resting on a speechgesture
map. As an alternativ...
Current semantic theory on indexical expressions claims that demonstratively used indexicals such as this lack a referent-determining meaning but instead rely on an accompanying demonstration act like a pointing gesture. While this view allows to set up a sound logic of demonstratives, the direct-referential role assigned to pointing gestures has n...
Our contribution situates Human-Robot Communication, especially the grounding of Natural Kind Terms, in the interface of Artificial Intelligence, Cognitive Psychology, Philosophy, Robotics and Semantics. We investigate whether a robot can be grounded in the sense favoured in Artificial Intelligence and Philosophy.
We thus extend the notion of groun...
Alignment in Communication is a novel direction in communication research, which focuses on interactive adaptation processes assumed to be more or less automatic in humans. It offers an alternative to established theories of human communication and also has important implications for human-machine interaction. A collection of articles by internatio...
When communicating in a face-to-face setting, interlocutors frequently produce multimodal meaning pack-ages consisting of speech and accompanying gestures. We present a systematically annotated speech-and-gesture cor-pus of directions that allows to study the interplay of both modalities empirically. After describing the primary and secondary data...
We investigate the question of how co-speech iconic gestures are used to convey visuo-spatial information in an interdisciplinary way, starting with a corpus-based empirical and theoretical perspective on how a typology of gesture form and a partial ontology of gesture meaning are related. Results provide the basis for a computational modeling appr...
Notwithstanding conclusive psychological and corpus evidence that at least some aspects of anaphoric and referential interpretation take place incrementally, and the existence of some computational models of incremental reference resolution, many aspects of the linguistics of incremental reference interpretation still have to be better understood....
A brief introduction to the topics discussed in the special issue, and to the individual papers.
Although not very well investigated, a crucial aspect of gesture use in dialogues is to regulate the organisation of the interaction. People use gestures decisively, for example to indicate that they want someone to take the turn, to 'brush away' what someone else said, or to acknowledge others' contributions. We present first insights from a corpu...
The paper is based on the Bielefeld Speech-And-Gesture-Alignment corpus (SAGA). From this corpus one video film is taken to
establish a typological grid for iconic and referring gesture types, i.e. a multiple inheritance hierarchy of types proceeding
from single gestural features like hand shape to sequences of entities filling up the whole gesture...
Collaborative completions are among the strongest evidence that dialogue requires coordination even at the sub-sentential level; the study of sentence completions may thus shed light on a number of central issues both at the `macro’ level of dialogue management and at the `micro’ level of the semantic interpretation of utterances. We propose a trea...
People communicate multimodally. Most prominently, they co-produce speech and gesture. How do they do that? Studying the interplay of both modalities has to be informed by empirically observed communication behavior. We present a corpus built of speech and gesture data gained in a controlled study. We describe 1) the setting underlying the data; 2)...
Gestures are usually looked at in isola- tion or from an intra-propositional per- spective essentially tied to one speaker. The Bielefeld multi-modal Speech-And- Gesture-Alignment (SAGA) corpus has many interactive gestures relevant for the structure of dialogue (Rieser 2008, 2009). To describe them, a dialogue theory is needed which can serve as a...
Empirical evidence from body measure- ments suggests that the referent of a demonstration is not directly specified, but obtained by applying a default infer- ence rule to the region specified by the pointing cone. Building on this evidence we propose a unified theory of anaphoric and demonstrative uses in which accessi- bility is obtained via reso...
Gestures are usually looked at in isolation or from an intra-propositional perspective essentially tied to one speaker. The Bielefeld multi-modal Speech-And-Gesture-Alignment (SAGA) corpus has many interactive gestures relevant for the structure of dialogue (Rieser 2008, 2009). To describe them, a dialogue theory is needed which can serve as a spee...
This paper deals mainly with iconic ges-ture in two-agent route description dialogue and focuses largely on the interface of word semantics and gesture. The modelling tools used come from formal semantics and pragmatics. The empirical background of the study is a partly annotated corpus of ca 5.000 gestures collected in the Biele-feld Speech-and-Ge...
We investigate task-oriented dialogue where utterances are produced by several agents. Starting out from an example where a directive is initiated by one agent and completed by the addressee, we explain under which conditions co-ordinations of this type can be successful. The explanatory devices developed are ‘action schema’ and ‘sufficiently infor...
This chapter presents a collaborative approach towards a detailed under- standing of the usage of pointing gestures accompanying referring expressions. This effort is undertaken in the context of human-machine interaction integrating empiri- cal studies, theory of grammar and logics, and simulation techniques. In particular, we take steps to classi...
We present a collaborative approach towards a detailed understanding of the usage of pointing gestures accompanying referring expressions. This effort is undertaken in the context of human-machine interaction integrating empirical studies, theory of grammar and logics, and simulation techniques. In particular, we attempt to measure the precision of...
Demonstratives, in particular gestures that "only" accompany speech, are not a big issue in cur-rent theories of grammar. If we deal with ges-tures, fixing their function is one big problem, the other one is how to integrate the representations originating from different channels and, ultimately, how to determine their composite meanings. The growi...
Classical SDRT (Asher and Lascarides, 2003) dis-cussed essential features of dialogue like adjacency pairs or corrections and up-dating. Recent work in SDRT (Asher, 2002, 2005) aims at the description of natural dialogue. We use this work to model situated communication, i.e. dialogue, in which sub-sentential utterances and gestures (pointing and g...
We describe an experiment to gather original data on geometrical aspects of pointing. In particular, we are focusing upon the concept of the pointing cone, a geometrical model of a pointing’s extension. In our setting we employed methodological and technical procedures of a new type to integrate data from annotations as well as from tracker recordi...
We present a statistical approach to as-sess relations that hold among speech and pointing gestures in and between turns in task-oriented dialogue. The units quantified over are the time-stamps of the XML-based annotation of the dig-ital video data. It was found that, on av-erage, gesture strokes do not exceed, but are freely distributed over the t...
Magritte reflektierte in den Jahren 1927 bis 1935 Unterschiede zwischen Depiktionen (Bildern), Wörtern und Gegenständen. In diesem Zusammenhang entstand sein wohl bekanntestes Bild „Dies ist keine Pfeife“, nachfolgend in einer Version von 1948 wiedergegeben.
The formal treatment of the semantics and pragmatics of dialogue became possible through a series of breakthroughs in foundational methodology. There is broad consensus on a couple of issues, like the fact that some variety of dynamic theory is necessary to capture certain characteristics of dialogue. Other matters still are disputed. This volume c...
Gegenstand dieses Beitrags sind Formen der nicht-wörtlichen Interpretation. Es geht dabei um Depiktionsmetonymien, also um Ausdrücke wie Flugzeug,Rumpf oder Fahrgestell, die verwandt werden, um bestimmte Spielzeug-Aggregate zu benennen. Dazu diskutieren wir zunächst die Depiktionsleistung derartiger Aggregate im Rahmen einer dazu entwickelten semio...
Koordination im Dialog bedeutet, daß auftretende Probleme unter wechselseitiger Kontrolle der Agenten nach festgelegten Verfahren gelöst werden. In aufgabenorientierten Dialogen ist der soziale Rahmen für Koordination festgelegt. Koordination wird erforderlich wegen des Informationsgefälles zwischen Instrukteur und Konstrukteur, des dominanten Dial...
In this talk we report about eye-movement research and the investigation of dialogue structure as it has been going on at several institutions at the University of Bielefeld (cf. the reports listed in the bibliography). Dialogues, one might argue, can be regarded as sequences of turns in which certain micro- and macro-structures can be distinguishe...
this paper was presented as a talk at the Twendial `98 and appeared in the proceedings as [11]. Mental States in Construction Dialogue 2 Contents
This paper is based on a talk I gave at the MunDial 97, University of Munich, Germany, March 10--12, 1997. Thanks go to Anton Benz, Gerhard J ager and other participants of the MunDial, who commented upon the things I said there. Most of the topics discussed in the paper are treated in a Bielefeld project on the syntax of spoken discourse (see [6]...
The paper reports on eye-tracking research as it has been going on in diierent institutions at Bielefeld University. Initially, a short introduction into eye-tracker technology is given and the two scenarios used in the investigation of agents' behaviour in task-oriented dialogue are presented: A 2D-setting working with one eye-tracker that records...
Magritte reflektierte in den Jahren 1927 bis 1935 Unterschiede zwischen Depiktionen (Bildern), Wörtern und Gegenständen. In diesem Zusammenhang entstand sein wohl bekanntestes Bild „Dies ist keine Pfeife“, nachfolgend in einer Version von 1948 wiedergegeben.
Introduction: Levels of Coordination Students of situated communication have observed for some time that agents rely on the coordination of verbal and non--verbal actions. Evidence supporting this observation has been gathered in a number of frameworks: D. Lewis showed in his "Convention" (1969, [13]) that coordination of agents has to go into the...
Den folgenden Ausführungen über Metonymie, Perspektive und Orientierung von Agenten liegt eine Studie zum Flugzeug-setting des SFB 360, “Situierte Künstliche Kommunikatoren” zugrunde. Das setting hat folgende Charakteristika: Ein Konstruktor baut nach Anweisungen eines Instruktors einen Spielzeug-Flieger der Marke “Baufix” aus Bauteilen (Lochleiste...
Wir gehen in dieser Arbeit von der Hypothese aus, daß die Augenbewegungsforschung eine Möglichkeit darstellt, Aufschlüsse über die Organisation kognitiver Strukturen und die Dynamik kognitiver Prozesse zu erhalten. Die Fruchtbarkeit dieser Hypothese zeigen wir anhand der Beschreibung einer Reihe von Experimenten und Studien, die in Projekten des SF...
Das Projekt “Theoretische Grundlagen und Simulation von Prozessen der Kohärenzkonstitution im gesprochenen Deutsch” hat sich schwerpunktmäßig mit dem Phänomen der “Reparatur” auseinandergesetzt. Reparaturen entstehen in der gesprochenen Sprache unter anderem dann, wenn einem Sprecher ein tatsächlicher oder ein vermeintlicher Fehler unterläuft, den...
This paper integrates several related lines of research in an implemented model. Its main aim is to show how principles of situation semantics concerning meanings, constraints and the preferred ontology can be represented and mapped onto expressions of natural language in a straightforward way. For assembling larger chunks of information a unificat...
In order to show the importance of J. Woods' The logic of fiction for theories concerned with the semantics of fictional discourse, first a brief survey of the topics discussed in this field is provided. These are in short: the existence of fictional objects, meaning and truth of fictional sentences, and entailments in fictional discourse. Then the...
0. The text grammars set up so far generalize the formal properties of specific types of sentence grammar.1 This procedure, widely accepted among text linguists, led us to the assumption that text grammars developed from a specific type of sentence grammar will have the formal and empirical properties of this model sentence grammar. This assumption...
If we consider how theoretical operations belonging to the methodological inventory of linguistics are carried out (i. e. the way linguistic theories are set up), three main criteria suggest themselves for classifying them: (1) Both, nature and type of the aims of the scientific knowledge applied which allow to specify the epistemological interests...
A new approach based on experiments aiming at the integration of content originating from pointing plus definite descriptions (objects called "CDs") in dialogue is presented. We develop it against the background of the early semiotic positions of Wittgenstein, Peirce, and Quine, the intentionalism of Kaplan, Neo-Peirce-Wittgenstein-Quine approaches...
The paper discusses how a gesture typology can be extracted form the Bielefeld Speech-And-Gesture-Alignment corpus (SAGA) making use of the anno-tated gesture morphology in the SAGA data. The SAGA corpus is briefly characterized. Using a portion of a MM dialogue, the interface between speech and gesture is shown focussing on the im-pact of gestures...
Metonymies are known as ways of conceptualizing entities and situations of various sorts. For example, users of PCs usually call the icon depicting a waste paper basket 'basket'. They thus use the name of the depicted object for referring to the depicting object. We use a model airplane ("Baufix" airplane) as the paradigm case for investigating the...
Questions
Question (1)
For example: taking the turn could be accompanied by eye-contact with other speaker, similarly for acknowledgements and in-turn repairs. Taking the turn could perhaps also be preceded by eye-contact: first focus, then speak. Eye-contact could also occur when the topic is changed, or with re-starts. I think, there is also "eye-communication" with the other communication participant, especially, if something "dramatic" is going to happen.
