Article

An empirical study of the use of speech and gestures in human-computer communication

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Multimodal human-computer interaction providing users with flexible «natural» means of expression, such as speech and gestures, has until now motivated few ergonomic studies. The need for reliable empirical data on users' preferences, behaviours and strategies (with respect to this new form of multimodality) is crucial for the definition of adequate user models that could significantly contribute to the efficiency and usability of future multimodal user interfaces. We describe a prospective ergonomic study which aims to provide empirical data on how users will use unconstrained speech and gestures for interacting with standard computer applications. Despite recent scientific advances in natural language interpretation, continuous speech recognition and gesture analysis, operational interaction systems capable of interpreting complex multimodal utterances accurately are not yet available. Therefore, in order to study the forms of multimodal expression that users will develop spontaneously while interacting with software systems capable of «understanding» speech and 2-dimensional gestures (on the screen), we have designed a realistic simulation of such an interaction system, using the Wizard of Oz experimental paradigm. Eight subjects performed various design tasks, using this simulated user interface during three weekly sessions. Analyses of the audio-visual protocols suggest great inter-individual differences, as regards subjects' styles of expression and their strategies for combining speech with gestures. Styles of expression can be divided into two broad classes according to the nature of the relationships between the subjects and the system: communication with a machine interlocutor or manipulation of a graphical representation of the application. After a short initial adaptation phase, which is confined to the first scenario or to the first session, the styles of most subjects do not evolve. We observed a few instances of later evolution which point to inter-individual differences relating to the capability of transferring competences acquired in the context of human communication to a human-computer interaction environment.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Et si le langage verbal n'est jamais désincarné (Bouvet, 2001), le geste peut être désolidarisé du discours, tout en faisant preuve d'une grande efficacité. Si le verbe sert à signaler les ajustements progressifs et précis nécessaires à localiser une cible (Carbonell et al., 1997), le geste est meilleur pour les explications et les indications relatives à des actions exécutables dans l'espace (Rapp et al., 2006). Après avoir appris comment assembler les pièces d'un objet, des sujets transmettent leurs nouveaux savoirs à des condisciples de façon plus efficace lorsqu'ils n'utilisent que des gestes pour le faire (Lozano et Tversky, 2006). ...
Article
The present paper reports the results of an experimentation that explores the cognitive dimension of gesture involved in many situations of social communication. The method is based on a protocol structured in four experimental modalities, with video sequences. The results from 82 subjects suggest that the role of this component of nonverbal behavior exceeds the apparent transmission of information. It clarifies the verbal speech of the locutor, and also allows to share his own representations, and to direct those of the interlocutor. This impact is all the more effective as the visual-spatial aptitudes of the later are high.
Article
Behavior is a commonly used concept in science, even if its definition is not clearly established. Its importance has changed during the history of psychology. It was both the central concept of a dominant paradigm until the emergence of the cognitivist concepts. Presently, it is used in studies on emotion and cognition, from perception to action without being univocally specified. For psychology and other specialities dealing with human behaviour, it becomes necessary to get on agreement on the signification of behavior. After a retrospective, including the confrontation with the cognitivist paradigms, it is proposed to analyse the behavior within the paradigm of complexity. This approach is now applied to such domains of psychology as perception, social organization, personality, development, and would also potentially bring relevant developments on the behavior concept.
Article
The paper reports two experiments which investigated the accuracy of verbal communication between people. The results indicate that communication is most accurate if the sender correctly anticipates the context in which the receiver attempts to understand the communication. If the sender expects the receiver will have more or less information than is actually available, communication suffers. Further, different sorts of information serve different functions. If the sender transmits general categorical information, that information is useful for the receiver in making a category selection, but is of no use in discriminating among categorically related alternatives. However, if the sender transmits information related to specific items, that information is useful for discriminating among alternatives, but less useful for categorical choices. The results are interpreted in terms of a contractual-contextual model of communication.
Article
Our research goals are to understand the nature of, construct and evaluate intelligent interfaces as knowledge-based systems. In this paper we demonstrate the need for help systems as an essential part of human-computer communication. Help strategies are based on a model of the task (to understand what the user is doing or which goals he/she ¹ wants to achieve) and a model of the user (to guarantee that these systems are non-intrusive and that they pay attention to the needs of individual users). We illustrate that passive and active help systems have to be constructed as knowledge-based systems. Two operational systems (PASSIVIST and ACTIVIST) are described to show the usefulness of this approach.
Article
This study presents an exploratory investigation into the perception of computer dialogue personality. Sixty-one subjects participated in a between subjects experimental design of computerized car ordering stimulus programs modeled to exhibit personality types: extravert-stable, extravert-unstable, introvert-stable, and introvert-unstable. Factor analysis of the perceived presence of twelve personality traits in the computer dialogue indicated subjects perceive computer dialogue personality in much the same manner as they characterize human personality. Perception of computer dialogue extraversion differences were more salient than neuroticism differences, but were not found to influence user satisfaction or performance. The study establishes an empirical groundwork for future development of computer dialogue personality methodologies.
Article
Three experiments are reported, dealing with a situation in which subjects carried on a dialogue, via a computer terminal, with what they believed to be either a computer system or another person. Analysis of the ensuing protocols concentrated on the use of anaphor and on lexical choice. Systematic stylistic variations result from placing subjects in a situation where they believe their interlocutor to be a computer system. Subjects in this condition sustain a dialogue with a restricted window of focussed content and, as a result, compared to person-to-person dialogues, utterances are short; lexical choice is kept to a negotiated minimum; and the use of pronominal anaphor is minimised, even when referring to the discourse topic. These dialogue characteristics persist over lengthy periods of interaction. Lack of evidence to support presuppositions concerning the capabilities of a defined computer source does not lead to a change in style. Similarly, attempts to manipulate features of computer output by producing more friendly surface forms did not influence subjects' behaviour. However, within the limits of the measures taken, subjects learned as much and performed as well, or better, in interactions defined as being with a computer.RésuméOn décrit trois expériences dans lesquelles des sujets poursuivent, par l'intermédiaire d'un terminal, un dialogue avec ce qu'ils croient être soit un ordinateur soit une autre personne. L'analyse des dialogues a porté de façon privilégiée sur deux facteurs: l'emploi de l'anaphore et le choix lexical. Il apparaît que, lorsque les sujets croient parler á un ordinateur, le style de leurs interventions varie systématiquement. Les sujets dans cette situation dialoguent avec une fenêtre restreinte de contenu focalisé, et donc leurs énoncés sont courts, par rapport à ce qui se passe dans le dialogue de personne à personne; le choix lexical est limité autant que possible; et l'on évite de recourir à l'anaphore pronominale, même lorsque c'est au thème du discours qu'il est fait référence. Ces caractéristiques perdurent lorsque l'interaction se prolonge. L'absence d'indices venant étayer les présuppositions initiales concernant les capacités de l'ordinateur n'entraîne pas de changement stylistique. De même, l'introduction dans les énoncés attribués à l'ordinateur de formes de surface plus amicales n'a pas eu d'influence sur le comportement des sujets. Toutefois, dans la limite de nos mesures, les sujets ont appris autant et la communication s'est déroulée aussi bien, voire mieux, lorsqu'ils interagissaient avec ce qu'ils croyaient être un ordinateur.