Article

An innovative framework to support multimodal interaction with Smart Environments

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Interaction with future coming Smart Environments requires research on methods for the design of a new generation of human–environment interfaces. The paper outlines an original approach to the design of multimodal applications that, while valid for the integration on today’s devices, aims also to be sufficiently flexible so as to remain consistent in view of the transition to future Smart Environments, which will likely be structured in a more complex manner, requiring that interaction with services offered by the environment is made available through the integration of multimodal/unimodal interfaces provided through objects of everyday use. In line with the most recent research tendencies, the approach is centred not only on the user interface part of a system, but on the design of a comprehensive solution, including a dialogue model which is meant to provide a robust support layer on which multimodal interaction builds upon. Specific characteristics of the approach and of a sample application being developed to validate it are discussed in the paper, along with some implementation details.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... The enterprise outcome comprises three factors such as improvement of customer satisfaction, enhancement of organizational competition power, and advancement of organizational image [11]. In non-financial research, the performance of a firm was examined by efficiency, effectiveness, profitability, quality of service, client satisfaction, and productivity [6]- [9], [11], [12], [15]. This is their satisfaction level about their firm's outcome in terms of growth in sale, increase in profits, and expansion in market share [12]. ...
... In non-financial research, the performance of a firm was examined by efficiency, effectiveness, profitability, quality of service, client satisfaction, and productivity [6]- [9], [11], [12], [15]. This is their satisfaction level about their firm's outcome in terms of growth in sale, increase in profits, and expansion in market share [12]. With investigating these studies, this research describes a firm outcome as the effectiveness and efficiency of its business activities that can be upgraded by applying firm business ability to its business activities. ...
... With investigating these previous studies, this research generates the measurement factors and items to gauge enterprise outcome in terms of a smart business (SB) as follows: SB execution performance (quality of services, efficiency of business process, and client satisfaction), SB increase performance (sale increase, sale revenue increase, market increase), SB benefit performance (increase of gain in annual profit, net income increase, and cash turnover ratio), and SB competitiveness performance (sale increase rate and customer share) [6], [7], [9], [10], [12]- [15]. Our research utilize these articles as measures to gauge the enterprise SB performance through the verification process of validity analysis and reliability analysis according to the criterion of previous literature. ...
Article
Smart business management has been built to efficiently carry out enterprise business activities and improve its business outcomes in a global business circumstance. Firms have applied their smart business to their business activities in order to enhance the smart business results. The outcome of an enterprise's smart business fulfillment has to be managed and measured to effectively establish and control the smart business environment based on its business plan and business departments. In this circumstance, we need the measurement framework that can reasonably gauge a firm's smart business output in order to control and advance its smart business ability. This research presents a measurement instrument for an enterprise smart business performance in terms of a general smart business outcome. The developed measurement scale is verified on its validity and reliability through factor analysis and reliability analysis based on previous literature. This study presents an 11-item measurement tool that can reasonably gauge a firm smart business performance in both of finance and non-finance perspective.
... The cited projects carried out an in-depth analysis of the activities to be carried out in the kitchen for feeding and the support that can be offered by the technology in the environment and/or remote and by networked people. In addition, the necessity of generalizing approaches to increase usability and grant accessibility to all people with adaptivity and adaptability was clearly pointed out [4][5][6]. ...
Conference Paper
In the “Design4All” project, a hardware and software architecture is under development for the implementation of adaptable and adaptive applications aimed to support all people in carrying out an independent life at home. In this paper, the problems of interactions with applications implemented in the Android platform, chosen for the experiments of interaction with the developed applications, are discussed with main emphasis on two main aspects: (i) the use of facilities supporting accessibility available in the most commonly used operating systems (mainstreaming) and (ii) the portability of solutions across different platforms.
Chapter
The smart business capability of a firm is very critical for the efficient execution of its business activities and for effective improvement of the performance of business tasks in a global business environment. In this business environment, the management and measurement of a firm smart business capability need to efficiently establish and improve the smart business environment appropriate for its management strategy and major business departments. This research presents a comprehensive measure of a firm smart business capability to totally manage and improve its smart business capability. The validity and reliability of the developed 14-item scale is verified by factor analysis and reliability analysis based on previous literature.
Chapter
Enterprises are endeavoring to effectively apply smart business technology to their management activities in order to raise their business results in a smart business environment. Enterprise’s smart business capability is very critical for the efficient execution of its management activities and to improve the performance of business tasks in a global management environment. An efficient measurement framework is necessary for efficiently measuring a firm’s smart business capability to manage and improve its smart business capability. The developed 12-item scale was verified based on previous literature. We found a 12-item framework that can reasonably gauge an enterprise smart business capability. This framework can be used for efficiently measuring a firm’s smart business capability in a comprehensive perspective.
Conference Paper
Mobile devices programming has emerged as a new trend in software development. The main developers of operating systems for such devices have provided APIs for developers to implement their own applications, including different solutions for developing voice control. Android, the most popular alternative among developers, offers libraries to build interfaces including different resources for graphical layouts as well as speech recognition and text-to-speech synthesis. Despite the usefulness of such classes, there are no strategies defined for multimodal interface development for Android systems, and developers create ad-hoc solutions that make apps costly to implement and difficult to compare and maintain. In this paper we propose a framework to facilitate the software engineering life cycle for multimodal interfaces in Android. Our proposal integrates the facilities of the Android API in a modular architecture that emphasizes interaction management and context-awareness to build sophisticated, robust and maintainable applications.
Article
Enterprise business environment has rapidly been changed with developing a variety of smart technologies. Firms are endeavoring to effectively apply them to their management activities in order to raise their competitiveness. In this business environment, the smart business (SB) capability of an enterprise is very critical for the efficient execution of its management activities and to improve the performance of business tasks. The analysis and management for a firm SB capability need to efficiently establish and improve SB environment appropriate for its management strategy and business departments. A scientific and Practical analysis tool is necessary for efficiently analyzing an enterprise's SB capability to manage and improve its SB capability. The validity and reliability of the developed 18-item scale are verified based on previous literature. This research finds an 18-item tool that can reasonably analyze an enterprise SB capability in a total SB capability perspective. The developed tool can be utilized for effectively analyzing an enterprise's SB capability in a comprehensive perspective.
Article
Full-text available
The challenge of designing universal access to knowledge demands considerations on multi-device interaction. A systematic review of inclusive environments built from multiple devices was conducted based on studies published during the period of 2002–2013. The search strategy combined manual and automatic searches from which 8889 studies were identified; 34 studies were found proposing software tools for building multi-device inclusive environments (0.38 % of the original sample). Thus, this study analyzes the ways academic and industrial communities have developed tools for building inclusive environments. The main findings of this review are: (1) an urgent need for the recognition of accessibility as an important non-functional requirement; (2) a need for taking into account the social conditions of users, such as illiteracy and people living in underserved communities; and (3) the identification of new research questions in the context of multi-device inclusive environments.
Article
Full-text available
There are a number of noteworthy advancements in the sensor technology domain. Sensors have become the eye and ear of new-generation IT. Sensor integration and fusion developments clearly portend a great future for sensor technologies in the realm of human society. Sensors are becoming smart and capable of communicating with one another wirelessly. Ad hoc networks of sensors are being readily formed and disbanded once the goal behind the network formation is over. Sensor-centric services are being highly popular and pioneering to sharply enhance personal as well as professional lives of humans. This research work deals with creating and maintaining a smart hotel. The existing hotel services and applications are remarkably increased with the synchronization of wireless sensors. The smart hotel is an environment that would make the entire stay a truly memorable and mesmerizing experience. The already available functionalities of a luxurious hotel are being significantly enhanced by the assimilation, adoption and adaption of various technologies such as ZigBee, Bluetooth, etc. Device profile for web services (DPWS) is the shrunken standard for creating best-of-breed middleware implementation that enables diverse and distributed devices to find, connect and collaborate.
Article
Full-text available
Without an iota of doubt, technology has been instrumental in envisioning and realizing a host of hither-to unforeseen things in our daily works and walks. It can be pervasive and still deliver many services without any loss in quality. With technology various devices are being integrated with our already existing environment to make them more comfortable or people friendly. One such scenario has been described here where a unique use case has been presented that can enhance the quality of train travel for travelers through the deeper connectivity and spontaneous integration of a plethora of electronic devices. Here for this example, a wireless sensor network had been dynamically created that would periodically obtain information from various parts of every train coach and contribute for the improvement of the quality of services being provided to the passengers. DPWS, a leading service-based device integration standard specification has been utilized as the device middleware in order to build this integrated application that serves as the foundation in making this use case a reality.
Article
Full-text available
The emergence of the Information Society is creating a complex intelligent environment where all citizens have to live. Even if the final technological embodiment of this new environment has not been completely determined, the general lines of development that are at present being discussed suggest that the normal approach to the problems of integration of people with disabilities, based on the adaptation of already developed technology, is no longer tenable. Therefore, on the one hand, it is necessary to redefine the integration problems of people with disabilities on the basis of the foreseen characteristics of the emerging environment; on the other hand, a procedure needs to be defined whereby new technology is produced so as to be accessible without adaptations. The corresponding concepts are discussed in the paper, followed by a summary of activities already carried out to substantiate the presented conclusions.
Conference Paper
Full-text available
This paper provides an overview of research efforts in the area of accessibility over the past decade in Europe, and follows the evolution of Research and Technological Development work from solutions based on 'a posteriori' daptation to the notion of User Interfaces for All. The aim of the paper is to outline the beginning of an evolutionary path driving from reactive accessibility solutions to the requirement for Universal Access in the Information Society.
Article
Full-text available
Virtual reality techniques allow one to interact with synthetic worlds, i.e. virtual environments. Tactile feedback means conveying parameters such as roughness, rigidity, and temperature. These information and many others are obtained thanks to the sense of touch. Tactile feedback is of prime importance in many applications. In this paper we examine the state-of-the-art in tactile interfaces design. Thorough reviews of the literature reveal a significant amount of publications concerning tactile human-machine interfaces, especially onwards the ~1990. This paper reports the progress in tactile interfaces technology in the following areas: teleoperation, telepresence, sensory substitution, 3D surface generation, Braille systems, laboratory prototypes, and games. Different parameters and specifications required to generate and feed back tactile sensation are summarized.
Chapter
Full-text available
SmartKom provides an adaptive and reusable dialogue shell for multimodal interaction that has been employed successfully to realize fully-fledged prototype systems for various application scenarios. Taking the perspective of system architects, we review the overall design and specific architecture framework being applied within SmartKom. The basic design principles underlying our approach are described, and the different instantiations of the conceptual architecture are presented to illustrate the adaptibility and flexibility of the generic framework.
Conference Paper
Full-text available
In this paper we propose a software architecture for observing and modeling human activity. This architecture is derived from an ontology for context awareness. We propose a model in which a user’s context is described by a set of roles and relations. Different configurations of roles and relations correspond to situations within the context. The components of a context model are used to specify processes for observing activity. The ontology for context modeling is derived from both a bottom up system’s perspective and a top-down users’ perspective. As we define each element, we describe the corresponding components of a process-based software architecture. Using these components, a context is translated into a federation of observational processes. This model leads to an architecture in which reflexive elements are dynamically composed to form federations of processes for observing and predicting the situations that make up a context.
Article
Full-text available
In the years ahead, as a result of the increasing demand for ubiquitous and continuous access to information and services, information technologies are expected to evolve toward a new computing paradigm known as ambient intelligence. Ambient intelligence will be characterized by invisible (i.e., embedded) computational power in everyday appliances and other common physical objects, including intelligent mobile and wearable devices. Ambient intelligence will have profound consequences on the type, content, and functionality of emerging products and services, as well as on the way people will interact with them, bringing about multiple new requirements for the development of information technologies. In addressing this challenge, the concept of universal access is critical. This paper discusses the anticipated opportunities and challenges that ambient intelligence will bring about for elderly people and people with disabilities, envisages new scenarios in the use of ambient-intelligence technologies by users with diverse needs and requirements, and identifies some of the critical issues that will have to be addressed.
Conference Paper
Full-text available
The mapping problem has been defined as the way to map models involved throughout the development life cycle of user interfaces. Model-based design of user interfaces has followed a long tradition of establishing models and maintaining mappings between them. This paper introduces a formal defini- tion of potential mappings between models with its corresponding syntax so as to create a uniform and integrated framework for adding, removing, and modi- fying mappings throughout the development life cycle. For the first time, the mappings can be established from any source model to any target model, one or many, in the same formalism. Those models include task, domain, presentation, dialog, and context of use, which is itself decomposed into user, platform, and environment. IDEALXML consists of a Java application allowing the designer to edit any model at any time, and any element of any model, but also to establish a set of mappings, either manually or automatically based on a mapping model.
Conference Paper
Full-text available
This article presents FAME, a model-based Framework for Adaptive Multimodal Environments. FAME proposes an architecture for adaptive multimodal applications, a new way to represent adaptation rules - the behavioral matrix - and a set of guidelines to assist the design process of adaptive multimodal applications. To demonstrate FAME’s validity, the development process of an adaptive Digital Talking Book player is summarized.
Conference Paper
Full-text available
In this paper we discuss the implications of recent research studies on disability-related design guidelines. We have investigated the quality of guidelines with respect to designers as their end-users, and we have conducted field studies of the use design resources in practice. We now look at gaps in the current knowledge regarding the conceptualized system that comprises: the designer of technology, end-users of technologies, and guideline-setting committees. We look at the practice of setting up accessibility program offices in large companies as a means to tackle accessibility issues, and examine the implications of this practice for product designers, and people creating disability-based guidelines for technology.
Article
Full-text available
The ways by which large-scale ubiquitous computing system can be tackled with a structure, flexible approach are illustrated. Ubiquitous computing embraces a model of users, services and resources discovering other users, services and resources and integrate. A conceptual framework which includes an ontological and an architectural foundation that structure the adaptation process in a unified way is also discussed. The importance of adaptation, which allows a system to maintain consistent behavior across variations in operating environments is also discussed.
Conference Paper
Full-text available
Speech interfaces have become increasingly popular to support mobile commerce. While interface frameworks and dialogue management have been well studied, there is a lack of analytical research on the acceptance of speech-enabled m-commerce services. In this paper, we propose a framework to explain the acceptance of m-commerce from the consumers' perspective. The framework incorporates value propositions of m-service into a service acceptance model derived from a meta-analysis. The significance of a speech interface to m-commerce is evaluated and confirmed in line with the proposed framework. Strategies for developing speech-enabled applications are also suggested.
Article
Full-text available
Device manufacturers, users, and authors have differing needs and expectations when it comes to Web content. Web software and hardware manufacturers naturally try to differentiate their products by supporting a special combination of capabilities, but few can expect Web authors to create content for their product alone. Users, however, do expect to access the same content from any device with similar capabilities. Even when device capabilities differ, users might still want access to an adapted version of the content. Due to device differences, the adaptation might not produce an identical presentation, but device-independence principles suggest it should be sufficiently functional to let users interact with it successfully. Web application authors cannot afford to create multiple content versions for each of the growing range of device types. Authors would rather create their content once, and adapt it to different devices-but they also want to retain control of presentation quality. Device independence is about trying to satisfy these differing needs, spanning the delivery path between author and user by way of diverse manufacturers' devices. The field's continued evolution within the broader Web standards framework aims to find solutions that are beneficial for all.
Article
Full-text available
The use of hand gestures provides an attractive alternative to cumbersome interface devices for human-computer interaction (HCI). In particular, visual interpretation of hand gestures can help in achieving the ease and naturalness desired for HCI. This has motivated a very active research area concerned with computer vision-based analysis and interpretation of hand gestures. We survey the literature on visual interpretation of hand gestures in the context of its role in HCI. This discussion is organized on the basis of the method used for modeling, analyzing, and recognizing gestures. Important differences in the gesture interpretation approaches arise depending on whether a 3D model of the human hand or an image appearance model of the human hand is used. 3D hand models offer a way of more elaborate modeling of hand gestures but lead to computational hurdles that have not been overcome given the real-time requirements of HCI. Appearance-based models lead to computationally efficient “purposive” approaches that work well under constrained situations but seem to lack the generality desirable for HCI. We also discuss implemented gestural systems as well as other potential applications of vision-based gesture recognition. Although the current progress is encouraging, further theoretical as well as computational advances are needed before gestures can be widely used for HCI. We discuss directions of future research in gesture recognition, including its integration with other natural modes of human-computer interaction
Article
Full-text available
The increasing availability of new types of interaction platforms raises a number of issues for designers and developers. There is a need for new methods and tools to support development of nomadic applications, which can be accessed through a variety of devices. We present a solution, based on the use of three levels of abstractions, that allows designers to focus on the relevant logical aspects and avoid dealing with a plethora of low-level details. We have defined a number of transformations able to obtain user interfaces from such abstractions, taking into account the available platforms and their interaction modalities while preserving usability. The transformations are supported by an authoring tool, TERESA, which provides designers and developers with various levels of automatic support and several possibilities for tailoring such transformations to their needs.
Book
From the voice on the phone, to the voice on the computer, to the voice from the toaster, speech user interfaces are coming into the mainstream and are here to stay forever. Soundly anchored in HCI, cognitive psychology, linguistics, and social psychology, this supremely practical book is loaded with examples, how-to advice, and design templates. Drawing widely on decades of research-in lexicography, conversation analysis, computational linguistics, and social psychology-author Randy Allen Harris outlines the principles of how people use language interactively, and illustrates every aspect of design work. In the first part of the book, Harris provides a thorough conceptual basis of language in all its relevant aspects, from speech sounds to conversational principles. The second part takes you patiently through the entire process of designing an interactive speech system: from team building to user profiles, to agent design, scripting, and evaluation. This book provides interaction designers with the knowledge and strategies to craft language-based applications the way users will expect them to behave. *Loaded with examples and practical synopses of the best practice. *An ideal combination of conceptual base, practical illustrations, and "how-to" advice-for design and for the entire design process. *Will bring novice voice designers fully up to speed, and give experienced designers a new understanding of the principles underlying human speech interaction, principles from which to improve voice interaction design. From the voice on the phone, to the voice on the computer, to the voice from the toaster, speech user interfaces are coming into the mainstream and are here to stay forever. Soundly anchored in HCI, cognitive psychology, linguistics, and social psychology, this supremely practical book is loaded with examples, how-to advice, and design templates. Drawing widely on decades of researchin lexicography, conversation analysis, computational linguistics, and social psychologyauthor Randy Allen Harris outlines the principles of how people use language interactively, and illustrates every aspect of design work. In the first part of the book, Harris provides a thorough conceptual basis of language in all its relevant aspects, from speech sounds to conversational principles. The second part takes you patiently through the entire process of designing an interactive speech system: from team building to user profiles, to agent design, scripting, and evaluation. This book provides interaction designers with the knowledge and strategies to craft language-based applications the way users will expect them to behave.
Book
Integrating new input-output modalities, such as speech, gaze, gestures, haptics, etc., in user interfaces is currently considered as a significant potential contribution to implementing the concept of Universal Access (UA) in the Information Society (see, for example, Oviatt, 2003). UA in this context means providing everybody, including disabled users, with easy humancomputer interaction in any context of use, and especially in mobile contexts. However, the cost of developing an appropriate specific multimodal user interface for each interactive software is prohibitive. A generic design methodology, along with generic reusable components, are needed to master the complexity of the design and development of interfaces that allow flexible use of alternative modalities, in meaningful combinations, according to the constraints in the interaction environment or the user's motor and perceptual capabilities. We present a design approach meant to facilitate the development of generic multimodal user interfaces, based on best practice in software and user interface design and architecture.
Conference Paper
An essential distinguishing capability of multimodal systems is the use these systems make of two or more natural input modalities such as speech, handwriting, gestures, facial expressions, and other body movements. Using such systems, users deal with a crisis management situation using speech and pen input over a map. An emergency response route can be established by sketching a line on a map along the desired route, using a digital pen or a Tablet PC stylus, while speaking “Create emergency route here.” Multimodal interfaces take many forms and have many aspects. Small portable devices host systems, such as on a cell phone that is taken to the field or on a tablet computer used within offices or in cafeterias. Groups of people may interact multimodally via large interactive boards or sheets of digital paper. The common aspect in all these cases is the interface support for interaction via natural modes of communication involving combinations, for example, of speech, pen input, gestures, gaze, or other modalities. The intrinsic advantage of multimodal systems is that they allow users to convey their intentions in a more expressive way, better matched to the way they naturally communicate. A well-designed multimodal system gives users the freedom to choose the modality that they feel best matches the requirements of the task at hand.
Article
Multimodality allows adaption of human-computer interac-tion on two different levels: on the one hand different users with different abilities can interact with information technology through a customized configuration of interaction devices supporting their individual needs. On the other hand output can be adapted to user context within different en-vironments, e.g. in the car, in the living room, in the office. This position paper introduces two aspects of our approach to multimodal interaction developed in the EMBASSI and the DynAMITE project with the special focus on the Ambient Intelligence initiative of the INI-GraphicsNet.
Article
Two video-based human-computer interaction tools are introduced that can activate a binary switch and issue a selection command. BlinkLink, as the first tool is called, automatically detects a users eye blinks and accurately measures their durations. The system is intended to provide an alternate input modality to allow people with severe disabilities to access a computer. Voluntary long blinks trigger mouse clicks, while involuntary short blinks are ignored. The system enables communication using blink patterns: sequences of long and short blinks which are interpreted as semiotic messages. The second tool, EyebrowClicker, automatically detects when a user raises his or her eyebrows and then triggers a mouse click. Both systems can initialize themselves, track the eyes at frame rate, and recover in the event of errors. No special lighting is required. The systems have been tested with interactive games and a spelling program. Results demonstrate overall detection accuracy of 95.6% for BlinkLink and 89.0% for EyebrowClicker.
Article
Multimodal interfaces are inherently flexible, which is a key feature that makes them suitable for both universal access and next-generation mobile computing. Recent studies also have demonstrated that multimodal architectures can improve the performance stability and overall robustness of the recognition-based component technologies they incorporate (e.g., speech, vision, pen input). This paper reviews data from two recent studies in which a multimodal architecture suppressed errors and stabilized system performance for accented speakers and during mobile use. It concludes with a discussion of key issues in the design of future multimodal interfaces for diverse user groups.
Article
This article argues that future generations of computer-based systems will need cognitive user interfaces to achieve sufficiently robust and intelligent human interaction. These cognitive user interfaces will be characterized by the ability to support inference and reasoning, planning under uncertainty, short-term adaptation, and long-term learning from experience. An appropriate engineering framework for such interfaces is provided by partially observable Markov decision processes (POMDPs) that integrate Bayesian belief tracking and reward-based reinforcement learning. The benefits of this approach are demonstrated by the example of a simple gesture-driven interface to an iPhone application. Furthermore, evidence is provided that humans appear to use similar mechanisms for planning under uncertainty.
Article
Content adaptation has shown significant benefits by enhancing Web accessibility to meet user’s requirements in heterogeneous environments. Adaptation rules can provide an efficient way to transform Web contents into adapted ones that conform to requesters’ contexts. However, the rule base management becomes a challenge while taking requesters’ contexts into account. In this paper, we refer requesters’ contextual requirements (or contexts in short) to the information that may affect their perceptions of browsing Web contents, such as requesters’ preferences and sensations, situated places, as well as devices and network connectivity. We have used JESS to design and implement dynamic adaptation strategies to guide the transformation process. Adaptation strategies are designed to improve rule base efficiency by performing dynamic scheduling of rule firing in an agenda. An adaptation strategy is composed of selected adaptation rules based on requesters’ contexts. When a user’s context changed, new adaptation strategy will be generated automatically by rearranging adaptation rules. By using JESS, we can dramatically reduce the number of adaptation rules from 54 to 22. The experiment results indicate our adaptation strategy can successfully adapt Web contents to satisfy requesters’ contexts and obtain superior performance.
Article
This paper discusses aspects related to the implementation of Design for All in ICT from a developer oriented perspective. In order to illustrate and exemplify the concepts, the discussion refers to characteristics implemented in the web site of the Italian EDeAN network. The paper proposes adaptation capable platforms as a possible organic solution for instantiating Design for All principles. It presents a modular platform that makes it possible to build services of various types, with focus on its adaptation framework. The platform is inspired by Design for All principles in that it is intended to give support, from the design time on, to a wide range of mobile and wired communication devices, a variety of user characteristics, preferences, and contexts.
Article
Designing universally accessible user interfaces means designing for diversity in end-users and contexts of use, and implies making alternative design decisions, at various levels of the interaction design, inherently leading to diversity in the final design outcomes. Towards this end, a design method leading to the construction of a single interface design instance is inappropriate, as it cannot accommodate for diversity of the resulting dialogue artifacts. Therefore, there is a need for a systematic process in which alternative design decisions for different design parameters may be supported. The outcome of such a design process realizes a design space populated with appropriate designed dialogue patterns, along with their associated design parameters (e.g. user- and usage-context-attribute values). This paper discusses the Unified Interface Design Method, a process-oriented design method enabling the organization of diversity-based design decisions around a single hierarchical structure, and encompassing a variety of techniques such as task analysis, abstract design, design polymorphism and design rationale.
Article
Model-based interface development systems have not been able to progress beyond producing narrowly focused interface designs of restricted applicability. We identify a level-of-abstraction mismatch in interface models, which we call the mapping problem, as the cause of the limitations in the usefulness of model-based systems. We propose a general computational framework for solving the mapping problem in model-based systems. We show an implementation of the framework within the MOBI-D (Model-Based Interface Designer) interface development environment. The MOBI-D approach to solving the mapping problem enables for the first time with model-based technology the design of a wide variety of types of user interfaces.
Article
The notion of ‘universal access’ reflects the concept of an Information Society in which potentially anyone (i.e. any user) will interact with computing machines, at anytime and anyplace (i.e. in any context of use) and for virtually anything (i.e. for any task). Towards reaching a successful and cost effective realization of this vision, it is critical to ensure that the future interface development tools provide all the necessary instrumentation to support inclusive design, i.e. facilitate inclusive development. In the meantime, it is crucial that both tool developers and interface developers acquire awareness regarding the key development features they should pursue when investigating for the most appropriate software engineering support in addressing such a largely demanding development goal (i.e. universally accessible interactions). This paper discusses a corpus of key development requirements for building universally accessible interactions that has been consolidated from real practice, in the course of six medium-to-large scale research projects, all completed, within a 10 years timeframe.
Article
In this paper, we review the major approaches to multimodal human–computer interaction, giving an overview of the field from a computer vision perspective. In particular, we focus on body, gesture, gaze, and affective interaction (facial expression recognition and emotion in audio). We discuss user and task modeling, and multimodal fusion, highlighting challenges, open issues, and emerging applications for multimodal human–computer interaction (MMHCI) research.
Conference Paper
We present an approach to control information flow in object-oriented systems. The decision of whether an informatin flow is permitted or denied depends on both the authorizations specified on the objects and the process by which information is obtained ...
Conference Paper
Integrating new input-output modalities, such as speech, gaze, gestures, haptics, etc., in user interfaces is currently considered as a significant potential contribution to implementing the concept of Universal Access (UA) in the Information Society (see, for example, Oviatt, 2003). UA in this context means providing everybody, including disabled users, with easy human-computer interaction in any context of use, and especially in mobile contexts. However, the cost of developing an appropriate specific multimodal user interface for each interactive software is prohibitive. A generic design methodology, along with generic reusable components, are needed to master the complexity of the design and development of interfaces that allow flexible use of alternative modalities, in meaningful combinations, according to the constraints in the interaction environment or the user’s motor and perceptual capabilities. We present a design approach meant to facilitate the development of generic multimodal user interfaces, based on best practice in software and user interface design and architecture.
Conference Paper
Ambient Intelligence (AmI) scenarios place strong emphasis on the fact that interaction takes place through natural interfaces, in such a way that people can perceive the presence of smart objects only when needed. As a possible solution to achieving relaxed and enjoyable interaction with the intelligent environments depicted by AmI, the ambient could be equipped with suitably designed multimodal interfaces bringing up the opportunity to communicate using multiple natural interaction modes. This paper discusses challenges to be faced when trying to design multimodal interfaces that allow for natural interaction with systems, with special attention to speech-based interfaces. It describes an application that was built to serve as a test bed and to conduct evaluation sessions in order to ascertain the impact of multimodal natural interfaces on users and to assess their usability and accessibility.
Conference Paper
This paper identifies several factors that were observed as being crucial to the usability of multimodal in-vehicle applications – a multimodal system is not of value in itself. Focusing in particular on the typical combination of manual and voice control, this article describes important boundary conditions and discusses the concept of natural interaction.
Conference Paper
Efficient, effective and intuitive access to multimedia information is essential for business, education, government and leisure. Unfortunately, interface design typically does not account for users with disabilities, estimated at 40 million in America alone. Given broad societal needs, our community has a social responsibility to provide universal designs that ensure efficient and effective access for all to heterogeneous and increasingly growing repositories of global information. This article describes information access functions, discusses associated grand challenges, and outlines potential benefits of technologies that promise to increase overall accessibility and success of interaction with multimedia. The article concludes by projecting the future of multimodal technology via a roadmap of multimodal resources, methods, and systems from 2003 through 2006.
Article
The ability to adapt to changes is among the most important issues in mobile information systems. We propose a rule-based modular framework for building self-adaptive applications in mobile environments. A common mobile event engine for all applications is employed to detect the status changes. The rule system plays the role of a reactive component for making proper decisions in response to the changes. A general database interface ensures seamless integration of the rule system with database systems for flexible information access. For efficient rule processing, we develop techniques that combine static and dynamic analysis to uncover phase structure and data access semantics of a rule program. The semantic information is used to facilitate intelligent caching and prefetching for conserving limited bandwidth and reducing rule processing cost. A distributed data management and rule execution strategy enable adaptive information services over changing environment, even under disconnection. We devise a performance model to characterize the exact condition for our approach to be superior than traditional approach. Trace-driven simulation results successfully demonstrate the feasibility and potential of our approach. The modularity of the framework permits fast implementation based on off-the-shelf rule systems and database packages. We present a prototype implementation and preliminary evaluation results to show that our techniques can be effectively materialized.
Article
This paper discusses the contribution of adaptive techniques to Universal Access in Human–Computer Interaction. To this effect, the paper revisits the concept of Universal Access, briefly reviews relevant work on adaptive techniques, and follows their application in efforts to provide accessibility of interactive systems, from the early, product- and environment-level adaptation-based approaches, to more generic solutions oriented towards Universal Access. Finally, the paper highlights some of the research challenges ahead. The normative perspective of the paper is that adaptive techniques in the context of Universal Access have the potential to facilitate both accessibility and high quality interaction, for the broadest possible end-user population. This implies the design of systems that undertake context-sensitive processing so as to manifest their functional core in alternative interactive embodiments suitable for different users, usage patterns and contexts of use. Such a capability needs to be built into the system from the early phases of conception and design, and subsequently validated throughout its life cycle.
Article
Recent technological advances in connected-speech recognition and position sensing in space have encouraged the notion that voice and gesture inputs at the graphics interface can converge to provide a concerted, natural user modality. The work described herein involves the user commanding simple shapes about a large-screen graphics display surface. Because voice can be augmented with simultaneous pointing, the free usage of pronouns becomes possible, with a corresponding gain in naturalness and economy of expression. Conversely, gesture aided by voice gains precision in its power to reference.
Article
Our aim is to point out the benefits that can be derived from research advances in the implementation of such concepts as ambient intelligence and ubiquitous/pervasive computing for promoting Universal Access in the Information Society, that is, for contributing to enable everybody, especially people with physical disabilities, to have easy access to all computing resources and information services that the coming worldwide Information Society will soon make available to the general public as well as to expert users. Following definitions of basic concepts relating to multimodal interaction, the significant contribution of multimodality to developing Universal Access is briefly discussed. Then, a short state of the art in ambient intelligence research is presented, including references to some major research projects in progress or recently completed. The last section is devoted to bringing out the potential contribution of advances in ambient intelligence research and technology to the improvement of computer access for physically disabled people, hence, to the implementation of Universal Access.
Conference Paper
Modern technology promises. mobile users Internet connectivity anytime, anywhere, using any device. However, given the constrained capabilities of mobile devices, the limited bandwidth of wireless networks and the varying personal sphere, effective information access requires the development of new computational patterns. The variety of mobile devices available today makes device-specific authoring of Web content an expensive approach. The problem is further compounded by the heterogeneous nature of the supporting networks and user behaviour. The notions of "typical user" and "typical user behaviour" are no longer applicable for many Internet applications. This research investigates the challenges posed by these problems, and proposes FACADE (FrAmework for Context-aware content Adaptation and DElivery) to bridge the gap between the existing Internet content and today's heterogeneous computing environments. A pilot implementation of a facility for testing and performance evaluation of FACADE is also discussed.
Article
In this article, we describe an adaptation proxy we developed as a part of a Kontti research project at VTT Information Technology that lets mobile users access Web content that's not directly targeted to mobile user agents. More and more content is now available on the Internet, and there's a growing need for mobile users to be able to access it. Thus the authors describe the adaptation proxy, which lets mobile users access Web content that's not directly targeted to user agents of mobile devices. The adaptation proxy can adapt Extensible Hypertext Markup Language (XHTML) documents into XHTML mobile profile (XHTML MP) and Wireless Markup Language (WML), and can perform media adaptation. At the system's core is an adaptation framework to which new source and target XML languages can be introduced with relatively little effort.
Article
Historically, the development of computer interfaces has been a technology-driven phenomenon. However, new multimodal interfaces are composed of recognition-based technologies that must interpret human speech, gesture, gaze, movement patterns, and other complex natural behaviors, which involve highly automatized skills that are not under full conscious control. As a result, it now is widely acknowledged that multimodal interface design requires modeling of the modality-centered behavior and integration patterns upon which multimodal systems aim to build. This paper summarizes research on the cognitive science foundations of multimodal interaction, and on the essential role that user-centered modeling has played in prototyping, guiding, and evaluating the design of next-generation multimodal interfaces. In particular, it discusses the properties of different modalities and the information content they carry, the unique features of multimodal language and its processability, as well as when users are likely to interact multimodally and how their multimodal input is integrated and synchronized. It also reviews research on typical performance and linguistic efficiencies associated with multimodal interaction, and on the user-centered reasons why multimodal interaction minimizes errors and expedites error handling. In addition, this paper describes the important role that selective methodologies and evaluation metrics have played in shaping next-generation multimodal systems, and it concludes by highlighting future directions for designing a new class of adaptive multimodal-multisensor interfaces.
Conference Paper
We present our current research on the implementation of gaze as an efficient and usable pointing modality supplementary to speech, for interacting with augmented objects in our daily environment or large displays, especially immersive virtual reality environments, such as reality centres and caves. We are also addressing issues relating to the use of gaze as the main interaction input modality. We have designed and developed two operational user interfaces: one for providing motor-disabled users with easy gaze-based access to map applications and graphical software; the other for iteratively testing and improving the usability of gaze-contingent displays.
Multimodal interfaces The human–computer interaction handbook: Fundamentals, evolving technologies and emerging applications (pp. 286–304) Mahwah, NJ: Lawrence Erlbaum Assoc Visual interpretation of hand gestures for human–computer interaction: a review. Pattern Analysis and Machine Intelligence
  • S L Oviatt
  • V I Pavlovic
  • R Sharma
  • T S Huang
Oviatt, S. L. (2003). Multimodal interfaces. In J. Jacko & A. Sears (Eds.), The human–computer interaction handbook: Fundamentals, evolving technologies and emerging applications (pp. 286–304). Mahwah, NJ: Lawrence Erlbaum Assoc.. Pavlovic, V. I., Sharma, R., & Huang, T. S. (1997). Visual interpretation of hand gestures for human–computer interaction: a review. Pattern Analysis and Machine Intelligence. IEEE Transactions on, 19(7), 677–695. <http:// www.dx.doi.org/10.1109/34.598226>.
Put-that-there: Voice and gesture at the graphics interface Ambient intelligence and multimodality
  • R A L Bolt
  • P L Emiliani
  • F Gabbanini
Bolt, R. A. (1980). Put-that-there: Voice and gesture at the graphics interface. In Proceedings of the 7th annual conference on computer graphics and interactive techniques (pp. 262–270). Seattle, Washington: ACM Press. Bukovics, B. (2008). Pro WF Windows Workflow in.NET 3.5. APress. Burzagli, L., Emiliani, P. L., & Gabbanini, F. (2007). Ambient intelligence and multimodality. In Universal access in human–computer interaction. In C. Stephanidis (Ed.). Lecture Notes in Computer Science (Vol. 4555, pp. 33–42).
Multimodal interfaces for in-vehicle applications. In Human–Computer Interaction. HCI Intelligent Multimodal Interaction Environments
  • R Vilimek
  • T Hempel
  • B Otto
Vilimek, R., Hempel, T., & Otto, B. (2007). Multimodal interfaces for in-vehicle applications. In Human–Computer Interaction. HCI Intelligent Multimodal Interaction Environments. In J. Jacko (Ed.). LNCS (Vol. 4552, pp. 216–224).