ArticlePDF Available

Human-Computer Interaction with an Intelligence Virtual Analyst

Authors:

Abstract and Figures

Defence R&D Canada has undertaken an R&D initiative to investigate and develop Intelligent Software Assistant (ISA) technologies to support intelligence analysts in sense making tasks. One aspect of the project is related to the Human-Computer Interaction between the virtual assistant and the users. Two key requirements are to support a dialogue between the ISA and the analyst(s), and to optimize the presentation of the results. In order to address these requirements, a number of technologies need to be exploited, including smart room environments, multimodal interaction, adaptive interfaces, augmented cognition, avatars and storytelling.
Content may be subject to copyright.
1
Human-Computer Interaction with an
Intelligence Virtual Analyst
Denis Gouin, Valérie Lavigne, Alexandre Bergeron-Guyard
Defence R&D Canada, 2459 Pie XI Blvd North, Quebec, Qc, G3J 1X5, Canada
Contact: denis.gouin@drdc-rddc.gc.ca
ABSTRACT
Defence R&D Canada has undertaken an R&D initiative
to investigate and develop Intelligent Software Assistant
(ISA) technologies to support intelligence analysts in
sense making tasks. One aspect of the project is related to
the Human-Computer Interaction between the virtual
assistant and the users. Two key requirements are to
support a dialogue between the ISA and the analyst(s),
and to optimize the presentation of the results. In order to
address these requirements, a number of technologies
need to be exploited, including smart room environments,
multimodal interaction, adaptive interfaces, augmented
cognition, avatars and storytelling.
Author Keywords
Intelligent Software Assistant, Virtual Assistant, Human-
Computer Interaction, multimodal interaction, adaptive
interfaces, augmented cognition, avatars and storytelling.
ACM Classification Keywords
H5.m. Information interfaces and presentation (e.g., HCI):
Miscellaneous.
INTRODUCTION
Military intelligence analysts have a mandate to collect,
process and analyze information, and disseminate
required intelligence products. In the context of modern
dynamic military operations, analysts are faced with
overload problems (information, task, cognition) and
knowledge system technologies become important. In
order to better address these problems, it is relevant to go
beyond traditional approaches and make use of emerging
sense making tools.
Recently, a novel, very promising paradigm in artificial
intelligence (AI) has emerged: the Intelligent Software
Assistant (ISA), identified by MIT’s Technology Review
[1] as one of 2009’s most promising emerging
technologies. The idea behind the ISA is to synthesize the
current state of AI research and to develop a personalized
assistant that organizes information, learns processes,
adapts to changing situations, and interactively supports
individuals in their tasks in a seamless, intuitive fashion.
Defence R&D Canada (DRDC) has undertaken an R&D
project to investigate and develop ISA technologies
towards the development of an Intelligence Virtual
Analyst Capability (iVAC). The iVAC is meant as a
computerized software assistant supporting the
intelligence analysts in sense making tasks, while being
ultimately capable of taking on autonomous analytical
tasks in concert with other analysts (virtual or human).
One aspect of the project is related to the Human-
Computer Interaction of the iVAC, which is how users
and the ISA communicate with each others.
The initiative is a three-year project started in April 2011.
The first part of the project has consisted of a requirement
elicitation and a literature survey. This included the
identification of technologies that would support
intelligence analysts in sense making tasks, such as
natural language processing, knowledge engineering and
machine learning. One key aspect is the Human-
Computer Interaction between the virtual assistant and the
users, which includes the dialogue between the ISA and
the analyst(s), and the presentation of the results. In order
to address these requirements, a number of technologies
need to be exploited, such as smart room environments,
multimodal interaction, adaptive interfaces, augmented
cognition, avatars and storytelling.
RELATED WORK
During the period 2003 to 2008, the US Defense
Advanced Research Projects Agency (DARPA), through
the PAL (Personalized Assistant that Learns) program,
investigated and integrated a number of AI technologies
into CALO (Cognitive Assistant that Learns and
Organizes). A good vision of the use of an ISA in a
military context is provided in the PAL video [2].
Figure 1 shows the main CALO functions. Figure 2
provides examples of CALO’s learning capabilities.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,
or republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a
fee.
KSCO 2012, February 15th
17th, 2012, Pensacola, FL, USA.
Figure 1 - CALO Functions
Figure 2 - Examples of the CALO Learning Capabilities
In the early 2000’s, Defence Science and Technology
Organisation (DSTO), in Australia, as part of their Future
Operations Centre Analysis Laboratory (FOCAL)
initiative, has explored the use of virtual advisers.
“Virtual advisers are real-time, photo-realistic, animated
characters that dialogue with users through spoken-
language understanding and speech synthesisVirtual
advisers can brief the command team on a developing
situation using text, images, video, and other multimedia;
point out significant events for further attention; and
suggest alternative courses of action. By tuning the
characters’ appearance and combining facial gestures and
emotional cues, the virtual advisers can also provide
context and convey appropriate levels of trust[3].
Researchers at the Learning Agents Center from George
Mason University have developed a “personal cognitive
assistant, called Disciple-LTA, that captures analytic
expertise, provides effective analytic assistance, and train
new analysts and facilitate collaboration with
complementary experts and their agents” [4].
More recently, IBM has promoted Watson, a question
answering (QA) computing system based on their
DeepQA technology. It is an application of advanced
Natural Language Processing, Information Retrieval,
Knowledge Representation and Reasoning, and Machine
Learning technologies to the field of open domain
question answering[5]. Watson doesn’t have a tangible
HCI but has created quite a breakthrough in natural
language processing and exploitation of massive
unstructured data.
A spin-off of the PAL project, SIRI has developed ISA
technology onto the iPhone 4s. Siri understand a variety
of questions from the user, in natural language, related to
his day-to-day activities, such as contacting relatives,
holding meetings, looking for restaurants, and it can take
appropriate actions. So the user can tell Siri things like
3
Text Ryan I’m on my way’ or ‘Will it be sunny in Miami
this weekend?[6].
ISA AS A KNOWLEDGE-SYSTEM COMPONENT
ISA can be viewed as a Knowledge System technology
that help solve the human cognitive overload by providing
assistance to the user by conducting a wide variety of
tasks. This includes searching and organizing information,
tracking people, managing schedules, assigning tasks,
summarizing documents, mediating interactions, guiding
and reminding the user, learning procedures and
preferences.
IVA HCI REQUIREMENTS
Two high-level HCI requirements have been identified in
support of the iVAC. The first requirement is to support a
dialogue between the ISA and the analyst(s). This would
address the following general questions: How does a user
interact with the iVAC in a natural manner? How does a
user task the ISA? How does the ISA communicate the
results of a question to the user? How does a user input
knowledge into the system?
The second HCI requirement deals with the optimization
of the presentation of the results. This would address the
following general question: How does the ISA adapts the
interface to the users’ role, tasks and preferences?
In order to address these requirements, a number of
technologies need to be exploited, including smart room
environments, multimodal interaction, adaptive interfaces,
augmented cognition, avatars and storytelling.
Smart Room Environments
Although an ISA can run on a single computer and be
associated to the user of that computer, it is anticipated
that in the future, an ISA or ISAs would be available to
multiple users in a room. In this case, there would be a
need to identify the people in the room or suite of rooms,
determining their location and tracking their activities.
Ideally, the users would operate in a smart room
environment which can be defined as: a physical space
that is instrumented with various networked sensors and
devices (e.g. cameras, microphones, biometry,
identification tags, computer vision) and support
ubiquitous computing permitting to sense, interpret and
react to human activity in order to enable better
collaboration, increased productivity, creative thinking
and decision making. Collaboration can take place within
and across meeting rooms.
Multimodal Interaction
An ISA should allow the users to interact with the
system(s) using multimodal interaction (e.g. voice,
pointing, writing, drawing, gesture, eye/gaze, neural/brain
interfaces, emotion detection). Moreover, “full
understanding requires identification of speakers and
addressees, along with resolution of reference to other
participants and objects, and integration of both verbal
and non-verbal communication” [7]. In a military context,
as illustrated in the PAL video [2], the communication
with an ISA will evolve around topics such as tasks,
planning activities, standard operating procedures (SOPs),
briefing material, documents.
In many cases, as the user will be using natural language
interaction combined with movement, there will be some
ambiguities. For example, in [2] the user says: ‘These are
my priorities… I’m attending this meeting… I need you
to setup my briefing package, while pointing on the
screen. The deictic references to ‘my priorities and this
meetingcannot be solved by speech alone, and in some
cases the location being pointed to cannot be resolved to a
high-enough precision by vision alone.
Within the CALO initiative, in order to support the
understanding of the interaction taking place and resolve
ambiguities, a unified multimodal discourse ontology and
knowledge base was designed. This ontology is coupled
with a dialogue-understanding framework which
maintains and shares multiple hypotheses between
discourse-understanding components [7].
Information Presentation and Adaptive Interfaces
From an HCI perspective, the ISA should be able to
interact with the users by conducting several tasks:
Present tools and information; answer questions; ask for
clarification; remind the user of some tasks or procedures,
and propose alternative possibilities. The ISA may even
give a briefing (see Storytelling subsection). This
interaction is provided through voice output and/or
information display. As illustrated in the PAL video [2],
the ISA should be able to interact directly with the user’s
screen, in: highlighting information elements already
displayed; display information in a new window,
organized as the user wants to see it, gather a set of
documents; filter information based on user requests.
The ISA could also provide tools to capture information
about a meeting. For instance, the CALO Meeting
Assistant provides for distributed meeting capture,
annotation, automatic transcription and semantic analysis
of multiparty meetings [8].
The interface should adapt to the role and current tasks of
the user, as well as his/her preferences. The ISA should be
proactive, observing what the user is doing and his mental
state (see Augmented Cognition subsection), and suggest
sequences of events. If the user moves to a large screen
display, the user should be recognized using biometry and
the interface should adapt to the distance of the user to the
display, in terms of font sizes, granularity and quantity of
information presented.
Augmented Cognition
ISAs are there to support the analysts. They must
intervene when asked to or step in at appropriate
moments. The later requires more attention. There might
be instances when they should not disturb the users. On
the other hand, there are moments where the ISA will be
very welcomed, as it could provide augmented cognition.
Using neural interfaces and biometry, augmentation
cognition tracks the sensory and cognitive overload of the
users and employs computational strategies to restore
operational effectiveness. This includes: Intelligent
interruption to improve limited working memory;
attention management to improve focus during complex
tasks; cued memory retrieval to improve situational
awareness and context recovery; modality switching (i.e.,
audio, visual) to increase information throughput [9].
ISA Representation / Avatars
The ISA is represented in some way to interact with the
user. It could be in the form of a speech output (voice), a
simple icon or a two- or three-dimensional representation
of a character (avatar). All of these can be personified, in
particular from a gender perspective (male, female,
neutral). The avatars can also exhibit other characteristics
(age, ethnic group, profession - civil vs military) based on
the users’ social-cultural context and preferences. Facial
and voice features of the avatar (serious / smiling, tone of
the voice) could reflect the importance / urgency, or
certainty of a message. Different avatars could be used to
support different ISA tasks. For example, the avatar for
the weather analyst might be different, in terms of gender,
age and profession, than the one that recommends the
course of action.
In most cases, the avatar would be transmitting
information by talking. The voice and facial components
are then the important elements. However, in some cases,
such as during ‘on-the-map planning or video
conferencing activities’, it may be useful to have a full-
size avatar embedded in the display as if it were another
user. An example of this is in Figure 3 from PAL.
Figure 3 - ISA Avatar Embedded in the Display
Experiments conducted by DSTO and the University of
Adelaide have shown that different facial characteristics
displayed by a speaker will influence a user’s affective
trust, implicitly imparting uncertainty to the user if
required, which at times can be a vital form of
qualification for any information delivered” [10].
We must remain careful, however. Wark and Lambert
[11] discuss at length the spectre of the Uncanny Valley
(discovered by Masahiro Mori [12]) where it is postulated
that as you make a simulacrum look more human, people
will identify with it more strongly until a point is reached
as the simulacrum approaches ‘human looking’ where
their affinity will suddenly drop steeply, as the differences
become more important than the similarities. What level
of realism should we implement in the avatar? Will
Mori’s principle still hold as a new wave of people,
grown up with 3D / virtual reality gaming experience,
become military analysts? More experimentation with
avatars will be needed.
Storytelling
ISAs could be used to present information in the form of
stories. Because of its richness, “storytelling is recognized
as an effective mechanism for establishing shared context
and transferring tacit knowledge throughout an
organization” [11]. Analysts ultimately tell stories in their
presentations, with the stories providing a way to organize
evidence by events and by source documents [13].
[14] discusses the formalism of stories. Stories should be
organized around the actions or events, identifying the
actors, action type, modality of action, context, rationale,
but most importantly identify the relationships between
these.
DSTO has conducted interesting work in the form of
multimedia narratives, similar to television news services.
In this work, the virtual advisers act as automated story
tellers and combine narrative with multimedia
presentation, to convey situation awareness about
complex events [11].
IVA IN COALITION OPERATIONS
The recent significant advances made by IBM in its
Watson [5] and the inclusion of the SIRI features within
the IPhone 4s [6] will provide a great stimulus in moving
ISAs into the military. Military staff will be able to use
ISAs onto their Personal Device Assistants (PDAs) and
within military command centers. It expected that the
ISAs will be able to interact not only with users but with
one another; in the longer run, one can expect that ISAs
will also become available to support coalition operations.
In coalition settings, ISAs could provide the following
benefits: Improved interoperability between coalition
forces in terms of disseminating information; translating
information between languages; sharing differences in
tactics, techniques and procedures (TTPs); synchronizing
coalition activities; improving cognitive assistance by
5
providing shared coalition awareness and task support;
enabling coalition organizations to learn by managing a
knowledge base; and providing better and faster decision
making through access to comprehensive knowledge
developed through time.
CONCLUSION
ISA can be viewed as a Knowledge System technology
that helps solve the human cognitive overload by
providing assistance to the user by conducting a wide
variety of tasks, including searching and organizing
information, tracking people, managing schedules,
assigning tasks, summarizing documents, mediating
interactions, guiding and reminding the user, learning
procedures and preferences. One key aspect is how users
and ISAs communicate with each others. This paper has
presented a number of the enabling technologies and
associated requirements, in particular, smart room
environments, multimodal interaction, adaptive interfaces,
augmented cognition, avatars and storytelling.
REFERENCES
1. Naone, E.(2009), TR10:Intelligent Software Assistant,
Technology Review, MIT, March/April 2009,
http://www.technologyreview.com/read_article.aspx?c
h=specialsections&sc=tr10&id=22117
2. DARPA (2011), Personalized Assistant that Learns
video, http://www.youtube.com/watch?v=BF-
KNFlOocQ&feature=youtu.be
3. Wark, S., Broughton, M., Nowina-Krowicki, M.,
Zschorn, A., Coleman, M., Taplin, P. and Estival, D.
(2005), The FOCAL Point - Multimodal Dialogue
with Virtual Geospatial Displays, in Proc. SimTecT
2005, Sydney, AU, 9-12 May 2005
4. Tecuci, G., Boicu, M., Ayers, C. and Cammons, D.
(2005), Personal Cognitive Assistants for Military
Intelligence Analysis: MixedInitiative Learning,
Tutoring, and Problem Solving, in Proc ICIA,
McLean, VA, 26 May, 2005
5. IBM Corporation (2011), pQA Project: FAQ,
http://www.research.ibm.com/deepqa/faq.shtml
6. Apple (2011), Your Wish is its Command,
http://www.apple.com/iphone/features/siri.html
7. Purver, M., Niekrasz, J. and Peters, S. (2005),
Ontology-Based Multi-Party Meeting Understanding,
in Proc. CHI’05, Portland, OR, April 2005
8. Tur, G., Stolcke, A., Voss, L., Dowding, J., Favre, B.,
Fernandez, R., Frampton, M., Frandsen, M.,
Frederickson, C., Gracairena, M., Hakkani-Tür, D.,
Kintzing, D., Leveque, K., Mason, S., Niekrasz, J.,
Peters, S., Purver, M., Riedhammer, K., Shriberg, E.,
Tien, J., Vergyri, D., and Yang, F. (2008), The CALO
Meeting Speech Recognition and Understanding
System
9. Dorneich, M.C., Mathan, S., Creaser, J.I., Whitlow,
S.D., and Ververs (2005), Enabling Improved
Performance though a Close-Loop Adaptive System
Driven by Real-Time Assessment of Cognitive State,
in Proc. HCI (Augmented Cognition), Las Vegas, NV,
22-27 July 2005
10. Australian DefenceScience (2008), Clones that
Counsel - Technical Sheet, Australian Defence
Science, Volume 16, Number 2 2008
11. Wark S. and Lambert, D. - Presenting the Story
Behind the Data - Enhancing Situational Awareness,
in Proc. MILCOM 2007, Orlando, Fl, 29-31 Oct 2007
12. Mori, M. (1970), Bukimi no tami (The Uncanny
Valley), Energy, vol. 7, 1970, pp. 33-35
13. Bier, E.A., Card, S.K. and Bodnar, J.W. (2008)
Entitybased Collaboration Tools for Intelligence
Analysis, in Proc. IEEE Symposium on Visual
Analytics Science and Technology, 19-24 Oct 2008,
Columbus, OH, 99-106
14. Baber, C., Andrews, D., Duffy, T. and McMaster, R.,
Sensemaking as Narrative: Visualization for
Collaboration, in Proc. VAW2011, University London
College, 7-8 Sept 2011, London, UK
... The Intelligence Virtual Analyst Capability (iVAC) [7] is a recent Defence Research and Development Canada initiative that forms an intricate part of a Future Intelligence Analysis Capability (FIAC) [8]. iVAC is a knowledge system with an important human computer interface component that aims to alleviate the problem of cognitive overload by conducting a wide-variety of tasks. ...
... The RECON system, currently in development, aims at providing an innovative context-aware case-based recommendation framework for the intelligence virtual analyst capability (iVAC). This work builds on previous research in intelligence analysis [7][8], context-aware systems [9], BCI [13], human factors [11][12] and classification modeling [15] [20]. It is expected that in situations involving information overload, uncertainty, and time pressure, the effectiveness of intelligence analysts can be significantly improved through context-aware adaptive systems. ...
... The approach described in this paper relies heavily on psycho-physiological measurement to infer the user's cognitive state in order to implicitly coordinate the system and the user. An alternative approach is to focus on explicit coordination through human-machine teamwork, enabled through interaction with a virtual assistant [7]. The iVAC initiative seeks to combine these two complementary approaches. ...
Conference Paper
Full-text available
The REcommending Cases based on cONtext (RECON) system is a prototype adaptive technology designed to support intelligence analysis using dynamic load balancing and advanced human-machine synergy. RECON combines a brain-computer interface, machine learning, and simulation in order to create an innovative case-based recommendation capability. Several dilemmas emerge when designing joint cognitive systems endowed with an adaptive capacity. Herein, we critically discuss these dilemmas related to human modeling and human-computer interaction.
... The RECON (REcommending Cases based on cONtext) system is a recent initiative aimed at providing a capability for intelligence analysts that takes into account their need for relevant information consumption in a time-sensitive environment. As part of Defence Research & Development Canada's iVAC (Intelligent Virtual Analyst Capability) project [27], RECON uses an adaptive-systems approach for information offloading and filtering to assist intelligence analysts. To this end, it focuses on the following three objectives, and in this section both the general use case and resulting architecture for RECON are described: ...
Article
The REcommending Cases based on cONtext (RECON) system is a prototype adaptive technology designed to support intelligence analysts in overcoming the problem of cognitive overload. Its central objective is to assist these analysts during the collection, processing, and analysis phases of the intelligence cycle through sense-making of both explicit and implicit contextual information. RECON combines machine learning, text-analysis, brain-computer interfaces, and simulation to create an innovative case-based recommendation capability. In developing RECON, multiple considerations have been explored based on key HCI dilemmas that emerge when designing joint-cognitive systems endowed with an adaptive capacity. Herein, eight architectural design considerations are discussed, related to human-modelling, human-machine interaction, and human-machine synergy, which have impacted the system development. The central RECON architecture and its components are also presented, including a context-sensitive cognitive model based on COCOM. This work aims to provide these core architectural components and their design considerations as a contribution toward aiding developers in designing, customizing, and improving future adaptive contextmanagement systems.
... This work is conducted in the context of a Defence Research and Development Canada (DRDC) project called Intelligence Virtual Analyst Capability (iVAC) [2] which is an intricate part of the Future Intelligence Analysis Capability (FIAC) [3]. The iVAC project investigates and develops ISA technologies towards the development of a computerized software assistant supporting the intelligence analysts in sensemaking tasks, while being ultimately capable of taking on autonomous analytical tasks in concert with other analysts (virtual or human). ...
Article
Full-text available
This paper describes a framework to support the construction of intelligent virtual assistants (IVAs). An IVA agent is a software assistant capable of interacting with a user to support sense-making tasks, to determine information needs, to provide relevant information and to improve its performance based on user feedbacks. Currently, there is no integrated software environment available to develop such agents. We are exploring how we can integrate machine learning and natural language processing technologies, available as open source software, to support the construction of intelligent virtual assistants. The framework relies on a combination of question answering (Q/A), information extraction (IE) and user modeling components. In this paper, we present an overview of the work that is being conducted to build a prototype of the framework.
... For example, avatars may help users immerse in the virtual environment easily due to similarity-attraction and consistency-attraction (Lee and Nass, 2003). Recent advances in artificial intelligence (AI) have paved the way for the development of the Intelligent Software Assistant (ISA) that can organize information, learn processes, adapt to changing situations, and interactively support individuals in their tasks (Gouin et al., 2012). A recent successful implementation of ISA is the Apple iPhone's Speech Interpretation and Recognition Interface (Siri) that confirms the importance of the virtual all-in-one personalized assistant for consumers. ...
Article
Interactive product presentation techniques have recently gained importance in online marketing of house products. Real-time simulation offers a high level interactive presentation with more powerful features to present product functions and variations. This article presents a novel computational approach for engaging and supporting consumers in making informed choices about purchasing house products. It involved the development and testing of an online interactive real-time simulation for house product marketing (the Virtual House Showroom-VHS). This model also included optional humanoid avatars, representing a virtual buyer–seller experience. The research assessed the effectiveness of the VHS on consumers’ cognitive and affective responses. In particular, it examined whether consumers’ responses to real-time simulation with a humanoid avatar are significantly different from their responses to the model without the avatar. Participants (n = 110) were organized into two groups. Group 1 (n = 56) evaluated VHS without the humanoid avatar and group 2 (n = 54) appraised the model with the avatar. The results confirmed that interactive online real-time simulation technologies without the avatar have significant effects on consumers’ cognitive learning, affect, and behavior. However, it was found that the humanoid avatar had no significant effect on respondents’ decision-making. Further research was suggested to improve the usability of real-time simulation in the marketing of house products.
Chapter
Adaptive systems require technologies to enable high synchronicity between its users and their unfolding situation dynamics, in concert with system response actions. To be effective, a multi-dimensional view of context must be considered and incorporated. This work advances the development of such a system for RECON, an initiative to support intelligence analysts with a novel contextmanagement and case-based recommendation capability. The central concepts involved in the management of explicit and implicit contexts are presented and are developed into a novel multi-agent approach. In particular a new context-sensitive cognitive model and a community of expert service-oriented agents are proposed to facilitate and improve system adaptations to user-specific, situational, and system states. These designs pave the way towards future developments and experiments in improving human–machine interaction with adaptive context-management systems.
Conference Paper
This position paper presents the initial designs for a novel functional, context-aware, case-based recommendation system to enhance the current intelligence capability of military intelligence analysts. Its central objective is to support these analysts during the collection, processing, and analysis phases of the intelligence cycle through load minimization and improved human-machine synergy. This involves sense-making from both explicit and implicit contextual information with a nexus of technologies and processes, including software modelling and simulation, meta-level modelling and recommendation, and psycho-physiological modelling and monitoring. The proposed system, RECON, will contribute a new architecture based on five core components: i) brain-computer interfaces, ii) human-computer interaction, iii) data, iv) context, and v) case-based recommendation. Together with human analysts these form the basis of an adaptive system that will advance toward a powerful future intelligence analysis capability.
Article
Full-text available
In this paper we explore the role of 'narrative' as a means of capturing the schemata that are essential to sensemaking practice. 'Narrative' offers a convenient structuring of information that is germane to Intelligence Analysis. Developing this notion further, it is proposed that techniques for Narrative Visualization can support collaboration in sensemaking activities. The paper proposes recommendations for the use of Narrative Visualization (and related applications) in the practice of collaborative sensemaking.
Article
Full-text available
This paper presents research on developing a new type of tool that can alleviate several sys- temic problems faced by the traditional intelli- gence analysis process. The tool is a personal cognitive assistant that can rapidly acquire ex- pertise in intelligence analysis directly from in- telligence analysts, can train new analysts, and can help analysts find solutions to complex prob- lems through mixed-initiative reasoning, making possible the synergistic integration of a human's experience and creativity with an automated agent's knowledge and speed, and facilitating the collaboration with complementary experts and their agents.
Conference Paper
Full-text available
Software tools that make it easier for analysts to collaborate as a natural part of their work will lead to better analysis that is informed by more perspectives. We are interested to know if software tools can be designed that support collaboration even as they allow analysts to find documents and organize information (including evidence, schemas, and hypotheses). We have modified the Entity Workspace system, described previously, to test such designs. We have evaluated the resulting design in both a laboratory study and a study where it is situated with an analysis team. In both cases, effects on collaboration appear to be positive. Key aspects of the design include an evidence notebook optimized for organizing entities (rather than text characters), information structures that can be collapsed and expanded, visualization of evidence that emphasizes events and documents (rather than emphasizing the entity graph), and a notification system that finds entities of mutual interest to multiple analysts.
Article
Full-text available
The Future Operations Centre Analysis Laboratory (FOCAL) at Australia’s Defence Science and Technology Organisation (DSTO) is aimed at exploring new paradigms for situation awareness (SA) and command and control (C2) in military command centres, making use of new technologies developed for simulation, virtual reality, and real-time 3D animation. Recent work includes the development of a multimodal user interface to aid natural interaction with a virtual geospatial display using the combination of spoken natural language input and pointing devices. The end-user can query the environment, displayed objects, and their relationships using multimodal input and receive multimedia responses, optionally presented by a virtual adviser. The virtual geospatial display is a 3D representation of a geographical area of interest, rendered in real-time using imagery, maps, weather and terrain data, providing an augmented photo-realistic 3D model of the area. It allows the dynamic, near real-time display of objects within the environment, enhancing users’ situation awareness of the environment and of the relationships between the entities within it. FOCAL’s virtual advisers are real-time, photo-realistic embodied animated characters that converse with users using spoken language understanding, speech synthesis and facial gestures/cues. Pointing within the environment is performed with a variety of input devices that include the traditional mouse, 3D wands, gaze tracking and a novel hand gesture recognition system. FOCAL implements a heterogeneous multi-agent architecture based on the DARPA CoABS grid as a flexible, distributed model for interaction, information retrieval and processing, information fusion, and display. This paper provides an overview of FOCAL as a vision for future command centres, followed by a description of the latest developments of the multimodal dialogue system with virtual geospatial displays, illustrated in the context of a military training scenario.
Article
First description of the uncanny valley theory
Conference Paper
Narrative is recognised as a powerful tool in the business community for corporate knowledge management and effecting cultural and organisational change. It is effective because it allows complex and contextual relationships to be conveyed in a simple and compelling format. This paradigm is also employed ubiquitously by television news services, which use a combination of narrative and multimedia content to convey situational awareness of complex local and global events. In this paper we discuss how such `multimedia narrative' could be used to extend the usual conceptualisation of situational awareness, exemplified by the common operating picture which only supports the decision makers' understanding of what is happening, to an understanding of what the actors are doing and what impact this has on the decision makers' intent. Thus, multimedia narrative extends the common operating picture by providing a mechanism by which higher-level information fusion, namely situation assessment and impact assessment, could be presented to decision makers to enhance their situational awareness. In essence, the common operating picture presents the data, while multimedia narrative presents the story behind the data. The Defence Science and Technology Organisation is exploring emerging technologies for multimedia narrative that range from the abstract to the specific, such as embodied conversational characters, 3D animations, and virtual representations of the battlespace. The aim of this work is not only to identify the limitations of these technologies and determine when they are appropriate, but to also exploit these limitations.
  • E Naone
Naone, E.(2009), TR10:Intelligent Software Assistant, Technology Review, MIT, March/April 2009, http://www.technologyreview.com/read_article.aspx?c h=specialsections&sc=tr10&id=22117
Personalized Assistant that Learns video
DARPA (2011), Personalized Assistant that Learns video, http://www.youtube.com/watch?v=BF- KNFlOocQ&feature=youtu.be
Your Wish is its Command
  • Apple
Apple (2011), Your Wish is its Command, http://www.apple.com/iphone/features/siri.html