ArticlePDF Available


Intelligence analysis activities are increasingly seen as a sense-making activity. Information systems supporting these activities have, therefore, to be designed in a way to enable analysts to engage in sensemaking in an efficient manner. In the context of the VALCRI project, we developed guidelines for an appropriate design of intelligence analysis systems. Such guidelines can be very valuable in the design process, but their application is sometimes not straightforward. We describe such problems and also suggest possible solutions for these application problems within the context of intelligence analysis.
Exploring the Challenges of Implementing Guidelines for the
Design of Visual Analytics Systems
Johanna Haider1, Margit Pohl1, Eva-Catherine Hillemann2, Alexander Nussbaumer2, Simon Attfield3,
Peter Passmore3, B. L. William Wong3
1Vienna University of Technology, Austria,
2Technical University Graz, Austria,
3Middlesex University London, UK
Intelligence analysis activities are increasingly seen as a sense-making activity. Information systems sup -
porting these activities have, therefore, to be designed in a way to enable analysts to engage in sense-mak-
ing in an efficient manner. In the context of the VALCRI project, we developed guidelines for an appropri -
ate design of intelligence analysis systems. Such guidelines can be very valuable in the design process, but
their application is sometimes not straightforward. We describe such problems and also suggest possible
solutions for these application problems within the context of intelligence analysis.
In recent years, there has been an increasing amount of re-
search concerning the cognitive processes which occur when
users interact with visual analytics systems. There have been
studies on a more theoretical level, but also empirical investig-
ations related to individual aspects of visual analytics systems.
Both types of studies can be used to inform the design of
visual analytics systems. In this context, a systematic over-
view of the literature is necessary. Based on this literature re-
view, guidelines can be developed. Such guidelines can help
to translate results from research into recommendations for the
design of specific systems (Spence, 2011). They can help de-
signers to apply the experience made by others for the design
of the system they develop. However, this process of guideline
development is not a straightforward one. There are several
difficulties which might be encountered. A general problem in
the development of guidelines is that the research results are
not always clear cut. Also, the results of empirical studies
might seem contradictory. Furthermore, the value of
guidelines is in their application, and yet some guidelines are
very general and not easily adapted to specific cases. Some of
the issues discussed here were already addressed in the context
of user interface design and human-computer interaction
(HCI) (Preece, Rogers, Sharp, Benyon, Holland and Carey,
1994). Nevertheless, there are specific issues which are typical
for applications in visual analytics, for example, the problem
of trade-offs in guidelines. Despite the problems encountered
when using guidelines, guidelines are a very valuable tool for
designers which help to improve the design of interfaces. Also
a better understanding of the characteristics of the specific
design problem may emerge over time through iterative cycles
of analysis (e.g. user studies) and synthesis (e.g. prototype
design). Finally, the sources of individual guidelines them-
selves will inevitably fail to engage with the tricky problem of
managing design trade-offs. The goal of this paper is to de-
scribe the framework of guidelines developed in the context of
the VALCRI project, and then to discuss the problems en-
countered when developing guidelines for the design of visual
analytics systems, specifically criminal intelligence applica-
VALCRI is an EU-funded Framework Programme or FP7 pro-
ject that is researching and developing a visual analytics-based
system for sense-making in criminal intelligence analysis.
With 18 partners and a project budget of more than €13 mil-
lion, the project aims to create a system that will facilitate hu -
man reasoning and analytic discourse, that will be tightly
coupled with a semi-automated human-mediated semantic
knowledge extraction capability. The goal is to develop an in-
teractive visualisation system to support human inference
making and explanation formulation in a dynamic way where
intelligence analysts are able to fluidly transition and navigate
through various data sets searching for both known and un-
known associations. The semantic knowledge extraction en-
gine will be based on a combination of technologies, such as
self-evolving ontologies, natural language processing and data
mining techniques, for the extraction of text based on similar-
ity between different crime reports and other documents. One
of the key goals is that a police analyst will be able to instruct
the system to, “Given one crime, find me other reported
crimes that look similar”, and “Given a few crimes, tell me
what is similar between them, and what might be different
about them”. In addition, the research and development in
VALCRI will be guided by four key priorities:
1. Encourage Imagination. The most important failure
leading to the [9/11] attacks was one of imagination (National
Commission on Terrorist Attacks, 2004). The “failure of ima-
gination” is a “… mind-set that dismissed possibilities.” (Na-
tional Commission on Terrorist Attacks, 2004, p. 336)
2. Enable Insight. “Insights change our understanding by
shifting the central beliefs in the story we use to make
sense of events…our new understanding can give us new ideas
about the kinds of actions we can take; it can redirect our at-
tention, changing what we see; it can alter the emotions we
feel; and it can affect what we desire.” (Klein, 2014, p. 148)
3. Ensure Transparency. The system should support trans-
parency of the analytical reasoning process, especially in
terms of the legal and privacy matters. It should also support
transparency in terms of the computation of complex al-
gorithms, and how they arrive at a recommendation that is to
be presented in court, that may affect a person’s innocence.
4. Interact with Fluidity and Rigour. Fluidity and rigour
are two conflicting design requirements that characterise the
nature of intelligence analysis work. By fluidity we mean the
ease by which a system can be used to express the variability
of our thinking processes and by rigour we mean the ability of
the system processes and results to withstand interrogation
(Wong, 2013).
The User’s Work Domain: Criminal Intelligence Analysis.
Criminal intelligence and crime analysis are two similar, but
different and complementary fields. With advances in techno-
logy and changes in policing philosophy, these two fields are
expected to merge into an integrated approachcrime intelli-
gence – in the future (Ratcliffe, 2008), as the basis for intelli-
gence-led policing. Criminal intelligence refers to the activit-
ies used to support individual reactive investigations. “The
aim was … to gather evidence to support a criminal prosecu -
tion” and was often “… a case-specific tool of crime con -
trol” (Ratcliffe, 2008, p.8). Crime analysis has been described
more broadly in support of crime and disorder problems in
general, rather than focusing only on investigation support: the
“systematic study of crime and disorder problems and po-
lice-related issues – including socio-demographic, spatial, and
temporal factors to assist the police in criminal apprehen-
sion, crime and disorder reduction, crime prevention and eval-
uation” (Boba, 2005, p. 22). The intelligence and investigation
analysis process can be viewed as a continuum where the pur-
pose of the process changes from the support for thinking and
reasoning about anticipating and pre-empting crime, to sup-
port for thinking and reasoning about specific investigations.
Thinking and Reasoning Processes. While the reasons for
the analysis may change (e.g. from establishing crime patterns
to assembling evidence) and the kinds of analyses change (e.g.
from statistical analysis to constructing arguments), and differ-
ent rules may apply (e.g. regarding the access and holding of
data vs. evidence) analysts still need to think and reason creat-
ively and critically, and analytically and inferentially. Rather
than focusing on the transactions such as searching and re-
trieving, VALCRI aims to prioritise its design to support the
thinking and reasoning processes, i.e. how information can be
rapidly accessed and brought together to help the analyst
structure and explain the unknown – regardless of whether the
purpose is for intelligence or for investigative analysis.
Figure 1: How analysts think: Inference making is not uni-dir-
ectional, oscillating in unpredictable steps (Wong 2014).
The Human Issues Framework for the VALCRI system incor-
porates important general design recommendations, e.g. ac-
cording to gestalt theory, applicable to the work specific re-
quirements of crime intelligence analysts. Human Issues are
grouped in the following themes: evidential structuring and
reasoning, advances in sense-making and insight, cognitive
bias mitigation, and legal, ethical and privacy aspects. In this
paper, we concentrate on the first three concerning cognitive
issues. We look at guidelines from three different perspect-
ives. Evidential structuring and reasoning is based on models
of argumentation and narratives and describes how analysts
create coherent representations based on the evidence avail-
able. Sense-making in a very general sense plays an important
role and is mainly influenced by cognitive psychology. The
third view is concerned with cognitive biases. Such biases
play an important role in the analyst’s work. Strategies how to
mitigate cognitive biases are essential for the development of
plausible arguments. We assume that these three views on
cognitive strategies concerning the process of structuring and
presenting information in intelligence analysis cover the most
important aspects of this process.
Evidential structuring and reasoning
Making sense of a domain usually, if not invariably, involves
some kind of ordering or structuring of information, whether
this is captured in representations “in the head” (Klein, 2007)
or also distributed across external representations (Russell,
1993; Pirolli & Card, 2005). The development of VALCRI of-
fers the opportunity to analyse evidential structuring and reas-
oning in crime analysis and investigations, giving rise to a
general guideline that these processes be supported explicitly
in the system. The role of a set of guidelines is to help the de-
signer in distinguishing between good and bad design. How-
ever, an evolving understanding of the design problem is typ-
ical of any design activity and this can make it difficult at the
outset to identify guidelines which are both in scope and at a
level of granularity that usefully supports making real design
decisions. If guidelines are too “broad” they may maximise
the chance of relevance, but risk being bland. If they aim “nar-
row” they risk making assumptions about the design problem
and design direction which can mean a loss of relevance. Our
approach has been to engage in an iterative process of
guideline development alongside the interactive design pro-
cess. Accordingly, an initial set of guidelines was defined
based on review of theoretical models and research findings,
and scoped in terms of early stage articulations of the design
problem and the solution space. At the same time, sensitisation
to concepts in the literature allows user data and design ideas
to be parsed and guidelines refined where the literature sup-
ports it, or research activities undertaken where it does not.
Within the scope of evidential structuring and reasoning, an
initial review of the literature led to consideration of a three
kids of structuring relations: thematic sorting, argumentation
and narrative.
Thematic sorting. Thematic sorting involves classifying in-
formation into thematic groups. A theme is some grouping rel-
evant to a sense-making task, and themes may emerge from
interacting with data. Typically themes evolve in the early
stages of sense-making with corresponding information needs
and information relevance beset with uncertainty (e.g. Taylor,
1968; Belkin, Oddy & Brooks, 1982) which may reduce over
time as a task progresses (Kuhlthau, 1993). Thus thematic
sorting typically belongs on the lower left of Figure 1. Them-
atic sorting is acknowledged in a number of models of intelli-
gence analysis such as that of Pirolli and Card (2005), where it
is shown as an early stage in analysis. However we seek to go
beyond such models to understand how such themes may
come to exist and evolve over time, so that this may be
fostered within VALCRI.
Argumentation. VALCRI needs also to allow representa-
tion of arguments. Argumentation is a form of structuring that
relates propositions and ideas through operators that make in-
ferential reasoning explicit. An argumentation representation
references both the investigated domain and the logic of the
investigators reasoning. Argumentation typically belongs on
the right of Figure 1. Arguments have been represented visu-
ally for some time. Wigmore (1931) introduced a diagram -
matic system for representing arguments in legal cases in the
early twentieth century which allowed visual encoding of as-
pects of competing arguments. Toulmin (1958) developed a
more general diagrammatic convention for representing every-
day arguments. In his scheme there are three main parts, the
claim (the conclusion of an argument), the data (the evidential
support for the claim) and the warrant (the reasoning that con-
nects the evidence and claim). We would like to know what
representations and what aspects of argumentation are most
relevant in the criminal intelligence domain in order to provide
relevant guidelines for the representation of them. But whilst
argumentation may be important for exploring and demon-
strating support for conclusions, it may not account terribly
well for how people naturally do evidential reasoning; here
narrative would appear to be more significant.
Narrative. Narrative is a spoken or written account of con-
nected events organised temporally, and is often a significant
part of evidential reasoning. Narrative has been considered by
many to be central to sense-making. Bruner (2003) for ex -
ample has suggested that stories are what we use to make
sense of the world. Pennington and Hastie (1992) demon-
strated experimentally that people actively construct narratives
as explanations of evidence, by reasoning from evidence and
general understanding of the world. It would seem that genera-
tion and representation of narratives would play a central role
in criminal intelligence analysis. Bex, Prakken and Verhey
(2006) have argued for a hybrid model that incorporates argu-
mentation and narrative based on anchored narrative theory. In
this model stories are constructed that are anchored in facts
but supported by common sense, mixing causal stories with
evidential argument. We will seek to synthesise approaches to
narrative and argument in a domain specific way leading to
the development of more detailed guidelines on how eviden-
tial structuring and reasoning may be supported. Given the ap-
parent significance of thematic sorting for early stage sense-
making (in terms of the logical of a workflow at least), of the
representation of argument for grounding interpretation, and
of narrative as a natural way of accounting for evidence, these
have found their way into the initial set of guidelines for evid-
ential structuring and reasoning. Following Bex, Prakken and
Verhey (2006) we have incorporated argumentation and nar-
rative as anchored narratives. Based on the analysis of user
data, we can identify a need for allowing intelligence analysts
to represent multiple competing narratives. From this we can
evolve more specific questions such as: What kinds of argu-
mentation entities and relations should be represented? How
should uncertainty be considered? How should non-temporal
relations such as spatial or social network associations be in-
tegrated? These questions can then form the basis for further
research and an iterative refinement of the guidelines.
Sense-making and insight
Research into sense-making and insight in general is mainly
influenced by cognitive psychology. Klein’s (2013) sense-
making research brought forth the “Triple Pathway Model of
Insight” that claims to be able to represent real-world de-
cision-making. Here sense-making gets defined as the “delib-
erate effort to understand events” (Klein, 2013, p. 114). Hypo-
thesised connections in the data iteratively get elaborated,
questioned and reframed, if necessary. The theory behind this
model is that we have to shift an anchor in our prior know-
ledge to make sense of events and get new understandings.
This can sometimes even be a central belief that needs to be
changed to alter the direction of thinking and to get new ideas
about the kinds of actions we take. This is not an easy thing to
do, which is why openness of mind and willingness to ques-
tion knowledge is seen as productive. Klein’s model further
suggests that provoking questions leads to novel insights. Es-
pecially in the domain of criminal intelligence analysis it is
important to show up missing data and point the attention to
data that is unusual and calls for an explanation. The model
distinguishes between coincidences and curiosities. Curiosities
are sparked by a single event, while coincidences are observed
as a repetition of pattern. As a consequence this leads to a very
general guideline that visual analytics systems should emphas-
ise outliers or other unusual data in a way to make such pat-
terns visible to the analyst.
Connections. Insights occur when we make new connec-
tions in the vast amount of data. There are different ways in
which connections can be shown. On the one hand, simple so-
cial networks or other network-like visualisations make con-
nections easily visible. On the other hand, such visualisations
can quickly get too large for standard screen sizes. If the net -
work contains too many nodes and links, it is difficult to
identify connections between single entities. Allowing the user
to filter the data is a possibility to overcome that problem.
This also enables the user to look at the data from a different
point of view, which is a sensible guideline in that domain.
Another possibility to support showing up connections is us-
ing multiple views. Multiple view systems combine different
views, which make it possible to explore interrelations of data.
Contradictions. Klein (2013) points out that contradictory
evidence can be a powerful motivation for getting novel in-
sights. There are different ways contradictory evidence can be
made visible. Semi-automatic analysis methods for example
can reveal patterns in the data. Nevertheless, a human analyst
is necessary to assess the significance of these patterns. There-
fore, such patterns should be made visible in a representation,
e.g. by overlaying the information on a map, or using anima -
tion. This visual support assists analysts in spotting contradic-
tions and encourages asking questions at the same time.
Saved search queries. A more concrete example to support
sense-making and insight is to ease the task of searching.
When data needs to be observed it is necessary to search for
specific keywords over time. Users appreciate the feature of
saved search queries because they save time if they can reuse
done work (e.g. revisit a selection of keywords). But this also
brings new problems to the application as, e.g., the tendency
to confirm beliefs or hypotheses gets supported instead of en-
couraging openness and considering changes that might hap-
pen over time. Such concrete guidelines require a trade-off
when they conflict with other guidelines, see section below.
Cognitive bias mitigation
In everyday life, intelligence analysts are faced and challenged
with a vast amount of different data of which they are con-
stantly required to make sense of and consequently to make
appropriate decisions. To effectively manage the complexity
of information and to avoid to be overwhelmed by it, humans
unconsciously apply so-called heuristics (“rule of thumb”)
“strategies that ignore part of the information, with the goal of
making decisions more quickly, frugally, and/or accurately
than more complex methods” (Gigerenzer & Gassmaier, 2011,
p. 454). These heuristics, although generally useful, can lead
to “severe and systematic errors in judgment” (Kahneman &
Tversky, 1973), so-called cognitive biases. Such failures can
have serious consequences, especially in the criminal investig-
ative process. Thus, increased attention has been given to the
development of interventions to mitigate especially informa-
tion-processing errors (e.g. Fischhoff, 1982). One of the most
influential cognitive biases in the field of decision- and sense-
making is doubtless the confirmation bias. The confirmation
bias is the tendency of human beings to seek, interpret and
create information to support their initial expectation and
thoughts (Nickerson, 1998). The occurrence of cognitive bi-
ases and possible strategies to avoid or at least mitigate them
is discussed in a number of recent publications. Concerning
confirmation biases, the following protection strategies have
been proposed, such as awareness building (Wilson & Brekke,
1994), offering warning about the possibility of the bias (e.g.
Fischhoff, 1982), considering the opposite or generating al-
ternatives (Anderson & Sechler, 1986; Heuer, 1999; Hirt,
Kardes & Markman, 2004), and training. Unfortunately, such
mitigation strategies have oftentimes reached no satisfying
results. There is a need for additional as well as more effective
mitigation strategies, which help on the one hand the analyst
to overcome or at least minimise their biases. On the other
hand these strategies should be translated into guidelines that
inform and are applicable to the design of a visual analytics
platform such as VALCRI. With this in mind, a set of
strategies and guidelines has been developed to mitigate the
confirmation bias, some of which are outlined below.
Visualisation types. Relevant data can be visualised
through the use of different visualisation techniques. The user
interface should provide multiple options to visualise retrieved
information in different ways.
Levels of uncertainty. If the provided data is not complete
or if data is not confirmed, an indication should be provided to
make the user aware of that done by visualisations. It can be
used to keep disconfirming evidence, probabilities of decep-
tion, and amounts of supporting information in front of the
user to help to reduce the likelihood of this information being
lost in data volume.
Computerised critic questions. This strategy is also called
the devil's advocate method. The procedure has the advantage
that analysts have to be prepared to defend their reasoning.
Feedback. If analyst’s think or analyse into wrong direc-
tions and the system can detect such failures, then this feed-
back would help to rethink the current hypothesis.
Sometimes, trade-offs are necessary when applying
guidelines. One guideline that can be derived from the sense-
making theory of Klein (2013) is to support the identification
of connections. This can be facilitated by providing the user
with access to a large amount of related material to enable the
users to see connections in the material. However, that comes
with a trade-off because users in intelligence analysis usually
only have a limited amount of time to come to a conclusion
about some specific issue. If they are overwhelmed by a large
amount of information coming to a conclusion will become in-
creasingly difficult. There is apparently a trade-off between
supporting the users to identify unexpected but relevant con-
nections and overwhelming them with information. The ques-
tion here is how much information is enough to enable the
user to come to valuable insights in a reasonable amount of
time, but still providing him or her with enough information to
be able to see unexpected patterns in the data. This problem
can only be solved by a detailed analysis of the work environ-
ment of the user. Designers have to know the time frame in
which analysts have to come to a conclusion and the possible
importance of finding unexpected patterns. They then have to
weigh these two necessities to come to a viable solution. In the
context of cognitive biases and their mitigation, the applica -
tion of a guideline that is used to mitigate a specific cognitive
bias, can induce another cognitive bias. Taking as example the
guideline concerning providing the same information in differ-
ent visualisation formats. The idea behind is that presenting
the same data from different perspectives can mitigate the con-
firmation bias as it allows for building alternative hypothesis
on the same background information. However, when using
visualisations dependent on the visualisation format, a framing
effect can occur if the same information is visualised differ-
From HCI we know that the application of very general
guidelines sometimes poses serious problems. One of the 8
golden rules defined by Shneiderman (1992) is that systems
should provide informative feedback. This is certainly sensible
advice but difficult to achieve because it is not always clear
what is informative feedback. Because the guideline is quite
general it is sometimes difficult to apply it in a specific con-
text. It makes sense to break down such guidelines into more
concrete guidelines which specify more clearly how to achieve
the goal defined in a general guideline. There are similar prob-
lems concerning the design of visual analytics interfaces. The
guideline specifying that the identification of connections by
the users should be supported is certainly sound but also diffi -
cult to realise because this can be done in many different
ways. Such a general guideline does not provide system de-
signers with concrete advice how to develop systems for intel-
ligence analysis. There are many different methods that could
be used here, but it is sometimes not clear which methods fits
best to a certain context. Methods like multiple views com-
bined with linking and brushing or the usage of network visu-
alisations are certainly useful in this context. Again, it is ne-
cessary to interact with the prospective users to make sure that
these methods are adapted to the needs and requirements of
the users.
As a consequence of developing concrete guidelines, indi-
vidual guidelines might sometimes be contradictory. One
guideline might have consequences that prevent the realisation
of another guideline. One example from the set of VALCRI
guidelines concerns the issues of privacy versus providing the
user with as much related material as possible. Privacy prin-
ciples sometimes prevent access to needed information which
can help to identify connections between information and pat-
terns in the data. On the other hand, privacy is an important is-
sue, and sensitive data has to be treated with care. In this con-
text legal considerations have to be taken into account. Within
these legal constraints, access has to be provided to all the ne-
cessary information. Guidelines concerning the issue of filter-
ing data could be also problematic in terms of the occurrence
of confirmation bias. Such a pre-selection of data can be based
on initial thoughts, hypotheses, but also unconscious preju-
dices, which may lead to wrong decisions.
In this paper we give an overview of design guidelines for sys-
tems in criminal intelligence analysis. We also discuss some
problems that might arise in the course of the application of
such guidelines and how these problems might be solved in
specific settings. In this context a human-centred approach
that includes the users in the design process is important to
find a trade-off in the application of design guidelines.
The research leading to these results in the VALCRI project
has received funding from the European Union 7th Frame-
work Programme (FP7/2007-2013) under grant agreement no
FP7-IP- 608142, to Middlesex University and Partners.
Anderson, C. A., & Sechler, E. S. (1986). Effects of explanation and counter-
explanation on the development and use of social theory. Journal of
Personality and Social Psychology, 50, 24–24.
Belkin, N. J., Oddy, R. N., & Brooks, H. M. (1982). Ask for Information Re-
trieval part 1 background and theory. Journal of Documentation, 38(2),
Bex, F., Prakken, H., & Verhey, B. (2006). Anchored Narratives in Reasoning
about Evidence. In Proceedings of the 2006 Conference on Legal
Knowledge and Information Systems: JURIX 2006: The Nineteenth An-
nual Conference. IOS Press, 11–20.
Boba, R. (2005). Crime analysis and crime mapping. Los Angeles, London,
New Delhi: Sage, 2013.
Bruner, J. S. (2003). Making Stories: Law, Literature, Life. Harvard Univer-
sity Press.
Fischhoff, B. (1982). Debiasing. In D. Kahneman, P. Slovic, & A. Tversky
(Eds.), Judgement under uncertainty: Heuristics and biases, Cam-
bridge, England: Cambridge University Press, 422–444.
Gigerenzer, G., & Gaissmaier, W. (2011). Heuristic decision making. Annual
Review of Psychology, 62, 451–482.
Heuer, R. J. Jr. (1999). Psychology of intelligence analysis. Washington, DC:
Central Intelligence Agency Center for the Study of Intelligence.
Hirt, E. R., Kardes, F. R., & Markman, K. D. (2004). Activating a mental sim-
ulation mindset through generation of alternatives: Implications for
debiasing in related and unrelated domains. Journal of Experimental
Social Psychology, 40(3), 374–383.
Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psy-
chological Review, 80(4), 237–251.
Klein, G. (2013). Seeing what Others Don't: The Remarkable Ways We Gain
Insights. New York, USA: PublicAffairs.
Klein, G., Phillips, J. K., Rall, E. L., & Peluso, D. A. (2007). A data-frame
theory of sense-making. In Expertise out of context: Proceedings of the
sixth international conference on naturalistic decision making. Mah-
wah, NJ: Lawrence Erlbaum Associates, 15-17.
Kuhlthau, C. C. (1993). A principle of uncertainty for information seeking.
Journal of Documentation, 49(4), 339–355.
National Commission on Terrorist Attacks. (2004). The 9/11 Commission Re-
port. W.W. Norton and Company, Ltd., New York.
Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in
many guises. Review of General Psychology, 2(2), 175–220.
Pennington, N., & Hastie, R. (1992). Explaining the Evidence: Tests of the
Story Model for Juror Decision Making. Journal of Personality and
Social Psychology, 62(2), 189–206.
Pirolli, P., & Card, S. (2005). The Sense-making Process and Leverage Points
for Analyst Technology as Identified Through Cognitive Task Ana-
lysis. In Proc. of International Conf. on Intelligence Analysis, 5, 2–4.
Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., & Carey, T. (1994).
Human-computer interaction. Wokingham, England, Reading, Mass.,
Menlo Park, Cal.: Addison-Wesley.
Ratcliffe, J.H.(2008). Intelligence-led policing. Cullompton:WillanPublishing.
Russell, D. M., Stefik, M. J., Pirolli, P., & Card, S. K. (1993). The cost struc-
ture of sensemaking. In ACM Proceedings of the INTERACT'93 and
CHI'93 conference on Human factors in computing systems, 269–276.
Shneiderman, B. (1992). Designing the user interface: strategies for effective
human-computer interaction (Vol. 2). Reading, MA: Addison-Wesley.
Spence, B. (2011). The broker. In: Ebert, A., Dix, A., Gershon, N., Pohl, M.
(eds.) Human Aspects of Visualization, Springer, 10-22.
Taylor, R. S. (1968). Question-Negotiation and Information Seeking in Lib-
raries. College and Research Libraries 29(3), 178–194.
Toulmin, S. E. (1958). The Uses of Argument. Cambridge University Press.
Wilson, T. D., & Brekke, N. (1994). Mental contamination and mental correc-
tion: unwanted influences on judgments and evaluations. Psychological
Bulletin, 116, 117–142.
Wigmore, J. H. (1931). The Principles of Judicial Proof. Boston: Little,
Brown and Company.
Wong, B. L. W. (2013). Fluidity and Rigour - Designing Visual Analytics for
the Demands of Intelligence Analysis. NATO IST-116 Symposium on
Visual Analytics, Defence Academy of the United Kingdom, Shriven-
ham, UK.
Wong, B. L. (2014). How Analysts Think (?): Early Observations. In IEEE
Joint Intelligence and Security Informatics Conference (JISIC), 296–
... The main risks identified by the IEB were: Diverse social, professional and legal mores, privacy and data protection risks, risks of poor communication, data and reasoning provenance and the transparency of the entire analysis process for addressing significant ethical concerns. Issues related to cognitive bias were noted early in the project and mitigation measures in the form of design guidelines produced as part of the Human Issues Framework in WP3 (Haider et al. 2015). ...
Full-text available
In developing a semi-automated decision support system using cutting-edge visual technologies to aid police intelligence analysts (the VALCRI project) it was recognised that addressing ethical, privacy and legal issues would need to be considered from the start. From the beginning, experts in these fields were embedded in the project and externally an independent ethics board was established and a number of ethical concerns were identified. Addressing the concerns presented some challenges both in terms of process and product and are the subject of this paper. Insights about these problems can contribute to other research projects beyond the area of crime visualization, for instance addressing concerns such as logging processes for auditing and evidence in other sensitive projects.
... We formulated a comprehensive research question RQ1 based on the Triple Path Model of Insight (Klein 2013) and previous work on sense-making in intelligence analysis (Haider et al. 2015), and research questions RQ2-RQ5 on the basis of the requirements in intelligence analysis. ...
Full-text available
Visualising how social networks evolve is important in intelligence analysis in order to detect and monitor issues, such as emerging crime patterns or rapidly growing groups of offenders. It remains an open research question how this type of information should be presented for visual exploration. To get a sense of how users work with different types of visualisations, we evaluate a matrix and a node-link diagram in a controlled thinking aloud study. We describe the sense-making strategies that users adopted during explorative and realistic tasks. Thereby, we focus on the user behaviour in switching between the two visualisations and propose a set of nine strategies. Based on a qualitative and quantitative content analysis we show which visualisation supports which strategy better. We find that the two visualisations clearly support intelligence tasks and that for some tasks the combined use is more advantageous than the use of an individual visualisation.
... Using previous work [12], we propose two possible designs that in conjunction address some of the issues encapsulated in the 3 requirements: a node-link based multigraph (graph) and a specialised adjacency matrix (matrix). The designs are based on 3 main ideas: First, to lay out the terrain of the criminal sub-system by making visible the constraints and boundaries. ...
The line-up task hides a plot of real data amongst a line-up of decoys built around some plausible null hypothesis. It has been proposed as a mechanism for lending greater reliability and confidence to statistical inferences made from data graphics. The proposition is a seductive one, but whether or not line-ups guarantee consistent interpretation of statistical structure is an open question, especially when applied to representations of geo-spatial data. We build on empirical work around the extent to which statistical structure can be reliably judged in map line-ups, paying particular attention to the strategies employed when making line-up judgements. We conducted in-depth experiments with 19 graduate students equipped with a moderate background in geovisualization. The experiments consisted of a series of map line-up tasks with two map designs: choropleth maps and a centroid-dot alternative. We chose challenging tasks in the hope of exposing participants’ sensemaking activities. Through structured qualitative analysis of think-aloud protocols, we identify six sensemaking strategies and evaluate their effects in making judgements from map line-ups. We find five sensemaking strategies applicable to most visualization types, but one that seems particular to map line-up designs. We could not identify one single successful strategy, but users adopt a mix of different strategies, depending on the circumstances. We also found that choropleth maps were easier to use than centroid-dot maps.
Full-text available
Confirmation bias, as the term is typically used in the psychological literature, connotes the seeking or interpreting of evidence in ways that are partial to existing beliefs, expectations, or a hypothesis in hand. The author reviews evidence of such a bias in a variety of guises and gives examples of its operation in several practical contexts. Possible explanations are considered, and the question of its utility or disutility is discussed.
Full-text available
We report the results of a British Library Research and Development Department-funded design study for an interactive information retrieval system which will determine structural representations of the anomalous states of knowledge (ASKs) underlying information needs, and attempt to resolve the anomalies through a variety of retrieval strategies performed on a database of documents represented in compatible structural formats. Part I discusses the background to the project and the theory underlying it, Part II (next issue) presents our methods, results and conclusions. Basic premises of the project were: that information needs are not in principle precisely specifiable; that it is possible to elicit problem statements from information system users from which representations of the ASK underlying the need can be derived; that there are classes of ASKs; and, that all elements of information retrieval systems ought to be based on the user's ASK. We have developed a relatively freeform interview technique for eliciting problem statements, and a statistical word co-occurrence analysis for deriving network representations of the problem statements and abstracts. Structural characteristics of the representations have been used to determine classes of ASKs, and both ASK and information structures have been evaluated by, respectively, users and authors. Some results are: that interviewing appears to be a satisfactory technique for eliciting problem statements from which ASKs can be determined; that the statistical analysis produces structures which are generally appropriate both for documents and problem statements; that ASKs thus represented can be usefully classified according to their structural characteristics; and, that of thirty-five subjects, only two had ASKs for which traditional ‘best match’ retrieval would be intuitively appropriate. The results of the design study indicate that at least some of our premises are reasonable, and that an ASK-based information retrieval system is at least feasible.
Full-text available
Social theories—beliefs about relations between variables in the social environment—are often used in making judgments, predictions, or decisions. Three experiments, with 146 undergraduates, examined the role of explanation in the development and use of social theories. It was found that explaining how or why 2 variables might be related led to an increased belief in and use of the explained theory. A counterexplanation task was effective in eliminating this initial explanation bias. These explanation and counterexplanation effects occurred with simple belief measures and with complex social judgments involving multiple predictor variables. New, explanation-induced beliefs did not lead to biased evaluation of new data. However, exposure to new data indicating a zero relation between the social variables in question moderated but did not eliminate the explanation-induced theories. Implications for decision making in real-world contexts and for understanding the cognitive underlying explanation effects in the present and in related judgment domains were also examined. (26 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Full-text available
Investigates the Story Model, N. Pennington and R. Hastie's (1986, 1988) explanation-based theory of decision making for juror decisions. In Exp 1, varying the ease with which stories could be constructed affected verdict judgments and the impact of credibility evidence. Memory for evidence in all conditions was equivalent, implying that the story structure was a mediator of decisions and of the impact of credibility evidence. In Exps 2 and 3, Ss evaluated the evidence in 3 ways. When Ss made a global judgment at the end of the case, their judgment processes followed the prescriptions of the Story Model, not of Bayesian or linear updating models. When Ss made item-by-item judgments after each evidence block, linear anchor and adjust models described their judgments. In conditions in which story construction strategies were more likely to be used, story completeness had greater effects on decisions. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Full-text available
Encouraging people to consider multiple alternatives appears to be a useful debiasing technique for reducing many biases (explanation, hindsight, and overconfidence), if the generation of alternatives is experienced as easy. The present research tests whether these alternative generation procedures induce a mental simulation mind-set (cf. Galinsky & Moskowitz, 2000), such that debiasing in one domain transfers to debias judgments in unrelated domains. The results indeed demonstrated that easy alternative generation tasks not only debiased judgments in the same domain but also generalized to debias judgments in unrelated domains, provided that participants were low in the need for structure. The alternative generation tasks (even when they were easy to perform) showed no evidence of activating a mental simulation mind-set in individuals high in need for structure, as these individuals displayed no transfer effects. Implications of the results for understanding the role of the need for structure, ease of generation, and mental simulation mind-set activation for debiasing are discussed.
In this paper we describe work-in-progress to develop a description of the ways by which intelligence analysts engage in the thinking and reasoning processes when engaged in the intelligence analysis task. Such a model will be used to inform the design the interactive visual interfaces for a next generation intelligence analysis system. We introduce the concepts of fluidity and rigour as key characteristics of the analysts' thinking landscape.
Seekers of information in libraries either go through a librarian intermediary or they help themselves. When they go through librarians they must develop their questions through four levels of need, referred to here as the visceral, conscious, formalized, and compromised needs. In his pre-search interview with an information-seeker the reference librarian attempts to help him arrive at an understanding of his "compromised" need by determining: (1) the subject of his interest; (2) his motivation; (3) his personal characteristics; (4) the relationship of the inquiry to file organization; and (5) anticipated answers. The author contends that research is needed into the techniques of conducting this negotiation between the user and the reference librarian.
This paper proposes an uncertainty principle for information seeking. The principle is based on the results of a series of studies conducted by the author into the user's perspective of the information search process. A basic principle of uncertainty is elaborated by six corollaries. The principle is proposed to explain the constructive process of information seeking and use bringing affective considerations to what has usually been regarded as a cognitive process.
Conference Paper
The task of a Broker is to interpret relevant knowledge acquired by cognitive and perceptual psychologists and bring it suitably to the notice of interaction designers, thereby avoiding the need for that designer to have knowledge of cognition and perception. The task is first illustrated by an example based on the concept of Design Actions and demonstrates the implication, for two different design challenges, of certain properties of the human visual processing system. It is then argued that the task of the Broker can be eased by the definition and classification of relevant concepts, in the illustrative example those of browsing, interaction and visualization. Finally, a current need for a Broker’s expertise is illustrated in the context of the interactive and dynamic exploration of the relationships associated with a multivariable system.