Characterizing Tool Use in an Interactive Drawing Environment

Abstract

The metaphor of tool use for describing the interaction between a human and a computer is pervasive in user interface design. The basic concept of tool use, however, is difficult to define precisely, for HCI purposes or in general. In this paper we argue that a close examination of physical tool use can improve the design of interactive software. We describe a drawing application, HabilisDraw, that incorporates some of the properties we associate with physical tools but are not commonly found in software: persistent tool objects that encapsulate behavior and information, that can be used in conjunction with one another, and that embody rich cues about their appropriate usage. Initial results from task analysis and formative evaluation suggest that the approach has some promise.

Full text

Page 1

Characterizing tool use
in an interactive drawing environment
Robert St. Amant and Thomas E. Horton
Department of Computer Science
North Carolina State University
EGRC-CSC Box 7534
Raleigh, NC 27695-7534
stamant@csc.ncsu.edu, tehorton@eos.ncsu.edu
ABSTRACT
The metaphor of tool use for describing the interaction between a
human and a computer is pervasive in user interface design. The
basic concept of tool use, however, is difficult to define precisely,
for HCI purposes or in general. In this paper we argue that a close
examination of physical tool use can improve the design of inter-
active software. We describe a drawing application, HabilisDraw,
that incorporates some of the properties we associate with physi-
cal tools but are not commonly found in software: persistent tool
objects that encapsulate behavior and information, that can be used
in conjunction with one another, and that embody rich cues about
their appropriate usage. Initial results from formative evaluation
suggest that the approach has some promise.
Categories and Subject Descriptors
H.5.2[InformationInterfacesandPresentation]: UserInterfaces;
I.2.0 [Artificial Intelligence]: General—Cognitive simulation
General Terms
Human Factors
Keywords
Metaphors, tool use, drawing, interface design
1.INTRODUCTION
The metaphor of tool use for describing the interaction between
a human and a computer is pervasive in user interface design. It is
surprisingly difficult, however, to define the concept of tool use pre-
cisely, especially in cognitive terms. Examples of tools in software
vary widely. At one extreme, a tool can be as simple as an icon in a
drawingapplicationthatsetsaspecificmode(e.g., “Holdthemouse
downontheellipsetool...”) Attheotherextremewefindhundreds
of systems in the HCI literature characterized as “tools” that have
little more in common than being interactive software. The goal
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to distribute to lists, requires prior specific
permission and/or fee.
Int. Symp. on Smart Graphics, June 11-13, 2002, Hawthorne, NY, USA.
Copyright 2002 ACM 1-58113-555-6/02/0600..$5.00.
of developing a detailed definition of tool use is not simply for lin-
guistic reasons, or to separate interactive software arbitrarily into
tools and non-tools; rather, we believe that a better understanding
of the cognitive characteristics tool-using behavior will lead to the
design of more effective software.
Thetoolmetaphor, evenlooselyapplied, hasbeenextremelysuc-
cessful in HCI, and has influenced the design of interactive sys-
tems in many ways. The tool metaphor has driven the develop-
ment of cognitive artifacts, which act analogously to simple tools
in the physical world, to amplify capabilities (e.g., written lists as
memory extensions), or to translate problems into more manage-
able form [10, 17]. Software environments have also taken lessons
from environments for physical tools—for example, a blacksmith’s
shop [12], a carpenter’s workbench [8], or a chef’s kitchen [1]—
by incorporating some of the same design properties: effective use
of space [13], conceptual organization that reflects functional rela-
tionships, andexploitationofconventions[19]. Itiseasytoseehow
spatial and functional organization of tools in a physical environ-
ment, such as the carpenter’s workbench, translates to comparable
structure in a software environment, in the arrangement of menu
options, floating palettes, icons on toolbars, and multiple window
placement. It is also easy to see at least some parallels between
cognitive and physical tools.
As user interface technology has matured, however, the distance
between the source of the metaphor (tools) and its target (inter-
active software) has widened. More abstract, cognitively oriented
tools make different demands on our tool-using software abilities
thanphysicaltools. Forexample, anexpertdraftspersonmayhavea
great deal of facility with drawing implements, straightedges, com-
passes, and related tools; how much of this expertise translates to
the common activities in a drawing application of clicking an ob-
ject type and then outlining its bounding box, or dragging objects
into different locations and orientations, or raising a dialog box to
edit object properties? The benefits of the more abstract software
interface (e.g., predictability, consistency, and generalizability) are
well understood. One important tradeoff, less often considered, is
that other elements, such as the directness of action, physical affor-
dances, and the transfer of motor skills, may be lost.
We are developing a drawing application, called HabilisDraw, to
help us explore the possible benefits and drawbacks of a more de-
tailed correspondence between physical tool use and software tool
use [25]. HabilisDraw includes persistent tools that encapsulate
behavior and information, that can be used in conjunction with one
another, and that embody richer cues about their appropriate us-
age than is common in other interfaces. Tools in HabilisDraw have
some similarities to past work, such as “local tools” in the KidPad
86

Page 2

drawing environment [3], but our research examines the conceptual
foundations of tools and their use in closer detail.
In this paper we present a preliminary, general approach to in-
terpreting tool use in interactive software. The first section below
describes tool use in the physical world. In the sections that follow
we discuss the implications for software tools, looking in particu-
lar at influences on the HabilisDraw system. The paper ends with a
brief description of our early evaluation efforts.
2.PHYSICAL TOOL USE
To understand how people might apply their tool-using skills in
a software environment, it would be helpful to have a model of the
use of physical tools. Unfortunately, human tool use has received
limited attention in artificial intelligence, human-computer inter-
action, cognitive modeling, and related fields, and we are aware
of no formal representations that we can exploit as a foundation
for improving software environments. We must therefore develop
our own foundation. This section organizes some of the basic con-
cepts of tool use; our account is based on research in anthropol-
ogy [12], animal cognition [11, 23], experimental psychology [28],
cognitive psychology [10], social psychology [22], situated cogni-
tion [1], and design disciplines [16, 19]. Figure 1 summarizes the
discussion points we will make in this section.
We first observe that physical tools are persistent artifacts. This
point has been made most strongly by Beck in research on non-
human primate cognition [2]: “Thus tool use is the external em-
ployment of an unattached environmental object to alter more ef-
ficiently the form, position or condition of another object, another
organism, or the user itself when the user holds or carries the tool
during or just prior to use and is responsible for the proper and ef-
fective orientation of the tool.” In software environments, “tools”
are often global modes (as in many drawing applications) or tran-
sientdialogs(asinmanywordprocessingapplications.) Inphysical
environments, tools are persistent, structured objects with special-
ized behaviors and properties that can be manipulated. One can
adjust a wrench or a T-square, for example, to a particular appli-
cation; one chooses a particular type of hammer or saw for a job.
The persistence of physical tools means that they can be reused [1],
they can encapsulate information as well as behavior, and they can
be combined with one another or interact for their effect.
In addition to characterizing tools as objects, we can make gen-
eral observations about procedural structure in their use. What dis-
tinguishes tool-using behavior from other types of activity? The
most colorful examples of procedural structure come from research
in non-human primate cognition, where accounting for behavior in
tool-using terms is an important issue.
• Tool use involves direct action [29]. A striking action with a
stone, with the goal of cracking open a nut, is an example of
tool use. In contrast, some primates show uncanny accuracy
in dropping objects onto researchers [11]; this is considered
tool-related behavior rather than actual tool use.
• Tool use often amplifies existing behavior [10]. Using a stick
to extend one’s reach (e.g., through a narrow opening, or to
touch a moderately distant object) is a common aspect of tool
use in experimental settings and in the wild [18].
• Tool use is goal-directed activity [11]. Sometimes desirable
ends are achieved through the incidental or even accidental
use of an object, which is not considered a tool in that case.
Inferred intention is an important part of judging whether an
activity constitutes tool use.
• Tool use involves effective behavior. One influential study
involved monkeys given the task of pushing a reward out of
a narrow, transparent tube; one monkey unwrapped a thick
bundle of reeds held together with masking tape, but then
tried to push with the tape instead of a reed [30].
In many domains, tools are not used in isolation; rather, we find
tool-using environments, such as a carpenter’s or mechanic’s work-
shop, which suggest that sets of tools can be organized by their in-
tended use. We can also identify abstract classes that give broad
coverage of tools in general:
• Tools that produce a persistent effect on materials or the en-
vironment. We call these effective tools. Examples in the
physical world include hammers, saws, screwdrivers, and so
forth. Although effective tools are those that first come to
mind when we think of tools, this category does not encom-
pass all types.
• Tools that provide information about materials or the envi-
ronment. These have been called instruments in the tool use
literature [10]. Instrumentation may be built into an effective
tool, aswhenatablesawindicatestheangleatwhichitiscut-
ting a board. Tools that act alone as instruments include mea-
suring tapes, calipers, microscopes and magnifying glasses,
and so forth.
• Tools that constrain or stabilize materials or the environ-
ment for the further application of effective tools. We call
these constraining tools. Examples include clamps, rulers,
and other devices that limit movement or flexibility. (No-
tice that screws and nails are not reusable and are commonly
thought as materials rather than tools, in that they remain in
the finished product.) We often find artifacts that are simul-
taneously constraining and effective tools. For example, a
handsaw is effective, in that the blade makes a cut in a piece
of wood, but it is also constraining, in that the breadth of the
blade forces (or at least facilitates) a straight-line cut. That
is, because the breadth of the blade must follow the toothed
edge through the groove as the wood is cut, it is easier to cut
in a straight line than not. We notice that jigsaws and keyhole
saws have a very narrow blade just to relax this constraint.
• Tools that demarcate the environment or materials. The goal
of demarcation is to distinguish similar areas or pieces of the
environment so that they can be treated differently. Exam-
ples include the carpenter’s pencil, pushpins, and working
surfaces inscribed with fixed markings.
This taxonomy has the property that all of its categories describe
relationships between a tool and the environment in which it is
used. This point is worth making explicit: the applicability of a
tool is determined by its ecological properties. For example, a car
mechanic who lacks a hammer can use any sufficiently solid, heavy
object that comes to hand. The ecological nature of tool use leads
to other significant properties:
• Toolusecanbeopportunistic. Toolscanbeusedforpurposes
not intended by their designers, as above.
• Tools provide rich cues about their appropriate use. The af-
fordances of a tool become obvious in its use; the hammer is
almost a canonical example.
• Tool use involves establishing and exploiting constraints be-
tween the user and the tool, the user and the environment,
87

Page 3

Tools are persistent artifacts:
Tools are replicatable and reusable; tools encapsulate information as well as behavior;
tools can combine or interact with one another for effect.
Tool use exhibits significant procedural structure:
Direct rather than indirect action; amplification; goal-directed activity; effective behavior.
Tools fall into a natural taxonomy:
Effective tools produce a persistent effect on materials; instruments gather information;
constraining tools constrain or stabilize materials or the environment; demarcating tools spatially structure the environment.
Tool applicability is determined by ecological properties:
Tool use can be opportunistic; tools provide rich cues about their appropriate use;
effective use of space is important; tool use exploits constraints between the user, the tool, and the environment.
Figure 1: A summary of the properties of tools and tool use
and the tool and the environment [28]. The issue of interact-
ing constraints becomes immediately obvious in the exam-
ple of hammer use, if one is working in a tightly enclosed
space, from an awkward position, with a sprained wrist, etc.
From a more positive perspective, establishing appropriate
constraints will enhance tool use, e.g., bracing oneself for a
stronger hammer blow, or clamping materials for appropri-
ate resistance. An important specialization of this property
is that tool use is associated with effective use of space [13,
29]. Many experienced tool users lay out their tools before
beginning a task, on the assumption that some common tools
are almost always eventually needed and should be ready to
hand.
These observations suggest that tool use is a rich area for explo-
ration and research from a variety of viewpoints. Two areas that
we have not touched on include the role of tools as mechanisms for
communication in social interaction, and the visual/motor execu-
tion aspects of tool use. The use of tools for communication raises
a new set of issues: for example, recognizing tool use sometimes
depends on culture or convention. The determination whether an
artifact is a tool or not can be difficult; a fist-sized piece of flint
may be stone tool to one archaeologist, for example, but only a
leftover artifact from the production of flint arrowheads to another
archaeologist [19]. At the level of visual and motor activities, it is
possible to see tool use as a form of visually guided activity [4].
This viewpoint provides some significant insights into tool use that
we have only begun to explore [21, 24].
We are aware of no interactive software systems that incorpo-
rate a significant portion of the concepts outlined above. In our
current research on the HabilisDraw system, described in the next
section, we examine the potential benefits of incorporating some of
the properties of physical tools into a drawing application.
In taking this direction an important open issue concerns the pos-
sible tradeoffs. We run several dangers. Software tools that more
closely resemble physical tools may be less efficient, in a task anal-
ysis sense. They may be more difficult to use, by being overly
constrained due to irrelevant physical considerations. They may
be more difficult to learn in the first place; for example, in a con-
ventional drawing application, many different types of objects are
created by selecting a mode and drawing a bounding box. This
kind of consistency is missing from drawing tools in physical en-
vironments; their uses must be learned separately. It might even
be argued that we are inappropriately conflating different types of
tools: physical tools are associated with amplification, while cog-
nitive tools are concerned with problem transformation [10]. Fi-
nally, we face a practical limitation in working with conventional
input devices, the keyboard and mouse—some types of physical af-
fordances simply will not transfer from richer, physical tool-using
environments.
These and related issues have been addressed to some extent by
others. Gentner and Nielsen argue that richer cues can improve
learnability and usability, offsetting the loss of consistency in the
use of different tools [7]. Constraints are often viewed as opportu-
nities for learning [6]. Other questions remain open, but in the end
these can best be answered by the description and analysis of the
design of specific systems.
3. HABILISDRAW
HabilisDraw1is a prototype drawing application for exploring
the use of software-based tools in an intereactive system. In typi-
cal drawing applications, tool use generally refers to the selection
and application of one of several independent modes—the cursor
changes shape and a mouse click takes on a new function. The
differences between drawing a picture on a computer and drawing
with pen and paper are striking; in the real world, tool use is a far
richer and more complex process, one that provides for multiple
degrees of freedom in the interaction, with users creatively apply-
ing cognitive, visual, and motor skills to solving problems. If a
mode-based model limits a user to a subset of his or her abilities as
a tool user, then the integration of real world styles of tool use into
the interaction might plausibly result in a more powerful and more
intuitive interface. Such an interface might also lead designers to a
greater understanding of intelligent tool-using behavior.
Figure 2 shows a diagram of the familiar “shoot the monkey”
physics demonstration, created using the current version of Habil-
isDraw. The tree and the monkey’s body and trajectory were drawn
using a pen tool by itself. The circles forming the monkey’s head
and eye, and the arc’s forming his arms, ear, and tail were all cre-
ated using a pen tool in combination with a compass, as was the
arc showing the trajectory of the arrow. The arrow itself, and the
hunter’s cap, head, and body were created using a pen and a pen
combined with the compass. The angles forming the hunter’s arms
and legs were drawn using a pen tool with two rulers set-up as jigs.
Finally, the bow was drawn using a pen tool, two rulers and a com-
pass, all in combination. (The point of shooting the monkey, by the
way, is to show that if the monkey lets go of the branch at the same
instant the hunter fires directly at the the monkey, then the arrow—
which, like the monkey, is accelerated by the Earth’s gravity—will
1The name is based on that of Homo habilis (“handy man”), the
first hominid species known to have made and used stone tools.
88

Page 4

Figure 2: A drawing generated in HabilisDraw
strike the monkey as it falls; the actual trajectory is approximated
here as a circular arc.)
While most of the user’s interaction occurs within the boundaries
of the drawing canvas, several functions are accessed by menu se-
lections.
View→Show Tools toggles the display of tools on the page. Hid-
ing the tools allows the user to see how the drawing appears
without any additional clutter and without having to put all
the tools away.
View→Show Lens toggles the visibility of the magnifying lens.
Since the lens is only useful for detailed work, most of the
time it is kept out of the user’s way.
Window→Show Detail toggles the detail window. When the lens
is displayed, the detail window shows the contents of the
lens, but with added scalability. By increasing the size of
the Detail Window, the user can zoom-in on the page further
than with the lens alone.
Window→ShowPropertiesbringsupthepropertieswindow. When
a drawn object or a tool is selected, its properties (length, line
width, angle, color, etc.) are displayed in the properties win-
dow and may be directly edited by the user.
Window→NewPalette, inthecurrentversionofHabilisDraw, does
nothing more than bring up a new empty window. In future
versions, however, each palette window might be capable of
holding a customized set of tools for quick access. Often
used sets (e.g., favorite ink well colors) could be saved for
use with later documents.
In addition, a command line allows for scripting of the inter-
face and for the generation and execution of macros. Our original
intention was to provide for programmatic access to HabilisDraw
functionality, but macro facility also suggests other directions, such
as the approach taken by Eager in programming by example [5], or
the possibility of end-user programming. Both of these possibilities
are discussed further in Section 5.
A few general rules are followed throughout the HabilisDraw
interface. To move or adjust any tool or drawn object on the page,
the user right-clicks on the object and drags (drawn objects may
also be moved with a left-click.) To activate a tool’s function, the
user left-clicks and drags. Clicking on a tool or drawn object will
select it (currently, only one object may be selected at a time.) A
selected object is displayed with the corners of a bounding box
drawn around its perimeter. The properties of the currently selected
object may be edited in the properties window.
3.1Tools in HabilisDraw
HabilisDraw implements six types of tools so far: rulers, com-
passes, pens, ink wells, pushpins, and lenses, as shown in Figure 3.
We do not claim that all these individual tools are unique (e.g.,
some CAD systems for architecture offer comparable functional-
ity, if we take a task analysis view [15], and KidPad has obvious
similarities [3].) The novelty of this work lies instead in our focus
on tools as first-class artifacts and our explication of their concep-
tual foundations.
3.1.1
The most basic tool in HabilisDraw is the pen tool. Currently,
this is the only tool capable of marking the page. The color of the
pen tip indicates the color of the line that the pen will draw. The
number on the body of the pen is the width (in pixels) of the line
that will be drawn.
When inactive, a pen tool lies horizontally on the page. Right
clicking on an inactive pen allows the user to drag the pen around
the page without affecting any other tools or objects. Left click-
ing over an inactive pen will activate (“pick up”) the pen. While
multiple pens may be on the page, only one pen may be active at
a given time. When active, the pen tool tracks the cursor position.
A left click while a pen is active and not over another tool initiates
free-hand drawing at the pen tip. The pen continues to mark the
page until the user releases the left button. A left click while the tip
is over an ink well will cause the pen to adopt the color of the ink
in that well. Right clicking while the pen is not over another tool
will “drop” the pen back to the page and it returns to an inactive
state. To minimize the time users must spend switching tools, right
clicking over another tool while a pen is still active, allows the user
to move the other tool around the page, just as though the pen tool
wasn’t there.
A pen tool has five properties that may be set through the prop-
Pens
89

Page 5

PenPushpinInk wellsLens
Drawing with a pen
Dynamic alignment with ruler Pen and compass
Pen and ruler interaction
Pen, compass, and ruler interactionPen, compass picture
Figure 3: Tools in HabilisDraw
erties window. Changing the width property sets the width of the
lines drawn with the pen. Changing the R, G, and B properties sets
the red, green, and blue components of the pen’s drawing color re-
spectively. Changing the A property sets the alpha value for lines
draw with the pen.
Some classes of tools, including the ruler and compass tools, as
described below, are designed to impose constraints upon the pen
tool. To use a ruler or compass in combination with a pen, the user
first picks up the pen by left clicking, then moves the pen to the
helper tool. When the pen’s tip is properly positioned, the helper
tool’s surface will be highlighted. Left clicking and dragging will
cause the pen to draw a line constrained by the helper tool.
3.1.2
Ink wells are some of the simplest tools in HabilisDraw. The
only property of an ink well is the color of the “ink” it contains.
This color is displayed on a label in the middle of the ink well.
Usually, the ink well is a passive tool. Its primary function is to
store a color for later use. Right clicking on the ink well and drag-
ging allows a user to position the ink well on the page. Left clicking
while dragging an ink well over a pen, another well, or a drawn ob-
ject causes the well to “tip over” and assign its color to the object.
Placing an active pen over an ink well and left clicking also trans-
fers the color of the ink from the well to the pen. By left clicking
on an ink well and dragging, a user can pull out a “siphon.” If the
user drags the end of the siphon to any point on the page and right
clicks, the ink well will adopt the color immediately under the end
of the siphon, be it the color of the background, a drawn object, or
another tool.
The RGB and alpha components of an ink well may also be set
directly through the properties window. In addition, by selecting
the replace option, changing the color of the ink well will replace
the color of all pens and drawn objects on the page currently col-
ored with the ink well’s old color with the ink well’s new color.
Ink Wells
3.1.3
While it seems that pushpins may hold great potential, they are
perhapsunder-usedinthecurrentversionofHabilisDraw. Pushpins
may be “stuck” anywhere in the page. If placed over a drawn ob-
ject, the object is “pinned” to the page and cannot be repositioned
until the pin is removed. A pin may also be used to constrain the
movement of other tools, acting as a stop and blocking the drag of
a ruler or the rotation of a compass.
Pushpins
3.1.4
Both compasses and rulers may be used to constrain the motion
of a pen tool as it draws. A compass tool constrains a pen’s mo-
tion such that it follows a circular path around the center of the
compass’s base. The compass tool consists of a circular base and
a circular activation site connected by a narrow arm. A right click
and drag on the base allows a user to position the compass on the
page. A right click and drag on the activation site allows a user to
set the angle and distance between the base and the activation site.
When an active pen tool is brought into range of the activation site,
the site is highlighted. When in range, a left click starts the pen tool
drawing in the center of the activation site. Dragging moves the pen
and the site around the base, drawing a circular path with a radius
equal to the distance between the centers of the base and activation
site. Releasing the left mouse button stops drawing. If the button
is released before the circle was closed, the resulting drawn object
is only part of a circle—thus it is very easy to create arcs (often a
complex task in conventional drawing programs.)
When moving the activation site, whether drawing or not, a label
Compasses
90