GOAL PROCESSING IN AUTONOMOUS AGENTS
A thesis submitted to the
Faculty of Science
University of Birmingham
for the degree of
DOCTOR OF PHILOSOPHY
School of Computer Science
University of Birmingham
Birmingham B15 2TT
The objective of this thesis is to elucidate goal processing in autonomousagents from a design-
stance. A. Sloman's theory of autonomous agents is taken as a startingpoint (Sloman, 1987;
Sloman, 1992b). An autonomous agent is one that is capable of using itslimited resources to generate
and manage its own sources of motivation. A wide array of relevantpsychological and AI theories are
reviewed, including theories of motivation, emotion, attention, andplanning. A technical yet rich
concept of goals as control states is expounded. Processes operating ongoals are presented,
including vigilational processes and management processes. Reasons forlimitations on management
parallelism are discussed. A broad design of an autonomous agent that isbased on M. Georgeff's
(1986) Procedural Reasoning System is presented. The agent is meant tooperate in a microworld
scenario. The strengths and weaknesses of both the design and the theorybehind it are discussed.
The thesis concludes with suggestions for studying both emotion("perturbance") and pathologies of
attention as consequences of autonomous goal processing.
"The problem is not that we do not know which theory is correct, butrather that we cannot construct
any theory at all which explains the basic facts" (Power, 1979 p.109)
"I think that when we are speculating about very complicated adaptivesystems, such as the human
brain and social systems, we should especially beware ofoversimplification—I call such
oversimplification “Ockham's lobotomy”. " (Good, 1971a p. 375)
To my wife, children, and parents
I wish to express my deep gratitude to Aaron Sloman for his energetic,consistent, and
insightful involvement in my research.
Thanks to Claude Lamontagne for showing me the importance of ArtificialIntelligence for
psychology; for demonstrating to me the poverty of empiricism; and formaking me aware of the
British tradition in computational psychology.
Thanks to everyone who was involved in the Attention and AffectProject.
Special thanks to Margaret Boden and Mike Harris for useful suggestionsand encouragement
for future research.
My doctoral studies were funded by a Commonwealth Scholarship from TheAssociation of
Commonwealth Universities in the United Kingdom, and scholarships fromThe Fonds pour la
formation de chercheurs et l'aide à la recherche, and the NaturalSciences and Engineering Research
Council of Canada.
Thanks to Patty for waiting in Canada while I indulged in research inBritain.
Chapter 1.Introduction........................................................................... 1
1.1 Human scenario ..................................................................... 1
1.2 Requirements of autonomous agents and of the theory......................... 2
1.3 Methodology—the design-based approach in context .......................... 5
1.4 Summary of the contributions of the thesis and the importance of itsobjectives 7
1.5 The technical nursemaidscenario.................................................. 8
1.5.1 A scenario in the nursemaid domain................................... 12
1.6 Overview of the thesis.............................................................. 12
Chapter 2. Literature Review.................................................................... 14
2.1.1 Goal theory ofmotivation............................................... 15
2.1.2 Autonomy theories of emotion......................................... 17
22.214.171.124 A communicative theory of emotion ....................... 17
126.96.36.199.1 Critique............................................. 19
2.1.3 Attention .................................................................. 21
2.2 AI and autonomous agents......................................................... 24
2.2.1 Wilensky on agents with multiplegoals............................... 25
2.2.2 Blackboard systems ..................................................... 26
188.8.131.52 A standard blackboard system.............................. 27
184.108.40.206 AIS: A blackboard model of autonomous agents......... 29
220.127.116.11.1 Assessment of AIS ............................... 30
2.2.3 Procedural reasoning systems.......................................... 31
18.104.22.168 The PRSarchitecture......................................... 33
22.214.171.124 Assessment of PRS.......................................... 35
2.3 Conclusion........................................................................... 38
Chapter 3. Conceptual analysis of goals ....................................................... 40
3.1 A provisional taxonomy of controlstates......................................... 40
3.1.1 Attributes of control states.............................................. 44
3.2. The conceptual structure ofgoals................................................. 45
3.2.1 The core information of goals.......................................... 45
3.2.2 Attributes of goals ....................................................... 47
126.96.36.199 Assessment of goals ......................................... 48
188.8.131.52 Decisions about goals........................................ 54
3.3 Competing interpretations of goal concepts...................................... 56
3.3.1 Formal theories of "belief, desire, intention"systems............... 56
3.3.2 Arguments against viewing goals as mental states................... 57
3.4. Conclusion.......................................................................... 61
Chapter 4. Process specification ................................................................ 62
4.1 Goal generation and goalmanagement............................................ 62
4.2 The control of management processing........................................... 67
4.2.1 Heuristic meta-management ............................................ 69
4.3 Resource-boundedness of management processing............................. 72
4.4 Goal filtering......................................................................... 80
4.4.1 Other functions offiltering.............................................. 84
184.108.40.206 Busyness filter modulation.................................. 84
220.127.116.11 Filter refractory period....................................... 85
18.104.22.168 Meta-management implementation ......................... 86
4.5 Summary of goal statespecification............................................... 90
4.6 Conclusion........................................................................... 92
Chapter 5. NML1—an architecture............................................................. 93
5.1 NML1—Design of a nursemaid................................................... 93
5.2 The Perceptual Module and the WorldModel.................................... 96
5.3 The EffectorDriver.................................................................. 97
5.4 Goals and Goal Generactivators................................................... 99
5.5 Insistence assignment............................................................... 101
5.6 Goal Filter............................................................................ 101
5.7 M-procedures and associated records............................................. 103
5.8. Databases of procedures........................................................... 106
5.9 The Goal Database .................................................................. 107
5.10 Epistemic procedures andprocesses............................................. 110
5.11 TheInterpreter...................................................................... 110
5.12 Algorithms form-procedures..................................................... 113
Chapter 6. Critical examination ofNML1...................................................... 121
6.1 Some strengths of the contribution................................................ 121
6.2 Valenced knowledge and conation ................................................ 123
6.3 Goal generators...................................................................... 127
6.4 The Interpreter and management processes ...................................... 128
6.4.1 Reasoning aboutprocedures............................................ 130
6.5 The need for a theory of decision-making—Problems withdecision-theory. 131
6.6 Conclusion........................................................................... 138
Chapter 7. Conclusion—summary of progress and directions forfuture research........ 140
7.1 Future research ...................................................................... 141
7.2 Attention andaffect.................................................................. 142
7.2.1 Perturbance and "emotion" ............................................. 143
7.2.2 Towards a study ofperturbance........................................ 146
7.2.3 Perturbance andobsession.............................................. 148
List of abbreviations.............................................................................. 152
References ......................................................................................... 153
Figure 1.1.......................................................................................... 10
Figure 2.1.......................................................................................... 34
Figure 3.1.......................................................................................... 41
Figure 4.1.......................................................................................... 63
Figure 4.2.......................................................................................... 64
Figure 4.3.......................................................................................... 69
Figure 4.4.......................................................................................... 84
Figure 5.1.......................................................................................... 95
Table 2.1......................................................................................... 22
Table 3.1......................................................................................... 47
Table 5.1......................................................................................... 99
Equation 6.1..................................................................................... 132
Chapter 1. Introduction
1.1 Human scenario
In the following scenario, consider the tasks and abilities of anursemaid in charge of four
toddlers, Tommy, Dicky, Mary, and Chloe. One morning, under thenursemaid's supervision the
four children are playing with toys. Mary decides that she wants to playwith Dicky's toy. So she
approaches him and yanks the object out of his hands. Dicky starts tosob, as he cries out "mine!
mine!" The nursemaid realises that she ought to intervene: i.e., to take the toy away from Mary,
give it back to Dicky, and explain to Mary that she ought not to takethings away from others
without their permission. This task is quite demanding because Dickycontinues crying for a while
and needs to be consoled, while Mary has a temper tantrum and also needsto be appeased. While
this is happening, the nursemaid hears Tommy whining about juice he hasspilt on himself, and
demanding a new shirt. The nursemaid tells him that she will get to himin a few minutes and that
he should be patient until then. Still, he persists in his complaints.In the afternoon, there is more
trouble. As the nursemaid is reading to Mary, she notices that Tommy isstanding on a kitchen
chair, precariously leaning forward. The nursemaid hastily heads towardsTommy, fearing that he
might fall. And, sure enough, the toddler tumbles off his seat. Thenursemaid nervously attends
to Tommy and surveys the damage while comforting the stunned child.Meanwhile there are fumes
emanating from Chloe indicating that her diaper needs to be changed, butdespite the
distinctiveness of the evidence it will be a few minutes before thenursemaid notices Chloe's
Fortunately, human life is not always as hectic as that of a nursemaid.Nevertheless, this little
scenario does illustrate some important human capabilities, and the"motivational" processes that they
evince. (We are focusing on the nursemaid, not the children.) Whiledirecting the planned activities of
the day, the nursemaid is able to detect and respond to problems,dangers and opportunities as they
arise, and to produce appropriate goals when faced with them. Forinstance, when Mary violates
Dicky's rights, the nursemaid needs to produce a collection of goalsincluding one to comfort Dicky,
to instruct Mary, and to comfort her too. The nursemaid is able toprioritise and schedule goals that
cannot be executed simultaneously. Thus she decides that cleaningTommy's dirty shirt can wait until
Dicky and Mary are sufficiently calm. Although very resourceful, thenursemaid is, of course, neither
omniscient nor omnipotent. When she is involved in a crisis, she mightfail to notice other problems
(such as Chloe's diapers). The nursemaid might even have to abandonsome of her goals (though this
scenario did not illustrate this). This nursemaid scenario is referredto throughout the thesis, and a
technical version of it is described.
The objective of this thesis is to elucidate goal processing in autonomousagents such as the
nursemaid: to try to give an account of the functions, constraints, andkinds of goal processes, and to
investigate the cognitive architectures that can support theseprocesses. This objective is expounded in
this chapter. Understanding this objective requires a preliminary notionof autonomous agency,
which is given in the following section along with the objectives of thethesis. Design-based
methodology is described in detail by A. Sloman (1993a) and summarisedbelow. The introduction
also summarises the accomplishments of the thesis, describes a technicalversion of the nursemaid
scenario, and gives an overview of the thesis.
1.2 Requirements of autonomous agents and of the theory
It is assumed that an agent is autonomous to the extent that it iscapable of producing its own
objectives but has limited resources with which to satisfy them. Some ofthese objectives are "top-
level", meaning that they are not ontogenetically derived as means tosome end, such as through some
planning process; or if they are derived, that they have achieved"functional autonomy" (Allport,
1961) in as much as the agent treats them as good in themselves.Similarly, some top-level goals are
derived from an evolutionary process even though the agent treats themas non-derivative. There is a
large and controversial literature on what are the "true" objectives ofhuman life. For instance,
Aristotle (1958) has argued that there is only one non-derivative goalin humans: happiness. For
behaviourists, the objectives of behaviour (if any) are to seekreinforcement and avoid punishments.
A few stimuli are innately reinforcing (or punishing); but mostreinforcing (or punishing) stimuli have
that status through association with other reinforcing stimuli. ForFreud, the ego seeks a compromise
between an id that works according to a "pleasure principle" and the superego thatincorporates
versions of parental values. There are many theories of the ends ofaction. This thesis is not
concerned with specifying the innate objectives of human life. It merelyassumes that an autonomous
agent has some number of top-level goals and a greater number ofderivative ones.
The word "autonomous" is used as a technical term, in order concisely torefer to a class of
agents. There is a long history of debate concerning what autonomy"really" means. However, the
current thesis is not meant to contribute to this debate. An arbitrarynew term could have been used
instead of "autonomy", but since this term has a colloquial meaning thatis close to the one referred to
here, it has been adopted. Normally one would not include the concept"resource-bounded" in one's
definition of autonomy (for in principle an agent whose resourcessurpassed its desires might still be
called autonomous). However, it is expedient to do so in this documentsince all the agents it
discusses are resource-bounded in some sense (and "autonomousresource-bounded agents" is too
wordy an expression for one that is used so frequently).
In order to explain goal processing in autonomous agents, one needs tounderstand what
requirements they satisfy. Doing this is an objective of this thesis andshould be read as a theoretical
contribution, since the requirements are falsifiable, or in principlecan be shown to be deficient in
number or organisation. Requirements analysis is roughly analogous tothe notion of "computational
theory" discussed by D. Marr (1982). Here follows an overview of therequirements of autonomous
As mentioned above autonomous agents have multiple sources ofmotivation. They do not
merely have one top level goal. These sources of motivation will leadthem to produce particular
goals, either as means to some end, or as an instantiation of themotivational source. The sources of
motivation can be triggered asynchronously to the agent's other mental processes. For example, the
(top-level) goal to eat can be triggered asynchronously to one'sprocess of planning how to get from
one place to another. Triggering of motivational sources can either bethrough internal or external
events. For example, if the nursemaid had a desire to eat, it might havebeen triggered by an internal
event (a drop in her blood sugar levels) or an external one (e.g., seeing palatable food). The
multiplicity of motivation implies that the agents have many different tasks that they must perform.
There are important temporal constraints acting on autonomous agents.They need
asynchronously to be responsive to the very sources of motivation thatthey activate. That is, motive
processes should be able to interrupt other process. For example, whenthe nursemaid produced a
goal to comfort Dicky, this interrupted her process of reading to Mary.The agent needs to be able to
discover, set, and meet deadlines for its goals. This implies that someof the algorithms that it uses
should be "anytime algorithms" (Dean & Boddy, 1988; Dean & Wellman,1991; Horvitz, 1987). An
anytime algorithm is one that can produce a result the quality of whichis a function of the time spent
processing. S. Russell and E. Wefald (1991) distinguish between twokinds of anytime algorithms.
A contract anytime algorithm is one which before it starts to execute is given anamount of time that it
can use before it must produce a response, and arranges to produce thebest solution that it can within
this time frame (e.g., it might select a method that requires the specified amount of time).An
interruptable anytime algorithm is one that can be interrupted as it is going and yetstill emit a sensible
response. Engineers have devised many anytime algorithms, but not alldevices use them. For
instance, a typical calculator is not interruptable—it either gives aresponse or it does not. Many chess
playing computer programs use contract anytime algorithms—the usercan set the amount of time
which the machine uses to make its move. Anytime performance is a formof graceful degradation,
or graceful adaptation. Further temporal constraints are discussed inthe core of the thesis.
There are various limits in the resources that autonomous agents havewith which to deal with
their goals. In particular, their beliefs are incomplete and may containerrors. They have limited
abilities to predict the consequences of actions. Their processors workat a finite (though possibly
variable) speed and have a finite set of mechanisms (though this setmight increase and diversify with
time). They have limited external resources of all kinds (principallyeffectors, tools, etc.). Temporal
constraints have already been noted.
The strategies of autonomous agents must be robust, in the sense thatthey must operate in a
wide variety of settings under various constraints. Autonomous agentsmust be adaptable, in that if
they do not immediately have strategies that they can apply to generatethe right goals and satisfy
those goals in a new environment, they can adapt their strategies at somelevel to function in the new
environment. This implicates requirements for learning. However,although requirements of
robustness and adaptability are important, they are not examined closelyin this thesis.
As B. Hayes-Roth (1993) points out, autonomous agents have to dealwith complex contextual
conditions. That is, there are usually many variables that are relevant to thecontrol of their behaviour,
some of which are internal, some external, and some both.
As will become increasingly obvious throughout the thesis, autonomousagents integrate a wide
range of capabilities. Thus the computational architectures that modelautonomous agents will be
"broad" (Bates, Loyall, & Reilly, 1991). Many architectural componentsare active simultaneously,
implying parallelism at a coarse grained level. For example, theirperceptual mechanisms operate in
parallel with motor processes, and processes that trigger sources ofmotivation ( e.g., new goals) and
that deal with the sources of motivation (e.g., planning processes).
There are many other requirements besides those listed here that can bederived from them e.g.,
the importance of directing belief revision as a function of the utilityof inferences produced (Cawsey,
Galliers, Logan, Reece, & Jones, 1993). The requirements are expandedin Ch 4. Other requirements
will not be addressed here, such as social communication with others.Some of these other
requirements will be easier to study once theories account for the mainrequirements.
An increasing number of researchers in computational psychology andArtificial Intelligence are
addressing the requirements of autonomous agents (though usually inisolation). It is therefore a very
exciting time to be performing research in this area. The requirementsdo not appear to be very
controversial; however, it is not clear that everyone realises thedifficulty of explaining how the
requirements could be met (let alone how they are actually met by humans). (For more on
requirements, see Boden, 1972; Hayes-Roth, 1990; Hayes-Roth, 1992;Oatley, 1992; Simon, 1967;
Sloman, 1985a; Sloman, 1987).
1.3 Methodology—the design-based approach in context
The foregoing discussion of "requirements" and "architectures", as wellas the title of the thesis
foreshadowed the current section, in which the design-based approach isdescribed and contrasted
with related methodologies.
Much has been written about the different ways to conduct science.Cognitive science is a
particularly rich area in that many methodologies are used. Here thetaxonomy of methodologies
related by Sloman (1993a) is given. Phenomena-based research proceeds either in a positivist or
falsificationist (Popper, 1959) manner by collecting empirical datawhich either support or refute
theories. In cognitive science, these data are supposed to shed light oncognitive systems through
correlational or causal links between observable states, processes, andevents. See (Keppel, 1982)
for prescriptions concerning empirical research methodology.Phenomena-based research is mainly
concerned with the "actual" rather than what is possible or necessary.In contrast, the current thesis
does not present new phenomena-based research. However, in order tospecify what needs to be
explained, it occasionally refers to fairly obvious facts about humans(as opposed to very detailed
empirical findings). Historically, institutional psychology (includingtheoretical psychology and
cognitive psychology) has almost exclusively been concerned withempirical research (Green, 1994).
There is also semantics-based research in which scientists study concepts and relations between
them. This involves techniques of "conceptual analysis" used chiefly(but not only) by philosophers.
(See Sloman, 1978 Ch. 4; Warnock, 1989). For example, A. Ortony,G. L. Clore, and M. A. Foss
(1987) have analysed the concept of emotion, and proposed a taxonomyof emotion concepts.
Psychologists and linguists often carry out a related kind of researchin which they try to specify what
people actually mean by colloquial terms. Conceptual analysis can useempirical data about what
people mean by terms as a starting point, but not as a final criterionfor the validity of their analyses.
Analysing concepts can be useful in the design-based approach, as well.In Ch. 3 some of the results
of a conceptual analysis of goals are presented.
The design-based approach, used chiefly in AI, involves taking an engineeringscientist
methodology for studying real or possible systems. It has five mainsteps some of which can be
executed recursively or in parallel. (1) Specify the requirements ofthe system in question. That is,
what capabilities does or should the system have? What are its tasks,and why does it have them? A
ply of requirements analysis of autonomous agents was presented in theprevious section. This is
extended throughout the thesis. (2) Propose designs which can satisfythe requirements. A design
comprises an architecture and its mechanisms. An architecture comprisesmodules (components) that
have causal links between them (e.g., data transmission, control, inhibition, etc.) The architecture
need not be described at a physical level, i.e. its components can exist in a virtual machine. (3)
Implement designs (which can be prototype designs) in a computersimulation or in hardware. This
helps to uncover lacunas and inconsistencies in a theory. (4) Analysehow, and the extent to which,
the design meets the requirements, and how the simulation embodies thedesign. The analysis can be
both mathematical and based on experimental tests of the implementation.(5) Study the space of
possible designs surrounding the proposed model: How could the modelhave been different? What
are the trade-offs that are implicated in the design? How would slightchanges in the requirement
impact on the design? What further capabilities could the system have ifits design were slightly
different? A complete understanding of a design requires that one cancharacterise it in relation to
other designs in the space of possible designs (Sloman, 1984; Sloman,1993a; Sloman, 1994c).
Although the design-based approach is distinguished from thephenomena-based
methodologies, that does not imply that it cannot yield theories abouthumans (or other species). In
fact quite the contrary is true, for in order to understand howindividuals of some species really
operate, one needs to have cogent theories about how they could operate. In other words, one can
only understand actual systems through reference to possible systems (ifa model could not possibly
be implemented to satisfy the requirements, then it cannot empiricallybe correct). The kind of
autonomous agency studied here involves such a sophisticated set ofcapabilities that it will take many
years (perhaps centuries) before we have plausible working conjecturesabout how they can be
realised. Once we have such theories, we will be in a good position tosuggest an empirical theory,
and then try to refute it. This is not to say, however, thatphenomena-based research is useless. There
is a need for many different types of research to be pursued inparallel, with some interaction between
There are many different ways in which design-based research can beconducted. See Sloman
(1993a) for a number of variables. One dimension of variation ofresearch is the "breadth" of the
requirements that are studied and of the architectures that areproposed. Most research in cognitive
science focuses on a very narrow set of capabilities, such as how visualperception of motion is
possible, how one can identify speakers solely on the basis of acousticinput, what is responsible for
spatial Stroop effects, etc. These questions can lead to the productionof very detailed models. Even
someone who is interested in autonomous agents does not necessarily tryto provide a broad picture
of the agents (e.g., she can focus on one of the requirements, such as time dependentplanning). In
this thesis, however, a very broad set of capabilities is addressed(compare previous section). This
makes the task more difficult, and implies that the solutions that areproposed will be more sketchy
for a longer period of time. J. Bates, A. B. Loyall, and W. S. Reilly(1991) have suggested a useful
way of representing the distinction between the resultant architectures.Some will be very narrow
(looking at a very specific task) but very deep (giving plenty ofdetail about the mechanisms
underlying the task). Others will be very broad, but shallow. Inpractice, depth and breadth are
traded-off. Of course, ultimately broad and deep architectures are mostdesirable.
This section has briefly expounded the design-stance not for the purposeof convincingly
defending it—that would require more space than is available—butin order to set the framework for
the rest of this thesis, which can be read as a case study indesign-based methodology.
1.4 Summary of the contributions of the thesis and the importance of itsobjectives
I have approached the objectives of this thesis by applying andimproving an existing theory of
motive processing in autonomous agents proposed by Sloman in variouspublications. Through
conceptual analysis and design exploration, this thesis directly buildsupon Sloman's work, and it
relates it to other theories. In this research I have
• systematically addressed the issue of how goals are processed inautonomous agents from a
• collected and reviewed a number of theories from a wide range ofresearch areas that bear on the
issue of autonomous agency. These theories had never been consideredtogether before. I have
shown how these theories contribute pieces to the puzzle of autonomousagency, and how they
can benefit from one another;
• further elaborated requirements for autonomous agents;
• provided a conceptual analysis of goals that views them as rich controlstructures with a variety of
attributes and dimensions. This analysis generalises and clarifiesprevious work;
• proposed a new taxonomy of goal processes that distinguishes betweenvigilational processes and
• described important unsolved problems in the control of goalprocesses;
• proposed new concepts, terminology, and conceptual distinctions, e.g., "busyness",
"management" processes, "deciding" goals, "generactivation","surfacing", "criticality", and a
distinction between the intentional and propensity interpretations ofinsistence;
• addressed the question, "Can some processing limitations be shown to beuseful or necessary
• analysed, adapted, and improved a promising extant architecture forautonomous agents,
(Georgeff & Ingrand, 1989);
• analysed the proposed architecture's strengths and weaknesses, therebysetting the scene for
• made a number of specific proposals for new research following on thework in this thesis;
• indicated a conceptual resemblance between emotion (as "perturbance")and a psychopathology
(obsessive compulsive disorder).
Contributions such as these stand as progress towards a deeperunderstanding of goal
processing in autonomous agents. Such an understanding is extremelyimportant for theoretical and
engineering reasons. It will help to explain human motive processingmechanisms, by situating them
within a space of possible designs. A deep understanding of goalprocessing should help to explain
emotion-like phenomena which are referred to as "perturbance" (cf. Ch.3 and 7). An understanding
of normal goal processing should also help to characterise pathologiesof goal processing and
attention, such as are supposed to occur in affective and anxietydisorders (American Psychiatric
Association, 1987). It is hoped that this understanding, in turn, willhelp to propose intervention
schemes to deal with such disorders, as well as with less severeproblems. Finally, one will be in a
better position to build autonomous systems (robots, programs, etc.)that can take on progressively
more responsibilities (e.g., security systems, unmanned space craft systems, emergency response
systems). However, these benefits will only fully be reaped after manyiterations of the slow and
difficult cycles of design-based, semantic, and empirical research.
1.5 The technical nursemaid scenario
The human nursemaid scenario described above is useful for expoundingthe problems of
autonomous agency. However, in order eventually to give an account of ahuman nursemaid (or any
other human autonomous agent) first one needs to design models ofsimpler agents—as research
progresses, the models will become increasingly sophisticated. For thisreason, a technical version of
the nursemaid scenario has been developed. (Hereafter, this is referredto as the "nursemaid scenario"
or simply "the scenario".) The scenario was originally proposed bySloman (1986), and was adapted
for this thesis (Beaudoin & Sloman, 1991; Beaudoin, 1991). Thescenario was created to require of
an agent capabilities that are similar (at some level of abstraction)to human—autonomous—agents
while ignoring other problems that are best left to other researchers,including 3-D vision, motor
control, and naive physics. Hence the agent faces multiple (sometimesindependent) problems that
can occur and develop in overlapping time intervals and that need to bedetected currently with and
asynchronously to the agent's other activities. The problems differ intheir urgency and importance
profiles. Some problems get worse at a faster rate than others. Someproblems have terminal
urgency, others do not. Some problems only have derivative importance;whereas others are
intrinsically aversive and some states are intrinsically good. (If theagent could learn, some of the
things that were extrinsically aversive could become intrinsicallyaversive to it, and similarly for the
good things.) However, the domain is biased in that there is anover-representation of aversive
sources of motivation in relation to positive sources. The agent'scognitive and physical behaviour
execute in parallel. The agent's perceptual focus is limited, and hence(unlike a typical program
playing chess) it does not know all of the facts about its world. Manyevents in this world are
unpredictable from the agent's perspective.
The "physics" and "psychology" of the domain can be extendedindefinitely as required for
testing later more complex versions of the theory.
The scenario is intended for a computer simulation, not primarily arobot implementation. The
scenario involves a "robot" nursemaid whose function is to care for"robot" babies that roam around
in a nursery, preventing problems and responding to them when theyoccur. Babies arrive at
intervals, have to be protected from various dangers, and can eventuallybe discharged when they
have reached a certain age. To discharge its function, the nursemaid hasa single camera that can see a
limited portion of the nursery at a time, and it has a claw with whichit can pick up and transport one
baby at a time. (For pragmatic reasons, it is assumed that thenursemaid's computer exists outside the
nursery, and that it has remote control of its claw and camera, whichcan be moved independently.)
The nursery comprises a set of rectangular rooms separated by walls andconnected by open
doors. The rooms are bounded by deadly ditches. One of the roomscontains a recharge point,
another an infirmary machine, and another a baby dismissal point. Theclaw and babies are
considered as shapeless points. (See Figure 1.1).
1 3 5
2 4 6
Figure 1.1. The Nursery. Room numbers are given in the upper right corners ofthe rooms.
There is a variety of problems and other contingencies to which thenursemaid must respond.
Babies grow older, and when they reach a certain age or, if they die,they need to be removed from
the nursery by being brought through the dismissal point. Babies die ifthey fall into ditches.
Therefore, the nursemaid needs to keep them away from ditches. Babiesalso die if their battery
charge expires; therefore, the nursemaid needs to recharge them in duecourse. It can do this by
connecting the babies to the recharge point. Babies can also die if theycontract certain illnesses. Ill or
injured babies can be healed at the infirmary. Babies cannot all be putin the same room; for if the
population density surpasses a certain threshold in one room, then thelikelihood that some babies
become thugs increases. Thug babies tend to attack and injure others.Thug babies should be isolated
in order for them to lose their malicious tendencies. New babies canarrive in the nursery. Dead
babies emit a magnetic field that can corrupt the memories of otherbabies; babies with corrupt
memories can die. Therefore, it is important to dismiss dead babies.Corrupt memory is the only fatal
"illness"; however, it can be cured in the infirmary. Other diseasesthat can develop are: the "shakes",
and the "melts". They are all intrinsically bad; the shakes are cyclical,whereas the melts get
monotonically worse. Injuries are not fatal. They can be incurred toeither arm or leg of a baby, or to
the head. The domain also affords a number of potential opportunities( e.g., in some circumstances it
will be possible to solve two problems at once, or prevent a problem ata lower cost than at other
junctures). Thus, there are a variety of potential problems andopportunities that are causally related,
and have temporal patterns.
There are a few domain rules which the nursemaid should follow in caseof a conflict between
its goals. If the nursemaid can only save one of two babies, it shouldprefer faster babies to slower
ones, healthier babies to less healthy ones, older babies to youngerones, and innocent ones to those
that have been thugs. But since it should preserve the health and wellbeing of as many babies as
possible, if the nursemaid has to choose between saving two low valuebabies and one high level
value, it should save the former. Notice that the domain does notexplicitly quantitatively specify
values for outcomes, instead the required preferences are stated interms of rules and partial orders.
There is no objective notion of "utility" (compare the discussion ofutility in Ch 6). The given
preferences are not sufficiently extensive for the nursemaid (or ahuman being, for that matter) to be
able to infer for every pair of outcomes which one is preferable. Thisis so even when the outcomes
are completely known. The designer of the nursemaid must invent a morespecific decision-making
scheme. (It would be insightful to observe the kinds of preferencesthat a human being playing a
game version of the nursemaid scenario would invent.) This "invention"will not be totally arbitrary,
since there are causal relations amongst problems and objectiveconstraints in the domain, and there
are some preference rules which in practice will usually preserve theobjective domain preference
rules. As an example of a potentially useful rule which the nursemaidcould follow is that isolating a
thug is usually more pressing than fixing the babies which it hasinjured. This is because injuries are
intrinsically bad, and the longer a thug is on the loose, the moreinjuries are likely to occur. There is a
potential for the rate of injuries caused by the thug to be greater thanthe rate at which they are fixed in
the infirmary; however, this depends on parameters of the domain, suchas the speed of travel of the
thugs, the number of hits that are required for an injury, the frequencywith which thugs tend to
attack babies. Therefore, this rule can be invalidated if the parameterschange. Moreover, the rule
breaks down in some situations, e.g., if all the other babies in the room are dead.
The main task is to design the nursemaid. This is not a study ofmultiple co-operating and
communicating intelligent agents. That is, the babies are considered asvery simple automata, whereas
the nursemaid is supposed to be a proper autonomous agent. The nursemaidrequires cognitive
abilities for detecting, prioritising, resolving problems, etc.,according to the requirements described
in Section 1.1. A design of a nursemaid (called "NML1") is given inCh. 5. Prototype computer
simulations were performed to help improve the design, but the finaldesign was not implemented by
the author, although Ian Wright of the University of Birmingham isimplementing his design. The
implementations are not reported here.
The domain is not designed to have any political, social, or economicsignificance. It is simply
meant to embody a set of high level design requirements of autonomousagents. Furthermore, it can
be extended in order to test the proposed design and better show how thedesign ought to be
improved. For instance, one could require that the nursemaid needs to berecharged too, give the
nursemaid auditory-like perception (to hear babies screaming, or judgepopulation densities on the
basis of wave-forms), allow robot "ogres" to snatch babies, give thenursemaid additional claws, or
replace the babies by workers in a workshop factory.
1.5.1 A scenario in the nursemaid domain
In a typical scenario, the initial state of which is depicted in Figure1.1, the nursemaid detects that
babyA has a low charge. Having no other pressing problem to solve, thenursemaid decides to
recharge it. As it is moving its claw toward babyA, the nursemaidnotices that babyB is perilously
close to a ditch. It decides that it had better interrupt its currentendeavour and rescue babyB. As it
starts to execute its plan to rescue babyB, it perceives babyC which isnow sick; however, with the
two other problems demanding attention, the nursemaid fails to "realise"that there is a problem with
babyC. Later, babyC dies of its fatal illness.
A model of how the nursemaid's behaviour in this scenario could beachieved is given in Ch. 5.
1.6 Overview of the thesis
The thesis provides a literature review, a conceptual analysis of goals,a process specification of
goals, an architecture for goal processing, a critique of thearchitecture, and a conclusion which
outlines future research.
Chapter 2 reviews relevant psychological and AI theories. The thesisobjectives implicate a very
wide range of theories, which themselves involve a broad range ofpsychological functions. The
review is necessarily selective. One theory from each of four areas ofpsychology is reviewed. In the
area of goal theory, which examines psychometric factors involving goalsfor predicting behaviour,
the theory of Thomas Lee and Edwin Locke is examined. In the area ofemotion, Keith Oatley &
Philip Johnson-Laird's Communication theory is selected. This isclassified as an "autonomy theory
of emotion". Richard Shiffren & Walter Schneider's theory of attentionis reviewed. From the AI
literature, Robert Wilensky's model of multiple motive agency ispresented. Two AI models of
autonomous agency are also reviewed: B. Hayes-Roth's AdaptiveIntelligence System and M.
Georgeff's Procedural Reasoning System. Each model contributes somethinguseful and can benefit
from the others.
Chapter 3 expounds the concept of goal in terms of "control states". Theconceptual structure of
goals is presented. Nearly a dozen features of goals are analysed,including their importance,
rationale, insistence, commitment-status, and intensity. The use ofintentional terminology, such as
"beliefs" and "goals", is quite controversial. It is therefore importantto justify the mechanistic
interpretation of these terms. One of the most persuasiveanti-mechanistic views on the issue—D.
Dennett's "intentional stance"—is summarised and criticised.
Chapter 4 gives a process specification of goals. A distinction between"high level"
management processes and "lower level" vigilational processes is drawn,the functions of these
categories are described, and the categories are subdivided. Thestate-transitions of goals are very
flexible—this raises the issue of how to control them. Sloman'snotion of insistence based goal
filtering is explained as a vigilational function. A distinction isdrawn between two interpretations of
insistence: an intentional interpretation and a propensityinterpretation. New functions for filtering are
supposed. Part of the rationale for filtering is that there is a limitto the amount of concurrency that
management processes can accommodate. This assumption is discussed. Theprocess specification
contributes to the requirements of autonomous agents.
Chapter 5 describes a design of a nursemaid, called "NML1", which willdisplay some (but not
all) of the processes described in Ch. 4, and which will be aprocedural reasoning system. The
architecture assumes a number of modules that execute in parallel(though some of them are partly
synchronised), including goal generactivators, insistence filters, aninterpreter, a collection of
management processes, and perceptual and effector devices. Algorithms forsome of these modules
are presented, but further research is required to explore a widervariety of algorithms, and better
select amongst them.
Chapter 6 presents a critical examination of NML1 and extant theory ofautonomous agents. It
describes the strengths and weaknesses of the design, and points atareas where more research is
needed. It is suggested that an autonomous agent should separate itsproblem description from its
goals, and be capable of representing valenced information. Some of thedifficulties with reasoning
about procedures are identified. The need for theories to help designmechanisms for controlling
management processing is identified. There is also a need for aqualitative theory of decision-making,
given a criticism of utility-based decision-making.
Chapter 7 concludes the thesis by summarising it, situating it withinthe context of a broader
project concerned with Attention and Affect, and suggesting fruitfulareas for future research.
Chapter 2. Literature Review
There is an enormous amount of disparate literature in psychology and AIthat is potentially relevant
to the topic of goal processing in autonomous agents. Thousands ofarticles have been published on
the topics of motivation, emotion, "self-regulation", and attention.Rarely are these topics considered
together. Affective processes are rarely considered by cognitivepsychologists; however, when they
are, the cognitive psychologists are usually concerned with the effects of these processes on
"cognition" (e.g., as biasing decision-making, or speed of information processing), butaffective
processes are often not considered as cognitive processes or informationprocesses themselves. In the
AI literature, goal processing has been examined, but usually does notuse the terms of motivation,
emotion, self-regulation, and attention. There are, of course,exceptions to this rule, ( e.g. Boden,
1972; Simon, 1967; Sloman, 1978; Sloman & Croucher, 1981). It isfitting for a thesis on broad
architectures to take a look at a broad spectrum of research.
Although many areas of research are examined here, only one theory perarea will be
considered. This survey has three main objectives. One is to demonstratethat many leading theories
in different areas can benefit from each other: each has strengths thatare lacking in the others. The
second is to indicate good ideas to build upon, and pitfalls or problemsto overcome ( e.g., limitations
of existing designs). The third is to illustrate the design-basedapproach to evaluating psychological
and AI literature on autonomous agents.
The first part of this chapter examines some psychological literature.The second part examines
AI literature on autonomous agents. The conclusion shows how the varioustheories complement one
another but do not provide a complete account of autonomous goalprocessing. Later chapters attempt
to integrate the contributions—but it will be years before such anintegration is complete.
Four main areas of psychological research are reviewed. Firstly, atheory of motivation based on the
notion of "goal setting" is presented. Secondly, a category of theoriesof emotion is described as
viewing emotion as a consequence of requirements of autonomous agents.The communicative theory
of affect of Keith Oatley and Philip Johnson-Laird, which is a member ofthis category, is discussed.
Thirdly, two theories which divide mental capabilities into attentionaland automatic processes are
discussed—namely the theory of Walter Schneider, Susan T. Dumais, andRichard M. Shiffrin, and
the theory of Donald Norman and Tim Shallice.
2.1.1 Goal theory of motivation
The social sciences have witnessed a proliferation of research on thedeterminants of goals, and the
impact of goals on behaviour (Bandura, 1989; Lee, Locke, & Latham,1989). These determinants are
seen as "factors". An important goal of this research has been todetermine what factors there are, and
the correlations and causal relations amongst them. However, theultimate aim of this research is to be
able to predict and control performance on the basis of motivationalmeasurements. (In contrast, the
current thesis is concerned with explaining possibilities. (See Ch. 2of Sloman, 1978). The theories
that are proposed differ slightly in their definition of factors, and inthe exact relations that are
hypothesised to hold amongst the variables.
In this section, the "goal setting" theory of T. W. Lee, E. A. Locke, and G.P. Latham (1989)
is discussed. The discussion aims to underscore some of thecontributions of goal theory, and to
distinguish goal theory from the theory proposed in this thesis.Although, goal theory is a
phenomena-based theory, it is discussed here without direct reference tothe empirical research that
led to its postulates.
Goal theory is supposed to provide a "specification of goal processes".The theory emphasises
the positive effect on performance of an individual "setting" specificand difficult goals. The main
assumptions are that (1) the content of a goal determines the mentalprofile of behaviour towards the
goal, and this profile in turn impacts on performance; (2) thesecausal relations are subject to
moderating influences, as described below. Goals have four components:(1) The goal level is the
difficulty of the state to be achieved. For instance, a student mightaim to be in the top 5th or 10th
percentile—the higher the percentile, the higher the goal level.(2) There is the degree of quantitative
specificity of the goal. For example, the goal to be within the 10th percentile ismore specific than the
goal to "do well academically". (The fact that the authors focus onquantitative specificity may be due
to a general prejudice against qualitative formulae in science, for inprinciple qualitative objectives can
be just as precise as quantitative ones.) (3) There is the "complexity" of the goal; by this they mean
the number of subgoals that are required to satisfy it. (The term"complexity" as used here is slightly
misleading, because whereas they say that it is a predicate of goalstheir concept pertains to plans. In
fact, a goal may be non-complex (in the logical sense) whiletriggering a complex plan, and vice-
versa. A more adequate notion of complexity is proposed in Section3.2.2.) (4) There is the conflict
between a target goals and other goals. Goal level and goal specificityare assumed to combine
additively to affect the behaviour profile. However, goal specificityonly has its effect if the goal level
is high. The "complexity" of the goal and goal conflict negativelyaffect the behaviour profile. Goal
specificity is assumed to affect the profile of behaviour so long as thegoal is difficult.
The behaviour profile comprises direction of effort. This simply meansthat behaviour is
selectively directed towards the goal. There is a quantitative dimensionof amount of effort, and one
of persistence in the face of external difficulties. And "task strategy" representsthe plans that are used
to execute the task.
There is a moderating factor between goal content and behaviour profile:goal commitment.
(See Hollenbeck & Klein, 1987) for a competing model of goalcommitment). That is, no matter how
"complex" or difficult the goal, the agent will only work for it if heis committed to it. Although this
variable is assumed on empirical grounds, it is apparent that there is aconceptual constraint operating
here as it is part of the logic of the colloquial concept of not beingcommitted to a goal that one will
not work towards it. (Admittedly, commitment and intention are slipperyconstructs that have been
hotly contested at least since the ancient times (Aristotle, 1958). Inparticular, there is a sense in
which one can be "not committed" to a goal whilst pursuing it.) Goalcommitment is said to be
affected by a number of factors: the legitimacy of the authority of the person who sets the goals (the
authors are interested in social settings where goals trickle down anorganisational hierarchy); peer
and group pressures; expectancy that one's endeavours will be successful; the extent of one's
perceived general and task specific self-efficacy; the value of the goal and its instrumentality (in
A collection of other variables is proposed which affect the linkbetween the behaviour profile
and performance, such as the person's knowledge, feedback, toolsavailable, etc. It is noteworthy
that these constructs are represented quantitatively.
Characterising goal theoretic research is useful in order to put thepresent thesis in a distinctive
context. Goal theory underscores a number of variables that need to beconsidered in goal processing.
Some of these factors are social (e.g., peer and group pressures) and the present research will ignore
them because it is believed that before characterising social agency oneneeds to characterise non-
social agency. There are some people who believe that intelligence,intentionality, consciousness, etc.
are only possible for social agents, but this is contrary to the author'sassumptions. Goal theory also
usefully describes motivation as a multi-dimensionalphenomenon—motivation is not simply the
amount of effort that a person is willing to exert for a goal. A similartenet is expressed in the
following chapter. The factors of goal theory, however, are specified ata high level, and not in
information processing terms. Thus the notion of "processes" that isused is different from the
engineering notion used in this thesis—e.g., the "process" diagrams of Lee et al. (1989) are not state-
transition diagrams or petri graphs. Lee et al. (1989) do not model the states of goals as low and high
level decisions are taken about them. Their "process specifications" arereally specifications of
statistical relations between variables (or factors). This is usefulfor Lee et al. (Lee, et al., 1989), to
the extent that they can measure the variables, manipulate them, andthereby exercise some control
over behaviour. Furthermore, the "processes" are not meant to beembodied in a computational
architecture, let alone an architecture that has to solve problems in anenvironment. Moreover, all of
the variables that are considered by goal theory are quantitative,whereas in a design-based
framework many of these "variables" would actually translate intomechanisms or structured data,
such as descriptions of states to be achieved or prevented. For example,there would be a mechanism
for feedback rather than just a quantitative causal link. Furthermore,the theory does not provide an
account of multiple goal processing. In Chapter 3, a concept of goal isproposed that is richer than the
one presented here.
2.1.2 Autonomy theories of emotion
In the very early years of AI and computational psychology, affect (e.g., motivation and emotions)
was a prominent area of investigation (e.g., Taylor, 1960; Tomkins,1963; Toda, 1962; see Boden
1972, 1987 for reviews). Perhaps because of the difficulty of the task,interest in affect waned in the
1960's and 1970's. However, since circa 1985 affect has been studiedby a growing number of
computationally minded scientists. Whether by coincidence or not, thisgrowth coincides with the
growing interest in autonomous agents in AI. Thus there are a number oftheories of emotion that
claim that emotions are a consequence of the requirements of autonomousagency: i.e., in order to
design an agent which meets these requirements, evolution (or any otherdesigner) must produce a
system with emotion producing mechanisms (e.g. Frijda, 1986; Oatley & Johnson-Laird, 1987;
Simon, 1967; Sloman & Croucher, 1981; Toda, 1962). Some of thesetheories are "functionalist" in
the sense that they view emotions either as being a process or system(Frijda, 1986) or as a state
resulting from a special purpose system specifically designed to dealwith emotional situations (Oatley
& Johnson-Laird, to appear); others are afunctionalist (Sloman &Croucher, 1981) in that they view
emotions as being an emergent property of a system made of componentseach of which has a
function, but none of which is specifically designed to produce anemotional state. (The issue of
functionalism is briefly discussed in Ch. 7.)
Rather than review the whole literature, this section focuses on onetheory, the communicative
theory of emotions. Sloman's theory is briefly described in Ch. 7.
22.214.171.124 A communicative theory of emotion
Keith Oatley and Philip Johnson-Laird have proposed an empirical (butpartly design-based)
communicative theory of affect. The theory was originally published in (Oatley & Johnson-Laird,
1987) but it has recently been revised in (Oatley & Johnson-Laird, toappear). This theory stems from
a recognition that autonomous agents need to be able globally toredirect attention when faced with
These theories have not been thus categorised before. They are sometimescalled "cognitive theories", but this is
a different category, since not all cognitive theories are autonomytheories.
significant junctures regarding their plans, such as changes inprobability of goal satisfaction. When
an agent detects that a goal is "significantly" more likely to beachieved than it previously believed,
this leads to a particular class of positive emotion (happiness).Decrease in probability of goal
satisfaction leads to negative emotions. These emotions serve tocommunicate this change both
between processors within the individual's mind, and betweenindividuals. (Here we will ignore
social requirements, however.) Emotional communication is supposed tobe both rapid and usually
effective. The communication leads to an interruption of processing andan adjustment in the system's
The communicative theory assumes that the mind comprises a hierarchy ofparallel processors
where the parallelism is coarse, and not necessarily neural. SeeJohnson-Laird (1988 Part VI). At the
highest level of the hierarchy there is a processor corresponding to"consciousness" which exercises
control over the lower levels, and uses semantic messages as well ascontrol messages. (The
distinction between these terms is theirs, not mine. See Oatley (1992Ch. 3). Semantic messages have
specific addresses or referents, whereas control signals do not. Controlsignals propagate in parallel
throughout the mind in a manner analogous to diffusion.
The theory assumes mechanisms that communicate control signals. There aretwo main aspects
to this: one is the detection of control conditions, the other is theproduction of control actions in
response to the control conditions. Some modules are concerned with theappraisal of events as being
relevant to the system's goals. These mechanisms encode controlconditions. For instance, one
mechanism might detect that a situation implies that a plan is verylikely to succeed. Each control
condition has associated with it a distinct control action. The controlactions are known as "emotion
modes". When a control condition is detected, the detecting module sendsa global broadcast
throughout the system that affects many processors and thereby triggersan emotion mode. Each
emotion mode is responsible for a distinct form of action readiness,cognitive organisation, and can
provoke "conscious preoccupation" with the event that caused it.
The most recent version of the communicative theory assumes four innatebasic types of
emotion (i.e. control dispositions consisting of pairs of control conditions andcontrol actions)
(Oatley & Johnson-Laird, to appear). These four emotions can beactualised without the agent
knowing their cause or without them having a particular object. Theauthors claim that the emotion
modes involve a broadcast of control signals that are devoid of semanticcontent. These basic
emotions are as follows.
1. Happiness is generated when subgoals are being achieved. The control action is tocontinue the
plan in question, modifying it if necessary.
2. Sadness occurs when there is a failure of a major plan toward a goal, or anactive goal needs to
be abandoned. The control action here leads to a search for a newplan.
3. Fear occurs when a self-preservation goal is threatened, or when there is agoal conflict. The
control action is to arrest the current plan, pay attention, freeze,and/or flee.
4. Anger occurs at the juncture where an active plan meets with someinterference. (This overly-
broad assumption is criticised in the next subsection.) The controlaction is to work harder, and/or
Besides these four basic emotions, there are five other innate controlstates that necessarily
involve a semantic object: attachment, parental love, sexual attraction,disgust, and rejection.
The communicative theory supposes that these control states usuallyinhibit each other but
occasionally can be active in parallel. Complex evaluations of asituation can lead to the simultaneous
activation of control states. "With loss of a loved one [one] may feelboth sad at the loss and angry at
those whose negligence was responsible for it" (Oatley & Johnson-Laird,to appear). In this
circumstance, both happiness control signals and anger control signalsare propagated in parallel
throughout the mind. Thus, different action tendencies will beelicited.
The theory can be evaluated in relation to three different questions.(1) Can the class of systems it
describes actually meet the requirements of autonomous agents? Or towhat extent does it? The next
question is most interesting if the first question is answeredaffirmatively. (2) Is there an empirical
correspondence between the described system and what happens in virtual orphysical machines
within the human mind? (3) To what extent does the account map ontofolk theory? Like Sloman
(1988), Oatley and Johnson-Laird assume that emotional termsimplicitly refer to internal mental
states and processes. Unlike Sloman and Beaudoin, however, Oatley andJohnson-Laird are very
interested in providing a theory that maps onto folk psychology. This iswhy they unabashedly use
the terms they do. Of course, such a mapping will never be perfect,because there is so much
variability (and continual evolution that is partly based on scientifictheories) in usage of intentional
idioms. Whether or not the theory gives an adequate account of"emotions" will partly depend on this
folk psychological criterion—many accounts fail because they do notmap onto what people think
they mean by the terms. In contrast, we are content to introduce a newterm instead of "emotion" (Cf.
Chapters 3, 4, 7). It is nevertheless important to separate (2) and(3) because even if the theory fails
on the third count, it may be successful on the second. In view of this,one could replace the terms
"emotion", "happiness", etc. with technical analogues.
Oatley and Johnson-Laird (to appear) review data supporting thecommunicative theory
according to empirical criteria (2) and (3). Although they claim itdoes well on both counts, it is weak
on criterion (3). For example, their definition of "anger" does notrestrict it to frustration due to the
actions of a cognizant agent who should have known better (cf. Ortony,Clore, and Collins, 1988
Ch. 7.) From a purely empirical perspective (2) the communicativetheory is one of the best cognitive
theories of "emotions", given the variety of phenomena it encompasses.(Many of the empirical
components of the theory were not described here.) From the designstance (1), the recognition of the
requirement that an autonomous agent must be able to redirect itsattention when faced with
significant junctures in plans is important. Whether this always requires global signalling is a
different question. The theory does provide principles that are worthinvestigating for designing an
architecture. Although these principles have at least face validity anddo seem plausible, it has not yet
been demonstrated that the theory describes a design which can meet the difficult requirements of
autonomous agency or be implemented. In particular, stronger analyticalarguments are required to
demonstrate that coherent shifts in behaviour can be achieved on thebasis of a diffusion of control
signals. A more specific design and implementation (e.g., of a nursemaid based on the theory) would
be useful in this respect. This would require that such questions as"How are the processors to be
designed?", "How many communication ports can a processor have?", "Whatspecific examples of
control messages are there?", and "Precisely how does a processor decidewhat do on the basis of
control signals?" be addressed.
A burden of explanation lies on the notions of the top level processorand the lower level
processors. However, even before proposing specific mechanisms for thesemodules, one ought to
provide a more systematic analysis of the tasks of the system that isnon-committal regarding which
modules are selected to execute the tasks or how they do so. Forexample, whereas the
communicative theory supposes that a process of "evaluation" is to beexecuted it seems that the
concept of evaluation is complex and subtle in ways not reflected by thetheory. There are different
kinds of evaluation that ought to be distinguished systematically. Forinstance, in Ch. 3 different
forms of evaluation of goals are expounded: e.g., concerning the importance, urgency, intensity, and
insistence of goals. And each of these dimensions of assessment isitself complex: there are different
kinds of importance, and different forms of urgency. Moreover, thedimensions of assessment can
have separate effects on the internal or external behaviour of an agent.Once these tasks are clarified,
it becomes possible to assign them to specific modules or interactionsbetween modules.
Finally, the focus on a small set of junctures is also worthinvestigating. It is possible that there
is a larger set of junctures to which an autonomous agent must besensitive than the communicative
theory posits. It would be useful for a handful of AI researchers whoare unaware of the
communicative theory to attempt to produce a taxonomy of plan junctures.What would they find? V.
R. Lesser, J. Pavlin, and E. H. Durfee (1989) investigate a similarissue and propose six types of
goal relationships. They suggest control actions that should be taken onthe basis of these control
conditions (junctures). These actions are solely concerned withincreasing the efficiency of
processing, whereas the communicative theory postulates a wider varietyof actions. Still, it is left to
future research to answer the above question and integrate theaforementioned theories.
When these issues have been addressed it may be possible to produce amodel of a nursemaid
which effectively processes goals in a manner that is consistent withthe design principles of the
communicative theory. In sum, the communicative theory fares wellempirically, and is likely to
generate useful design-based research.
In psychology the issue of internal resource boundedness has beenstudied in terms of limitations on
"attention". Definitions of attention differ, but most of them imply theselection (or suppression) of
information for (or from) higher order processing (Christ, 1991).Why is there a need to select some
information? This is usually (but not always) said to be because thereis one (or many) processor(s)
that has (have) "limited capacity". With regard to these generalissues, psychologists have asked
many questions, some of which were fairly misguided
, others insightful. Among the better
questions are "What are the limits on contemporaneous mentalprocessing?", "Which mental
processes go on in series, which go in parallel?" and "What is theordering of processes that are serial
in relation to each other?" In order for these questions to be answered,models need to be proposed in
which there are stages of information processing, and possibly parallelprocesses.
R. M. Shiffrin and W. Schneider (1984; 1977) provide a controversialexplanation of a body of
literature on attention.
This model is dated, but it serves to illustrate the points concisely.They
suggest that there are two qualitatively different sets of mentalprocesses: automatic and controlled.
Automatic processes are supposed to be quick, parallel, "effortless",and "uncontrollable"; and they
do not use a capacity limited short term memory. In contrast, controlledprocesses are supposed to be
slow, serial, "effortful", and largely controllable. Both processes areassumed to have their effects by
varying the degree of activation of memory structures in a short termstore. They argue that tasks can
become automatic if they involve a consistent mapping between stimuliand responses, whereas if the
mapping is variable then control is needed for successful execution.They explain Stroop interference
(Stroop, 1935) by supposing that both colour identification (orspatial judgements) and reading are
automatic processes that vary in speed. Hence, in order for the correctresponse to be given on
incompatible trials, the faster, incompatible automatic process needs tobe inhibited, which is
something that requires "attention".
For instance Allport (1989) demonstrates that the huge debate onwhether "selection of information for
attentional processing is early or late" is based on a conceptualmuddle.
This kind of theory is appealing because it seems parsimoniously to maponto a distinction
which is familiar in folk psychology. Most of us believe that there arethings which we do
"automatically" and things which require "attention". And, because of itssimplicity, it appeals to
scientists who value the parsimony edge of Occam's razor. Table 2.1shows how Schneider et a.
(1984) distinguish between 11 dimensions on the basis of theirtwo-fold distinction.
Some characteristics of Automatic and Control processes according to(Schneider, et al., 1984).
Characteristic Automatic processes Control processes
Central capacity Not required Required
Control Not complete Complete
Indivisibility Holistic Fragmented
Practice Results in gradual improvementHas little effect
Modification Difficult Easy
Seriality dependence Parallel Independent Serial Dependent
Storage in LTM Little or none Large amounts
Performance level High Low, except when task is simple
Simplicity Irrelevant Irrelevant
Awareness Low High
Attention Not strictly required Required
Effort Minimal Great
However, there are important criticisms to be made against the model,not the least of which is
that it buys parsimony at the cost of blurring important distinctions.Three comments are in order.
The first concerns the complexity of attention. Although it is temptingto reduce mental
phenomena to two distinct categories of dimensions, the reality of thesituation is much more
complex. Conceptual analysis reveals attention and automaticity to bepolymorphous concepts
(White, 1964), i.e., concepts that are multiply instantiated by different activities.Moreover, they have
"neighbours" in conceptual space along dimensions that are not capturedby the authors, such as the
concepts of attending, noticing, realising, desiring and intending(White, 1964), and various failures
thereof (Austin, 1968).
Even if sense could be made of attention in terms of control andautomatic processes, J. D.
Cohen, K. Dunbar (1990), and J. L. McClelland and G. D. Logan(1989) show that these processes
are not as empirically distinct as is supposed. For instance, Loganreports that subjects in the Stroop
paradigm who are informed of the probability of compatibility andincompatibility of information
cancel out Stroop effects. This and other evidence is used by Logan toconclude that automatic
processes can indeed be controlled by subjects.
The second comment concerns the supposed autonomy of control processing.A. Allport, E. A.
Styles, and S. Hsieh have recently empirically criticized this model(and (Norman & Shallice,
1986)'s theory that there is a "supervisory attentional system" (orSAS)
). They take it as a defining
feature of the controller that it is "autonomous". "An underlyingtheoretical distinction is made
between a “controlled” system, that is essentially stimulusdriven, and an autonomous system that
does not depend on stimulus triggering." (p. 8). They find that thetime to switch tasks is much
higher when the switch occurs at a memorised pre-specified juncture thanwhen a stimulus cue for the
new task is given. From these and similar data they conclude that thecontroller is also triggered into
action by sensory stimuli, and hence is not autonomous. However, onemight counter that Allport
and his colleagues are attacking a straw-man, since those who propose acontroller do not really
believe (or need not believe) that it is autonomous, only that itcontrols other parts of the system in
which it is embedded. Indeed it would be foolish to believe that thecontroller does not itself respond
to events in the world. Yet, unfortunately, there are some foundations toAllport's claim. For
instance, Schneider et al. (1984) write:
We suggest a two-part definition that is sufficient to establish thepresence of a large class of
automatic and control processes. It may be stated as follows:
1. Any process that does not use general, non-specific processingresources and does not
decrease the general, non-specific capacity available for otherprocesses is automatic.
2. Any process that demands resources in response to external stimulus inputs, regardless of
subjects' attempts to ignore the distraction, is automatic. (p. 21).
In this passage, automatic processing is distinguished from controlprocessing on the basis of its
being "stimulus driven", or prompted by stimuli. Nevertheless, even ifthis clause is removed,
Schneider's theory does not immediately crumble. However, if adistinguished expert on attention,
such as Allport is, is wrong in believing that environmental autonomy isa critical feature of
Schneider's controller or Norman and Shallice's SAS, then perhaps thisis partly because the
working of these modules is not specified clearly enough. In any case,as mentioned in the
introduction, the definition of "autonomy" varies widely according totheoretical persuasion.
The third comment concerns the specification of the control module.Perhaps the main problem
with these models is that we are given purported characteristics ofcontrol processing without being
presented with a breakdown of its components. The controller isessentially a black box. It is said to
be "unitary", but this does not make sense: it cannot perform itscomplex tasks if it does not have
components. And those components are not sufficiently obvious from thespecification that they can
One might ask "Why are the controllers so poorly specified?" Perhapsthis is because the
authors think that there are few data to go on. (Allport and colleagues claim that there are few data to
go on.) Rather than make detailed theories which are likely to befalse, they prefer more abstract and
The models of Schneider and Norman and Allport are different in manyrespects. However, they are treated
together for the remainder of this section because they both distinguishbetween a black-box like supervisory (or
control) mechanism and automatic mechanisms.
non-committal theories. However using a design-based approach allows oneto overcome this
difficulty by sketching the space of possible designs without (in theshort run) committing oneself to
any specific design as the one that is really representative of humannature.
There are other psychological theories that do break down the "controlmodule" and in this
respect they fare better than these theories of attention. Two examplesare the theories of Nico Frijda
(1987) and Julius Kuhl (1988). Indeed work in AI, to be reviewed inthe next section, does even
better in this respect.
Whereas it seems that almost all of the research on human attention hasdirectly or indirectly
tried to further our understanding of the nature of the constraints onhuman ability to process
information, very few researchers have systematically tried to answerthe question: What constraints
should there be on a person's ability to process information? That is, one canask "What purpose can
limiting processing resources serve for an agent?" Trying to explainattention in these terms does not
require that one make a "reification move" of proposing a module forattention. If attention is viewed
as selective processing then it can be viewed as an aspect of processingrather than a module.
Attempts to frame or answer analogous questions include (Allport, 1989;Boden, 1988 166-8; Heuer
& Sanders, 1987; Simon, 1967; Sloman, 1978 pp. 138 and 251-2). InCh. 4 a variant of this
question is formulated as "What should be the constraints on anautonomous agent's ability to
manage goals in parallel?" This question is not an empirical one and itcannot adequately be answered
without reference to possible designs, design options, and environmentalrequirements. In order
properly to answer that question, therefore, design options need to beproposed first. This is done in
the next section and in Chapters 5 and 6.
2.2 AI and autonomous agents
Since the inception of AI, many architectures for problem solving andaction have been produced.
(For reviews see Boden, 1987 Ch. 12; Chapman, 1987; Cohen &Feigenbaum, 1982 Ch. XV;
Georgeff, 1987). The space of possible designs is extremely large, andalthough many designs have
been produced, only the tip of the iceberg has been studied. Producingdesign taxonomies is an
important but arduous task.
Building agents that meet the requirements of autonomy has recentlybecome a more prominent
research goal of AI. The resultant systems are often called "reactiveplanners", because they are
capable of directing their behaviour on the basis of intentions thatmight be long standing or recent. In
this section, three of the main research projects in this area arereviewed. The first project is headed
by Robert Wilensky. It focuses on systems with multiple goals. Thesecond project, headed by B.
Hayes-Roth, adapts a type of architecture that is prominent in AI, namelyblackboard architectures, to
the task of autonomous agents. The third project, headed by M. Georgeff,investigates systems that
are based on procedural knowledge.
2.2.1 Wilensky on agents with multiple goals
Wilensky was one of the first people in AI to work specifically onsystems with multiple top level
goals (Wilensky, 1980). His is a distinguished contribution to AI,providing insightful requirements,
scenarios, and an architecture. He notes that human-like agents need tobe capable of generating their
own goals, and that there are many conflicts which arise in systems withmultiple goals such that the
agents need to know how to notice and resolve them. Moreover autonomousagents have to form
plans that solve many goals. (M. Pollack 1992 later referred to this as"overloading intentions".)
Wilensky proposes an architecture to meet these requirements. It has a Goal Detector
generates goals on the basis of changes in the state of the world orthrough means-ends reasoning, or
in order to solve a planning problem. (The latter alternative involves"meta-goals".) The Plan
Generator suggests candidate plans for goals, and expands them to the point wherethey can be
passed on to the executor. It has three components. (1) The Proposer: suggests possible plans. (2)
The Projector predicts the effects of executing plans, and stores them in a Hypotheses database.
Interestingly, goal detectors are sensitive not only to changes in themodel of the world, but also to
hypothetical changes in the world ensuing from plans. (3). The Revisor edits plans that might have
problematic effects (as signalled by goal detectors responding to datain the Hypotheses databases).
These can either be pre-stored plans or "fairly novel solutions"(Wilensky, 1990). The Executor
carries out plans and detects execution errors.
The Projector can detect that two goals conflict, e.g., because the effects of a hypothetical plan
to satisfy one goal interfere with another goal (which is notnecessarily a means to a common top
level goal). When the system detects a conflict, it will have to cometo a decision that involves a meta-
planning process (i.e., planning the planning). The Plan Generator has a number of meta-plans.(1)
RE-PLAN involves trying to find a plan of action in which the goals canbe satisfied without a
conflict. However, it is not always possible to find one. (2)CHANGE-CIRCUMSTANCE is a meta-
plan that involves changing the situation which led to the conflict. It isnot always possible to
eliminate a conflict between goals, so it is sometimes necessary toabandon a goal. The (3)
SIMULATE-AND-SELECT meta-plan involves simulating courses of action thatfavour one goal or
the other, and selecting between them. An attempt is made to violateboth goals as little as possible.
However, this raises an important theoretical question, "How to selectamongst future states on some
(not necessarily explicit) basis of value and certainty?" In Ch. 6 itis argued that existing theoretical
This concept has had many names and close conceptual cousins,including "monitors" (Sloman, 1978),
"motivator generators" (Sloman & Croucher, 1981), "opportunityanalyzers" (Bratman, Israel, & Pollack, 1988),
"relevance detectors" (Frijda, 1986).
principles for selecting amongst alternate actions and conflictingstates are inadequate and need to be
Although Wilensky's model relies on the ability to simulate the outcomeof a plan, it does not
have a theory for how this should be done. Moreover, it does notrepresent effects of possible actions
in a temporal manner. This implies that it cannot characterise the timecourse of the importance of the
effects of actions (hence that it cannot compute generalised urgency,described in Section 126.96.36.199).
Nowadays, it is fashionable to argue that planning itself is intractable(Agre, 1988; Agre & Chapman,
1990). Some of the sceptics appeal to the frame problem, others to therecent argument that
combinational planning is NP complete (Chapman, 1987). However, thisclaim actually is only
demonstrated for certain classes of planners. It has not beendemonstrated that it is impossible to
produce a mechanism which heuristically proposes possible behaviours andheuristically predicts
effects of these actions. Since Wilensky is not committed to anyspecific form of planning, his system
is immune to the formal arguments. It will be necessary for these detailsto be spelt out in future
work. My design also makes use of predictive abilities, but I do nothave an adequate theory about
how these abilities are realised.
Unfortunately, the requirements of autonomous agents are even morecomplicated than the ones
Wilensky's model was designed to address. For instance, goals need to begenerated and managed in
interaction with dynamic environments the temporal features of whichimpose temporal constraints on
the design. In particular, an architecture needs to be able to producegoals asynchronously to its other
mental processes, and respond to them. Hence it needs to be able (1)to store (descriptions of)
reasoning processes, (2) to interrupt these processes, (3) to resumethese processes while being
sensitive to changes that happened since they were last run (e.g., the basis for decisions and
conclusions might be invalidated). This requires a much moresophisticated set of mechanisms than
those used by contemporary computer operating systems—one cannotsimply freeze a process
descriptor at one moment and resume the next, expecting it to be stillvalid. Moreover, representation
of time and truth maintenance also need to be considered. Theserequirements are not addressed by
Wilensky's work. But one person cannot solve all of the problems. Recentdevelopments in
blackboard systems and procedural reasoning systems have looked at someof these other
2.2.2 Blackboard systems
In this section attention is directed at a class of architectures knownas blackboard systems (BBSs)—
a particular kind of rule-based system (Hayes-Roth, 1987). Systemslabelled as BBSs admit great
variety: there is probably no statement which applies to all blackboardsystems and which
distinguishes them from systems not labelled as such. (D. Corkill, 9Jun 1993, makes a similar
point.) For reviews of literature on blackboard systems the interestedreader is referred to
(Jagannathan, Dodhiawala, & Baum, 1989; Nii, 1986a; Nii, 1986b). Theapproach taken in this
section is to focus on a particular lineage of blackboard systemsproposed by B. Hayes-Roth. I start
by discussing a standard blackboard system for problem solving. Then Iexplain why this system
was not suitable for autonomous agency. Then I present a design thatimproved upon the former for
the purpose of achieving autonomous behaviour.
It is worth noting that the problems addressed by Hayes-Roth as well asthe methodology she
uses are extremely similar to those of this thesis.
188.8.131.52 A standard blackboard system
A blackboard system developed by B. Hayes-Roth, the Dynamic ControlArchitecture (DCA)
(Hayes-Roth, 1985), is worth discussing here, amongst other reasons,because (1) it is insightful to
see the additions that need to be made to them to address the particularissues of autonomous agents;
and (2) autonomous agents might have mechanisms in common with systemsthat do not have
temporal or epistemic constraints. The DCA is quite complex and thefollowing presentation is by
necessity a simplification.
The DCA has a global database (known as the blackboard), procedures(known as Knowledge
Sources), and a scheduler. Knowledge Sources have conditions ofapplicability which determine
whether on any given cycle they should be considered as candidates forexecution. The scheduler
verifies which Knowledge Sources are applicable, creates KnowledgeSource Activation Record
(KSARs) out of applicable Knowledge Sources, rates every KSAR, findsthe preferred KSAR on the
basis of the current rating preference policy, and executes it. When aKSAR executes, it records its
results on the blackboard.
Blackboard systems such as the DCA have the following features (1)they solve problems
incrementally, in that solution elements are gradually added to theblackboard, and (2) their solutions
implicate a parallel decomposition of the main task, in that multipleaspects of the problem can be
worked on in an interleaved fashion (through the sequential interleavedexecution of different
KSARs), (3) they activate solution elements "opportunistically" whenrequired; that is, Knowledge
Sources can be executed when their conditions of activation are met(unless the scheduler decides not
to choose them).
The use of non-interruptable KSARs and a global blackboard is animportant feature of DCA.
Since KSARs are not interruptable, there is no need for a KSAR to beaware of another's
intermediate computations: from the perspective of one KSAR the executionof another is
instantaneous. Thus the values of local variables of KSARs are not openfor inspection by other
processes. And this holds even if a KSAR incrementally adds values tothe blackboard.
These features of opportunistic interleaved execution of tasks can bequite useful. By design,
they allow the system to take advantage of computational opportunities,and hence potentially to use
its reasoning time effectively. Moreover, its capacity to work onmultiple tasks makes it possible to
process multiple goals.
However, DCA in itself is not suitable as a model of autonomous problemsolving. (Laffey et
al. 1988 make a related point regarding AI systems in general.)Communication between KSARs
makes use of a global database. This is tedious and can cause unwantedinteractions: direct
communication between KSARs is sometimes preferable. Moreover, if KSARsand the blackboard
need to be implemented on a distributed architecture, then it willsometimes be faster for two adjacent
KSARs to communicate directly than via a blackboard. The DCA is slow andnot highly
interruptable. This is partly because (1) the scheduler operatessequentially (and exhaustively); (2)
there is single write-access to the blackboard, implying a communicationbottle-neck; (3) KSARs
execute in a non-interruptable fashion; (4) KSARs execute one at atime; (4) the scheduler reassesses
each KSAR after the execution of every Knowledge Source. Even Hayes-Rothadmits:
Given today's hardware and operating systems, if one's goal is to buildhigh performance
application systems, the blackboard control architecture is probablyinappropriate (Hayes-Roth,
1985 p. 299).
However in Hayes-Roth's view the performance problem with DCA is notmainly the lack of
parallelism but its exhaustive scheduling. See Section 184.108.40.206. Likemany models in AI, DCA can
partly be judged on the basis of its improvableness. And in this respectit has faired well, evolving
over the years with the demands of the times. For instance, in such anarchitecture, adding facilities
for interruptability is not very difficult since by nature the focus ofreasoning shifts dynamically. And
there are many ways in which the standard architecture can be modifiedto allow parallel processing.
(See Corkill, 1989, for a cogent exploration of the design options inthe parallelisation of blackboard
models.) DCA's scheduler has provisions for ordering KSARs on the basisof ratings; therefore, it is
easy to modify it to include such values as urgency and importanceratings. Indeed, the model in the
next section did this. Therefore, blackboard systems have not onlybenefited from improvements in
hardware but also from design changes. This brings us to an expositionof a blackboard system
designed specifically for requirements of autonomous agents.
220.127.116.11 AIS: A blackboard model of autonomous agents
B. Hayes-Roth has developed an "Adaptive Intelligent System" (AIS)
which is one of the most
advanced autonomous agent architectures in the AI literature(Hayes-Roth, 1990; Hayes-Roth, 1992;
Hayes-Roth, 1993; Hayes-Roth, Lalanda, Morignot, Pfleger, & Balabanovic,1993). AIS has three
modules executing in parallel, each one of which has an input-outputbuffer for communication with
the others. The perception module takes in information from the environment, abstracts, filters,and
annotates it. The action module receives commands for actions (in its input buffer) and"translates" it
into sequences of commands. The cognition module performs all of the high level reasoning. It is the
cognitive system that uses a blackboard architecture (adapted from theDCA).
The cognitive system has four main procedures which are cyclicallyexecuted. (1) A dynamic
control planner edits a blackboard entry known as the "control plan" which containsdecisions about
the kinds of reasoning and domain tasks to perform, and how and when todo so. (2) An agenda
manager, identifies and rates applicable KSARs. (3) A scheduler selects KSARs for execution simply
based on their ratings and scheduling rules (e.g., most important first). (4) An executor simply runs
the executable KSAR.
A few features distinguish AIS from its ancestors. A major differencelies in its sensitivity to
temporal constraints. Its agenda manager (which prioritises KSARs)follows a "satisficing cycle"
rather than an exhaustive one. That is, it keeps prioritising the KSARsuntil a termination condition is
met. This condition is not necessarily that all KSARs have beenevaluated, but can be that a certain
amount of time has elapsed. When the condition has been met, thescheduler then selects the best
candidate KSAR for execution, and passes it to the executor. This is ananytime algorithm (cf.
Section 1.2). Moreover, it is capable of reflex behaviour.Perception-action arcs are not mediated by
the cognitive system. Furthermore the control planner makes plans thatare adjusted as a function of
deadlines, which are computed dynamically. Guardian, an implementationof AIS (Hayes-Roth, et
al., 1992), uses a novel anytime algorithm for responding to externalproblems (Ash, Gold, Seiver,
& Hayes-Roth, 1992). This algorithm hierarchically refines its theoryabout the nature of a problem,
i.e. its diagnosis. At any time in the process, it can suggest an actionbased on the current diagnosis.
The system delays its response until the best diagnosis is produced, oruntil something indicates that
an immediate response is necessary. (There is a general problem in thecontrol of anytime algorithms
concerning when a response should be demanded.) This algorithm could beused by other
architectures (e.g., by PRS, which is described in Section 2.2.3).
Hayes-Roth doesn't provide a consistent name for this design. Sinceshe sometimes refers to it as an "Adaptive
Intelligent System", that's what it is called here. A partialimplementation of the design is called "Guardian"
(Hayes-Roth, Washington, Ash, Hewett, Collinot, Vina, et al.,1992).
18.104.22.168.1 Assessment of AIS
Although there are ways to improve AIS, it is on the whole an advance onthe state of the art in
blackboard systems. It is worth noting that although AIS is not proposedas a model of human
cognitive architectures, it fares well in its ability to meet therequirements of autonomous agents.
Again, this is probably because of the tendency in the psychologicalliterature to favour non-
committal models over false ones. Moreover, AIS has been implemented.
There are some features of the design that could be improved upon.
• One could argue that greater efficiency can be obtained by increasingAIS's macro-parallelism.
More specifically, one could improve its responsiveness if the cognitivesystem had multiple
KSARs executing in parallel. The blackboard, which is identified as themajor bottleneck of
blackboard systems, in AIS is still a single write data structure.However, B. Hayes-Roth (1990)
argues against the parallelisation of the cognitive system. She says:
Although we have considered distributing cognitive tasks among parallelprocesses [...], our
experience with Guardian suggests that cognitive tasks have manyimportant interactions,
including sequential constraints, and associated needs forcommunication. Operating on a single
processor in the context of a single global data structure supportsthese interactions, so we
would favour distribution of cognitive tasks only in a shared-memoryarchitecture (p. 121)
B. Hayes-Roth's claim that cognitive tasks should not be parallelisedbecause of their "important
interactions" is unjustified: she does not provide arguments for it inthe cited publications. Decker et
al. (1991) and R. Bisiani and A. Forin (1989) have reported successfulparallelisation of blackboard
systems. Moreover, it appears that whereas some tasks have importantinteractions, others do not:
e.g., there might be no important interaction between playing chess andverbally describing a
previous event. It is easier to demonstrate lack of interaction in somewell defined domains than in
very abstract terms. However, the theoretically important issue oflimits on cognitive parallelism is
moot and deferred to Ch. 4.
• The modularity assumption that motor, sensory, and cognitive systems donot overlap, although
popular, seems to be inconsistent with human cognition. For example,there is clear evidence that
visual information does not merely output its results to a cognitivesystem, it is also part of a
posture control mechanism involving effectors as well (Lee and Lishman,1975). Moreover, for
many purposes overlapping systems are more useful. Sloman (1989)discusses many examples
• Sensitivity to temporal constraints is a critical requirement of thesystem. Although D. Ash, G.
Gold, A. Seiver, and B. Hayes-Roth (1992) have presented an anytimealgorithm, their notion of
deadlines is too simple. A more general notion of urgency is requiredthat considers graded
"deadlines". (See Ch. 3.) Incorporating this might not require anarchitectural change.
• Although the blackboard data structures are quite rich—particularlythe KSARs and the decisions
(Hayes-Roth, 1985)—there are some important types of informationabout tasks ( i.e., goals) that
are not represented. For instance, there is no way to express that atask has been rejected. Also, it
is not possible to express that the acceptance (not the scheduling) ofa task or decision is
conditional upon some proposition being true. For example, such an agentcould not express "I'll
only try to solve the problem of fixing this valve if I manage to solvethe problem of fixing the
two gauges." (It could probably do this if the tasks were generatedsynchronously as subgoals of
a task, rather than asynchronously to planning on the basis ofperceptual information.)
• The provisions for preventing the distraction of the agenda manager aremeagre. This is especially
problematic since the agenda manager follows a "satisficing cycle". Ifthe number of KSARs is
very high, then it could happen that important/urgent KSARs do not evenget considered because
the satisficing cycle terminates before they are rated. Hayes-Roth mightrespond that this is a price
that needs to be paid in order to obtain quick responses. However,whereas it is true that the
requirements preclude an optimal agenda manager, it would neverthelessbe possible to decrease
the risk by using additional attentional mechanisms, such as one whichrates and orders the
KSARs asynchronously to the rest of the cognitive operations, or thatproduces and uses heuristic
measures of the ratings (e.g., "insistence" (Sloman & Croucher, 1981), as discussed below).
However, this is not to say that the architecture cannot be changed toallow for these
improvements. Moreover, regardless of whether the architecture can beimproved, it serves as a
reference point in design space for autonomous agents. NML1 improves onsome of AIS's
shortcomings, although unfortunately it does not match AIS in everyrespect.
2.2.3 Procedural reasoning systems
M. Georgeff and his colleagues have developed a system to meet therequirements of
autonomous agents. These researchers were particularly concerned withthe system being
able to change its intentions and goals rapidly as events unfold. Thecommunicative
theory has similar aims. It is worth discussing PRS in detail herebecause NML1 is based
PRS is based on procedures (Georgeff & Lansky, 1986). Procedures are
essentially plans, or instructions denoting sequences of actions thatachieve a goal state or
lead to some other state if the procedures fail or are aborted.Procedures have applicability
conditions, which are expressions in a temporal logic. They can beexecuted when their
conditions are met. They can be invoked either as subroutines or as toplevel responses to
world or self knowledge. That is, their applicability conditions can beunified with goal
expressions or beliefs. Procedures' instructions are either goalexpressions or primitive
actions. Procedures are executed by an interpreter that either causesthe performance of
the primitive action (if the next instruction is a primitive action) orpushes the goal on a
goal stack and later selects a procedure whose conditions ofapplicability unifies with the
goal. If many procedures are applicable to a goal then a meta-procedureis invoked to
select amongst them. PRS goals can be complex temporal instructions, and they may
include rich control structures such as conditionals, iterators, andrecursive calls.
Procedures differ from KSARs (and productions) in many ways: e.g., some
procedures are required to be active for long periods of time. Moreover,their execution
can be interleaved with the interpreter's other activities, includingthe execution of other
procedures, whereas KSARs execute uninterruptedly and typically duringshort periods
Georgeff provides many justifications for basing a system on procedures,as he
defines them. One purported advantage over combinational planningsystems such as that
of Wilkins (1985) is that procedures conveniently allow (quick) runtime expansion.
Moreover, Georgeff claims that procedural systems are better thanproduction systems at
encoding and executing solutions to problems:
[...] much expert knowledge is already procedural in nature [...] Insuch cases it is
highly disadvantageous to "deproceduralize" this knowledge into disjointrules or
descriptions of individual actions. To do so invariably involvesencoding the
control structure in some way. Usually this is done by linkingindividual actions
with "control conditions," whose sole purpose is to ensure that therules or actions
are executed in the correct order. This approach can be very tedious andconfusing,
destroys extendibility, and lacks any natural semantics (Georgeff &Lansky, 1986
The rule-based system is less efficient because it needs to include testsin more
rules, whereas a procedural system can make assumptions about aprocedure's context of
execution based on the previous goals that must have been satisfied.This implies that
sensing needs are relaxed in procedural systems. These are advantagesthat PRS has over
the more recent system, AIS.
A few terminological issues need to be flagged. Georgeff refers toprocedures as
"knowledge areas". But this term is misleading since it suggestssomething whose
function is primarily denotational rather than operational. He refers toactive procedures
as intentions rather than processes. In this thesis, the term "knowledgearea" is not used,
but procedures are distinguished from processes. And the term "procedureactivation
record" is used to refer to the information about an active procedure.(This is standard
computer science terminology, and also used in blackboard systems.) Theconcept of a
procedure activation record is elaborated in Ch. 5. Georgeff refers togoals as behaviours;
but this is confusing, especially since procedures are also referred toas behaviours. In
this thesis, goals are not behaviours.
22.214.171.124 The PRS architecture
Procedures cannot execute outside an architecture. Georgeff provides aPRS architecture
with a view to meeting the requirements of autonomous agents (Georgeff& Ingrand,
1989; Georgeff & Lansky, 1986; Georgeff & Lansky, 1987; Georgeff,Lansky, &
Schoppers, 1987). The PRS architecture has an internal and an externalcomponent. (See
Figure 2.1.) The external component is made of sensors, a monitor,effectors and a
command generator. The internal component has a number of modules.Procedures are
stored in a procedure library. Facts about the world or the system are either built-in or
produced either by processes (i.e., procedure activations) or the monitor and are stored in
the database. The monitor translates sensor information into database facts. Goals can
either be generated as subgoals by processes or by the user. PRS doesnot allow goals to
be triggered directly by beliefs in the database. Goals are stored inthe goals structure.
The process structure is a list of process stacks. A process stack is a stack of procedure
activation records. (Processes can be active, unadopted, orconditionally suspended.)
Each process stack occupies a particular slot of the process structure.An interpreter
selects procedures for execution and pushes procedure activation recordson, and
removes them from, the appropriate process stacks on the processstructure. The
command generator translates atomic efferent procedure instructions into commands
usable by effectors.
Figure 2.1. Georgeff's Procedural Reasoning System.
The interpreter runs PRS. It goes through the following cycle when newfacts are
asserted or new goals appear. (1) It runs through the procedurelibrary verifying for each
procedure whether it is applicable to the new facts or goals. Conditionsof applicability
are stored in the procedures and they can refer to facts in the worldand/or current goals.
If a procedure applies to a fact rather than a goal, then a new processrecord is created and
the procedure is put at the root of the process record's process stack.If only one
procedure is applicable to a given goal, this procedure is put on top ofthe process stack
that pushed the goal. If more than one procedure is applicable to agoal, the interpreter
invokes a meta-process to select amongst these procedures; if more thanone meta-process
is applicable, a meta-process will be selected to select amongstmeta-processes, and so on
recursively (compare Sloman, 1978, Ch. 6); otherwise, a new processrecord is formed
and put in the process structure. Having determined and selectedapplicable procedures,
the interpreter moves on to the next step. (2) The interpreter selectsa process for
execution. (3) The interpreter executes the selected process until arelevant asynchronous
event occurs. This processing works as follows. The interpreter readsthe next instruction
from the procedure activation record on top of the selected process'sprocess stack. (3.1)
If this instruction indicates that an atomic action is required, thenthis action is performed
(this can involve modifying goals or beliefs, or sending an instructionto the command
generator.) (3.2) Otherwise the next instruction specifies a goal; inthis case this goal is
on top of the process's goal stack. The appearance of this goal willcause the
interpreter to go to step 1.
One of the main features of PRS is that it does not need fully to expanda plan for a
goal (or in response to a fact) when the goal (or fact) arises. PRSonly needs to select a
procedure, and this procedure will have many nodes that need only betraversed (and
possibly expanded) at run time.
126.96.36.199 Assessment of PRS
PRS is a general and promising architecture. It supports shifting ofattention,
changing intentions and suspending and resuming physical and mentalprocesses. It
allows for planning and execution to be performed in an interleaved orparallel fashion.
Its use of procedures simplifies control (as noted above). The factthat procedures can be
selected in whole without being totally expanded before run-time is usefulbecause, of
course, it is often impossible before run time to have a very detailedplan (usually because
necessary information is not accessible before run time). However, oneof the problems
concerning PRS, which will be described in Ch. 6, is that, conversely,it is not possible
for it to expand a procedure's sub-goals until they are to be executed.Procedures allow
for a task level decomposition, whose virtues have been expounded byBrooks (1986a;
1991a). Unlike the vertical systems explored by Brooks, however, PRSalso allows top
down control of the vertical systems, and it allows processes to controlone another
(whereas Brooks sees "behaviours" as protected). PRS's operational anddenotational
semantics have been studied. It is integrated with a temporal logic. Ithas been used to
program a robot that was supposed to act as an astronaut's assistant,which could execute
external commands and respond to problems that it detected. Because ofthese advantages
of PRS, it was selected as a basis for the architecture presented inthis Ch. 5.
The purported advantage of PRS over combinational planning systems islost if the
latter also contain a mechanism which can produce plan templates thatcan be invoked and
readily executed. As for the purported advantage of PRS over rule-basedsystems, it
To push a goal is to place it on a goal stack.
depends on the relation between the speed with which control conditionscan be verified
and the speed with which procedures can be selected. Moreover, as Sloman(1994b)
One way to get the best of both worlds is to have a more generalrule-based system
which is used when skills are developed and then when something has togo very
fast and smoothly bypass the general mechanism by copying the relevantactions
inline into the action bodies of the invoking rules. This change increasesspeed at
the cost of control and subsequent modifiability. Some human skilldevelopment
feels exactly like that!
One problem with PRS is that it can only deal with goals for which ithas pre-
formed procedures whose applicability conditions match its goal statedirectly and which
can operate in the current state of the world (i.e., whose preconditions have been
satisfied). And it deals with these goals by selecting pre-formed(though unexpanded)
procedures. That is, for each goal which it adopts it selects whole plans (procedures)
which it expands at run time. In contrast, combinational AI planningsystems are capable
of considering combinations of operators that might achieve the goalstate ( e.g., Cohen &
Feigenbaum, 1982 Ch. XV; Fikes & Nilsson, 1971). Therefore, in complexdomains
PRS procedures may have to be very elaborate with many conditionals; orthere might
need to be many different [initial state|goal] pairs. The latterstrategy involves having so-
called "universal plans", i.e., a huge collection of plans that map situations onto actions.
(The difference between the two methods is that with universal plansthe conditionals are
verified only once (in the selection of the plan) whereas with highlyconditional
procedures boolean search occurs both in the process of selectingprocedures and as a
procedure is executing.) There is a time/space trade-off betweenperforming combinatorial
search and anticipating and storing complex (or numerous) procedures(or plans).
In defence of PRS it can be conjectured that one could extend the systemto allow it
to define meta-procedures that combine procedures into new procedures.Achieving this
might require the ability to run procedures in hypothetical mode—asWilensky's model
and NML1 assume. This might be facilitated by the fact that PRSprocesses already
encode preconditions and effects. D. E. Wilkins (1988 Ch. 12) reportsthat a project is
underway to create a hybrid system combining PRS with his combinatorialplanner,
SIPE. A mechanism might also need to be proposed for automaticallygenerating
procedures at compile time. Compare (Schoppers, 1987 section 4).
A related concern is that it is not clear how PRS can be used forlearning new
procedures—whereas production systems, such as Soar (Rosenbloom,Laird, Newell, &
McCarl, 1991) do support learning. Perhaps, the kind of learningexhibited by
Sussman's (1975) Hacker would be available to PRS, and automaticprogramming
techniques should also apply.
Georgeff is committed to represent PRS information in first order logic ina
monolithic database. Although this simplifies the interpreter's task,such a restriction is a
hindrance to efficiency and accessibility, which requires a structureddatabase, multiple
types of representation, and multiply indexed information. (See Agre,1988; Bobrow &
Winograd, 1985; Funt, 1980; Gardin & Meltzer, 1989; Sloman, 1985b)
The goal structure and the process structure do not allow theconvenient
representation of certain relations between goals. For instance, whereasit can implicitly
be expressed that one goal is a subgoal of another, it cannot be statedthat one goal serves
as a means of achieving two goals at the same time. It might be usefulto have a structured
database containing a variety of kinds of information about goals.
Although the interpreter's method of verifying whether a procedure isapplicable is
designed to be efficient because it uses pattern matching of groundliterals only, it is
inefficient in that it sequentially and exhaustively verifiesprocedures, and the pattern
elements are matched against an arbitrarily large database of beliefs andgoals. This is an
important drawback because any slow down of the interpreter decreasesthe whole
system's reactivity. The problem is linear in complexity with the numberof rules and the
size of the database. This situation can be improved by assuming thatthe applicability of
procedures is verified in parallel, that the applicability conditionsare unified with
elements of a smaller database (e.g., goals only) and that a satisficing cycle (as opposed
to an exhaustive one) is performed by the interpreter. One might alsoassume that
applicability detection for a procedure can take place over many cyclesof the interpreter,
so that more time consuming detection can take place without slowingdown the system.
In PRS goals can only be generated by procedure activation records as subgoals or
by the user as top level goals. It might be advantageous to haveasynchronous goal
generators (Sloman, 1978 Ch. 6; Sloman, 1987) that respond to certainstates of affairs
by producing a "top level" goal. (See Chapters 4 ff.). That is, it issometimes possible to
specify a priori that certain conditions should lead the system to generate certaingoals.
For instance, a system can be built such that whenever its energy levelgoes beyond a
certain point a goal to replenish its energy supply should be generated.The system's
ability to generate its own top level goals is an important feature forits "autonomy", and it
also favours modularity.
As in AIS, provisions to prevent the distraction of the PRS interpreterare minimal.
(The need for this is discussed in Ch. 4).
NML1 will improve on the last five of these limitations of PRS. AlthoughPRS is an
architecture that processes goals, it is not based on a theory of goalprocessing. This makes it difficult
to design processes for PRS. The following two chapters present a theoryof goals and goal
processing. Other strengths and weaknesses of PRS will be discussedin Ch. 5 and Ch. 6.
Each theory reviewed in this chapter contributes pieces to the jig-sawpuzzle of goal processing.
None, however, can complete the picture on its own. The strengths andweaknesses of the work
reported in the literature are summarised below along the dimensions of:the concepts of goal that are
used; the data structures and processes that are supposed; the overallarchitectures that are proposed;
and the principles of decision-making that are used. Most of thecriticism refers to the main articles
reviewed here, rather than articles mentioned along the way.
The concepts of goals that have been proposed can be assessed in termsof the amount of
relevant information they provide, the rigour of the conceptualanalysis, whether they are design-
based, and whether they situate goals within a taxonomy of other controlstates. Of the main papers
reviewed here, the concept of goal is analysed most thoroughly by goalsetting theorists. That is,
these theories provide the most information about the dimensions ofvariation of goals. However,
these theories are still not sufficiently general and systematic. Forinstance, they do not take into
consideration the core qualitative components of goals. Most of theother theories are goal based but
do not give much information about goals. The PRS model stands out fromthe rest in providing a
syntax for goal expressions and an interpreter for goal expressionswhich can cope with sophisticated
control constructs. This is useful for designing agents. And the presentthesis uses the notation and a
similar interpreter. None of the theories reviewed here situate goalconcepts in relation to a taxonomy
of control states; this is done in Ch. 3 and in Boden (1972), whichanalyses the work of McDougall.
(See also Emmons, 1989; Ortony, 1988; Sloman, 1992b). A theory ofgoals is required that fares
well according to all of these criteria.
A small set of data structures and control processes and processors isposited by most theories.
Most architectures suppose the use of explicit goals, although AIS doesnot (it has "tasks" which are
similar). AIS and PRS have specific structures that act as a substratefor process types, namely
KSARs and procedures. PRS procedures have fewer fields than KSARs andthey can execute for
longer periods of time. Procedures can serve as plans in their ownright, whereas KSARs usually
must be strung together to act as plans (within the unique controlplan). These two systems offer the
most proven control methods of the papers reviewed, and both could serveas a basis for the
nursemaid. Oatley and Johnson-Laird's system supposes a form of controlbased on non-semantic
messages, but it is not yet clear how well that will fare.
The theories reviewed here do not provide a wide variety of goalprocesses. But see
(Heckhausen & Kuhl, 1985; Kuhl, 1986; Kuhl, 1992), which describehow goals can be
transformed from wishes, to wants, and intentions, and lead to goalsatisfying behaviour. A rich
process specification along these lines is given in Ch. 4. A systemsuch as PRS is particularly
suitable as a substrate for the execution of such processes. Georgeffdoes not propose a theory
determining which goal processes should take place, he merely proposesmechanisms for selecting
and executing processes.
No theory is yet up to the task of specifying both the broad variety ofgoal processes nor
sufficiently detailed principles which should guide decision-making. Thedecision-making rules
provided by goal theory and Kuhl are perhaps the most specific. However,they do not provide a
sufficiently broad context: i.e., they specify transition rules for specific goals without considering
interactions with other goals, e.g., that the achievement of one goal can be traded-off with another.
This is a problem that stands out throughout the current thesis. Seeespecially Ch. 6.
The overall architecture of all reviewed designs is at least slightlyhierarchical. (For non
hierarchical systems see, e.g., Brooks, 1990; Minsky, 1986). The most deeply hierarchical models
are those of the communicative theory and AIS. They all draw a sharpdistinction between some sort
of higher order processor and lower level processors. Oatley andJohnson-Laird and Sloman (1978
Ch. 10) go so far as to equate the higher order process withconsciousness in a substantive sense;
Schneider and Shallice speak in terms of a higher order attentionalprocess. Stagnant debates can be
circumvented by refusing to map these control concepts onto thesecolloquial substantive terms. As
McDougall (according to Boden, 1972) remarks, the adjectival forms ofterms like consciousness are
usually sounder than the nominal forms. Only the AIS architecturesupports reflexes which can by-
pass goal based behaviour. It would be trivial to add this to PRS butnot to Wilensky's model.
Chapter 3. Conceptual analysis of goals
As the title of this thesis indicates, the concept of goal figuresprominently in the present account of
autonomous agency. It is therefore imperative to explicate the meaning ofthe term and to relate it to
other concepts. In section 3.1, a taxonomy of control states ispresented, and goals are thereby related
to other control states. In section 3.2 the concept of goal is analysed,and its dimensions and
structural attributes are presented. This results in a notion of goalsthat is richer than the one usually
presented in AI and psychology. In section 3.3, alternative conceptionsof goals are reviewed,
including Daniel Dennett's argument against mentalistic interpretationof intentional terminology.
3.1 A provisional taxonomy of control states
The current section summarises and expands Sloman's view of goals andother folk psychological
categories as control states (Sloman, 1992b; Sloman, 1993b). Therationale of the exposition is that
in order to characterise goals, it is useful to present a taxonomy ofrelated concepts in which goals
figure. Since goals are understood as a class of control states, thismeans relating them to other
Sloman views the mind as a control system. Control states aredispositions of a system to
respond to internal or external conditions with internal or externalactions. They imply the existence
of mechanisms existing at least at the level of a "virtual machine". (Avirtual machine is a level of
ontology and causation that is not physical, but is based on anotherlevel which is either a physical
machine or a virtual machine. An example of a virtual machine isPoplog's Pop-11 virtual machine
Sloman supposes that a human mind non-exclusively comprises belief- anddesire-like control
states. These states are not "total" but are actually sub-states of asystem. Belief-like control states are
relatively passive states that respond to and tend to track externalevents and states. Desire-like control
states are states that initiate, terminate, or moderate processes,typically with a view to achieving
some state. Sloman writes:
Thermostats provide a very simple illustration of the idea that a controlsystem can include
substates with different functional roles. A thermostat typically hastwo control states, one
belief-like (B1) set by the temperature sensor and one desire-like (D1), set by the control knob.
• B1 tends to be modified by changes in a feature of the environment E1 (its temperature),
using an appropriate sensor (S1), e.g. a bi-metallic strip.
• D1 tends, in combination with B1, to produce changes in E1, via an appropriate output
channel (O1)) (I've omitted the heater or cooler.) This is a particularlysimple feedback control
loop: The states (D1 and B1) both admit one-dimensional continuous variation. D1 is changed
by 'users', e.g. via a knob or slider, not shown in this loop.
Arguing whether a thermostat really has desires is silly: the point isthat it has different
coexisting substates with different functional roles, and the terms'belief-like' and 'desire-like'
are merely provisional labels for those differences, until we have abetter collection of theory-
based concepts. More complex control systems have a far greater varietyof coexisting
substates. We need to understand that variety. (Sloman, 1992b Section6)
Figure 3.1 presents a taxonomy of control states including (at the toplevel) beliefs,
imagination, motivators, moods, perturbance, and reflexes. Here followprovisional definitions for
these terms. These definitions are provisional because they need to berefined following design-based
research. In this thesis, among these states only goals are examined inmore detail.
Figure 3.1. Hierarchy of control states.
• Imagination-like control states are similar to belief-like states inboth form and content, but their
origin and function are different. They are typically used to examinethe consequences of possible
actions, but they also seem to be used for learning what when wrong in apast endeavour, finding
possible causes of events, etc.
• The term "motivator" has been used in two different ways in the literature. In the narrowway
(Beaudoin & Sloman, 1993), it is roughly equivalent to the notion ofgoal which is presented
below. In the more general way (Sloman, 1987), it encompasses a widevariety of sub states that
have in common the fact that they contain dispositions to assesssituations in a certain way— e.g.,
as good or bad, right or wrong—and that they have the disposition toproduce goals. The more
general definition is used for this thesis. As Figure 3.1 shows, themain kinds of motivators
identified in the theory are: goals, attitudes, and standards.
• A goal can be conceptualised as a representation of a possiblestate-of-affairs towards which the
agent has a motivational attitude. A motivational attitude is a kind of"propositional attitude". The
motivational attitude might be to make the state-of-affairs true, tomake it false, to make it true
faster, prevent it from becoming true, or the like. The representationhas the dispositional power
to produce action, though the disposition might be suppressed orover-ridden by other factors.
The goal concept used here is similar to other usage of the term in AIand psychology, except that
its structure (as given in section 3.2) is richer. There are two mainkinds of goals, structural goals
and purely quantitative goals. Some goals are combinations of both.
• Structural goals are goals in which the objective is not necessarily describedquantitatively. I.e.,
the objective denotes relations, predicates, states, or behaviours. Mostgoals studied in AI are
structural in this sense.
• Quantitative goals (or "reference conditions") are goals in which the objective isdescribed
quantitatively; e.g., the objective might be to elevate the room temperature to 18 degreesCelsius.
It is useful to distinguish between structural and quantitative goalsbecause the mechanisms which
deal with these kinds of goals can be different. Indeed, there is abranch of mathematics,
engineering, AI, and psychology (Powers, 1973) that have evolvedspecifically to deal with
quantitative goals: they have been labelled "control theory". However,the label is misleading
because the research it refers to does not study all kinds of controlsystems, only quantitative
ones. A thermostat can be described as a quantitative control system.Such goals have a
"reference condition" denoting the desired value of a variable thatvaries along a certain dimension
or set of dimensions. Usually, negative feedback is involved: when an"error" with respect to the
reference condition is detected, the system initiates activity whichtends to bring the controlled
quantity back to the reference condition.
• Attitudes may be defined as "dispositions, or perhaps better, predispositions tolike some things,
e.g., sweet substances, or classical music or one's children, and todislike others ( e.g., bitter
substances, or pop art or one's enemies)" (Ortony, 1988 p. 328). Manyattitudes involve intricate
collections of beliefs, motivators, likes and dislikes: e.g., the dislike of communists might be
combined with a belief that they are out to remove our freedom.
• Standards are expressions denoting what one believes ought to be the case asopposed to what
one simply wants—or would like—to be the case. Related terms areprescriptions, norms, and
ethical, social, or personal rules. If a person is cognisant that somestate, S, violates one of his
standards, then he is disposed to produce the goal to counteract S and/or condemn the agent that
brings S about.
• Perturbance is an emergent dispositional state in which an agent loses control oversome of its
management of goals. This technical definition will only make sense tothe reader by Ch. 4, once
goals and management processes have been described. All that matters forthis section is that a
difference between goals and perturbance be noted by the reader. A stateof perturbance is not a
goal, but it arises out of the processing of goals. In Ch. 7, a relationbetween perturbance and
"emotion" is discussed.
• Sloman says of certain moods that they are "persistent states with dispositional power to color
and modify a host of other states and processes. Such moods cansometimes be caused by
cognitive events with semantic content, though they need not be. [...]Similarly their control
function does not require specific semantic content, though they caninfluence cognitive processes
that do involve semantic content." (Sloman, 1992b Section 6). Asimilar view is taken in (Oatley,
1992). To be more precise, moods are temporary control states whichincrease the prominence of
some motivators while decreasing others. In particular, they affect thelikelihood that certain "goal
generators" are triggered. Moreover, moods affect the valence ofaffective evaluations, and the
likelihood of affective evaluations (perhaps by modifying thresholds ofmechanisms that trigger
evaluations). It is not yet clear whether moods as defined here areuseful, or whether they merely
emerge as side-effects of functional processes.
• A reflex is a ballistic form of behaviour that can be specified by a narrow setof rules based on
input integration and a narrow amount of internal state. There are twokinds of reflexes: simple
reflexes and fixed action patterns. A simple reflex involves one action,whereas a fixed action
pattern involves a collection of actions. Usually, at most only a smallamount of perceptual
feedback influences reflex action. This would require a definition ofaction, which is not provided
in this thesis.
• Future research may try to elucidate the concept of personality traits as higher order motivators.
This taxonomy is quite sketchy. Every definition by itself isunsatisfactory. This thesis is
concerned with goals. Given the controversies surrounding the terms"moods" and "attitudes", these
terms could be replaced by technical terms without the theory being worseoff because of it.
Providing more elaborate distinctions requires expanding thecomputational architecture that supports
the control states.
There are related taxonomies in the literature. Powers (1973) presentsa quantitative control-
theoretic account of perception and action. Power's framework has beenused by C. Carver and M.
Scheier (1982). R. A. Emmons (1989) presents a hierarchical theoryof motivation, which breaks
down the "personal strivings" of individuals into decreasingly abstractcategories. M. Boden (1972)
reviews William McDougall's theory of psychology, which involves suchcontrol states as instincts,
sentiments, and emotions. K. J. Holyoak, K. Koh, and R. E. Nisbett(1989) present a mechanistic
model of learning in which rules of various degrees of abstraction aregenerated and subject to a
selection process. C. Lamontagne (1987) presents a language fordescribing hierarchical cognitive
systems. It would be a useful experiment to use Lamontagne's languagefor expressing a hierarchy of
The foregoing taxonomy is clearly oversimplified—but it will do as asketch for this thesis,
which is mainly concerned with goals. It is left for future research toanalyse other control states in
3.1.1 Attributes of control states
In order to distinguish between classes of control state and betweeninstances of classes of control
states, one needs to know what their attributes are. Mathematically,there are two types of attributes:
dimensional and structural attributes. Dimensions are quantitative attributes. Structural attributes are
predicates, relations, and propositions. A. Sloman (1992b; 1993b)discusses some of the attributes of
control states: e.g., their duration, the indirectness of their links with behaviour, thevariety of control
states which they can effect, their degree of modifiability, whethertheir function requires specific
semantic content or not, with what states they can co-exist, thefrequency with which the state is
generated or activated, the time it takes for the state to develop, howthey are brought about, how they
are terminated, how they can be modulated, how sensitive they are to runtime events, which states
do they depend on, etc. Values on dimensions can be explained in termsof the structural attributes—
e.g., the duration of a perturbance can be explained in terms of beliefs,mechanisms for activating
goals, and the system's ability to satisfy the goals.
Formally speaking, there are many ways of distinguishing between controlstates. One method
for distinguishing amongst classes of control states involves findingwhether their typical or mean
values on one of the attributes differ. Classes rarely differ by havingnon-overlapping distributions of
attribute values. For example, personality traits are by most definitionsmore long lasting than moods.
Another method involves finding whether one type of control state has agreater variance along one of
the dimensions than another. For example, perhaps perturbance tends onlyto have very direct causal
links with behaviour, whereas goals can either have very direct links orvery indirect links with
behaviour. This is analogous to the kinds of differences detected byANOVAs in inferential statistics.
A third method is by determining that one category does not have thesame attributes as another.
A clear example of this from another domain is the case of thedifference between lines and points in
mathematics or visual perception: lines have a length and an orientationwhereas points do not have
such a dimension of variation. An example of this in terms of controlstates is that some attributes of
goals that are described below do not apply to other control states:goals differ from attitudes, moods,
and standards in that scheduling information can be attached to them. Ifone decides to be in a bad
mood at a certain time, then one is not processing the mood directly;rather, one is processing a goal
which has as its object a mood or behaviours that will induce a mood.This method of distinguishing
between control states suggests that a useful improvement should bebrought to the taxonomy
(hierarchy) of control states: it should specify an inheritancehierarchy of attributes. Such a hierarchy
would state which attributes are relevant to which class of controlstates, what the class/subclass
relations are, and (implicitly) inheritance of attributes from classto subclass. The author is not
primarily suggesting that values along the attributes should be inherited (although that too would be
possible), but that the type of attribute should be inherited as in object oriented design. As istypical in
the course of object oriented design, this would probably lead to areorganisation of the hierarchy, the
postulation of new classes, new relations amongst classes, and newattributes of classes. If nothing
else, this would allow us to provide a much more generalcharacterisation of the features of goals.
But this is left for future research.
It is important to distinguish between distinctions between types ofcontrol states ( e.g.,
comparing the category of goals with the category of prescriptions) anddistinctions between
instances of a type of control states (e.g., comparing one goal with another). This section focused on
categories. In the next section, the attributes of goals are discussed.
3.2. The conceptual structure of goals
Conceptually, goals are complex structures involving core componentswhich individuate particular
goals, and a wide variety of other components that can be associatedwith them. In this section, the
concept of goal is expounded. This can be read as the requirements ofpurposive control states. An
important caveat is in order: there is no implication that thecomponents of this conceptual structure
are to be implemented explicitly as fields in a record. A system mightbe capable of maintaining goal
information in an implicit or distributed fashion.
3.2.1 The core information of goals
As was said above, to a first approximation a goal is a "representation of a possible state of affairs
towards which the agent has a motivational attitude." This is the coreof a goal. The representation of
a state-of-affairs can be expressed propositionally (e.g., in predicate calculus), and referred to as the
"proposition" of a goal. For instance, if the nursemaid wishes torecharge babyA, it might express the
propositional aspect of this goal as
A motivational attitude determines the kind of behavioural inclinationwhich an agent has towards a
proposition. This can be to make the proposition true, to make it false,to prevent it from being true,
to keep it true, to make it true faster, or the like. In the example,the nursemaid's motivational attitude
towards this proposition is "make-true". The proposition of a goal has adenotational semantics and
can be interpreted as being true or false with respect to the subject'sbeliefs or objectively. However
the motivational attitude when applied to the proposition yields astructure which is neither true nor
false: it is an imperative, i.e., it specifies something which is to be done.
• The foregoing specification of goals has the disadvantage that everygoal only has one
motivational attitude towards one proposition: it does not allow one toexpress multiple propositions
and attitudes within a single goal concept. For instance, it does notallow one to express a goal to
"maintain q while preventing p", which contains an attitude of "maintenance" and one of
"prevention". Moreover, standard predicate calculus cannot express a"while" constraint, e.g.,
"(achieve) q WHILE (doing) p"—which is not strictly equivalent to "q and p". A temporal logic is
required which can express such propositions. Thus, a more generalnotion of goals is proposed: a
goal is a proposition containing motivational attitudes and descriptors,where the former are applied to
the latter. It is left to future research to lend more precision to thisdefinition.
Meanwhile, the PRS notion of goals is provisionally used, since it comesrelatively close to
meeting the requirements (Georgeff & Lansky, 1986). The PRS goalspecification calls propositions
"state descriptions", and it uses temporal operators to express"constraints" (which are similar to
motivational attitudes). The temporal operators "!" (make true), and"#" (keep true) are used. They
are applied to propositions in predicate calculus notation. The coreinformation of the goal to recharge
babyA without going through room 5 could be expressed as
!charged(babyA) and #(not(position(claw) = room5))
In the language of PRS, this goal is called a "temporal actiondescription". This is because it is a
specification of required behaviour. For brevity, in this thesis, suchexpressions are simply called
"descriptors"; however, the reader should not be misled into believingthat descriptors are non-
intentional statements. Of course, the interpretation of particulargoals requires a system which is
capable of selecting appropriate behaviours that apply to goaldescriptors; the interpreter described in
(Georgeff & Lansky, 1986) and summarised in Ch. 2 fulfils thatfunction.
Unfortunately, the PRS representation of goals (cf. Ch. 2) does nothave the expressive power
that is ultimately required. That would necessitate temporal operatorsthat stand not only for
achievement and preservation attitudes, but the other attitudes listedabove as well— e.g., to make a
proposition true faster. Furthermore, the interpretation of "while" and"without" in terms of the #
operator and negation is impoverished since it does not fully capturethe intervals during which the
constraints should hold. (See (Allen, 1991; Pelavin, 1991; Vere,1983) for more comprehensive
temporal logics.) However, as a provisional notation it is acceptablefor this thesis. Future research
should improve upon it.
3.2.2 Attributes of goals
Like other control states, goals have many attributes. The attributesthat are enumerated in this section
are the knowledge that an agent typically will need to generate withregard to its goals. They are
summarised in Table 3.1. Most of this knowledge refers to assessment ofgoals and decisions about
goals. Along with the enumerated attributes, other relevant goalattributes are presented below.
The conceptual structure of goals
Attribute type Attribute name
Essence: Goal descriptor
Decision: Commitment status
One fact emerges from the following analysis of goals: goals are verycomplex control states
with many subtle links to internal processes which influence externalactions in various degrees of
indirectness. It is possible that some of the historical scepticism aboutthe usefulness of the concept of
goal is due to the fact that goal features have not yet beencharacterised in enough detail and in terms
that are amenable to design-based specification. Other reasons areexplored by Boden (1972). The
author does not claim to have produced such a characterisation; but hedoes claim to have taken some
step towards it.
(1) Goals have a descriptor, as explained in the previous section. This is the essential
characteristic of goals, i.e. what makes a control state a goal. The fact that goals have conceptualor
propositional components implies that all attributes of propositionsapply to goals. Exactly which
attributes there are depends on the language used to express goals. Forexample, if predicates can
vary in degree of abstraction, then goals would differ in degree of abstraction. If the language allows
propositions to differ in degree of articulation (specificity vs. vagueness) then so will goals (Kagan,
1972; Oatley, 1992). Descriptors along with other knowledge stored inthe system implicitly indicate
the kind of achievability of a goal. Goals are achievable either in an all-or-none fashion or ina partial
(graded) fashion (Haddawy & Hanks, 1993; Ortony, Clore, &Collins, 1988 p. 44). For instance,
the goal to charge a baby is a partially achievable goal, because ababy's battery can be more or less
charged. The goal to dismiss a baby is an all-or-none goal, because itis not possible merely to satisfy
this goal partly. Even for all-or-none goals, however, it might bepossible to take actions which bring
one more or less close to satisfying it, in the sense that havingperformed some of the work toward a
goal, less work is now required to satisfy it. Hence achievability canbe relative to the end or the
means to the end.
(2) Beliefs are associated with goals. They indicate what the agent takes to be thecase about the
components of the goal's descriptor, such as whether they are true orfalse, or likely to be true or
false, along with information about the certainty of the beliefs—e.g., (Cohen, 1985). Thus far, the
theory is non-committal about how beliefs are processed or represented.Beliefs about the goal state
together with the goal descriptor determine a behavioural disposition.For example, if the descriptor
expresses an (adopted) achievement goal regarding a proposition P and P is believed to be false then
the agent should tend to make P true (other things being equal).
188.8.131.52 Assessment of goals
In order to take decisions about goals, a variety of evaluations can becomputed and associated with
goals, as follows.
(3) importance descriptors represent the costs and benefits of satisfying or failing to satisfythe
goal. The notion of importance or value is intentional and linked withcomplex cognitive machinery
for relating goals amongst themselves, and anticipating the outcomes ofactions. One cannot
understand the importance of a goal without referring to other aspectsof the agent's mental life. For
instance, one of the main functions of computing importance of goals isdetermining whether or not
the agent will adopt the goal.
In contrast with decision theory (cf. Ch. 5) here it is not assumedthat importance ultimately
should be represented by a quantitative value or vector. Often, merelynoting the consequences of not
satisfying a goal is enough to indicate its importance. For instance,someone who knows the relative
importance of saving a baby's life will find it a sufficientcharacterisation of the importance of
recharging a baby's battery that the consequence of not recharging the battery is that the babydies . In
other words, if such an agent could speak English and he were asked "Howimportant is it to
recharge this baby's battery?" he might answer "It is very important,because if you do not then the
baby will die." No further reasoning would be required, because therelative importance of a baby's
death is already known: in particular no numeric value of the importanceis required. However, if a
decision between two goals were required, then the system would comparethe consequences of
satisfying or not satisfying either goal. For some systems, this couldbe realised by storing partial
orders, such as the rules described in section 1.5. (By definitionpartial orders are not necessarily
total, and hence the system might be unable to make a principled choicebetween two goals.
Furthermore, goals can have many consequences, and that complicatesdeciding.) Further discussion
of the quantitative/qualitative issue is deferred to Ch. 6.
For partially achievable goals, it might be useful to find the value ofdifferent degrees of
achievement and non-achievement. The distinction between partialachievement and partial non-
achievement is significant with respect to the importance of goalsbecause, for instance, there might
be positive value in partially achieving some goal while there mightalso be adverse consequences of
failing to achieve a greater portion of the goal. For example, X might have the task of purchasing 10
items. If X just purchases eight of them, this might contribute to X's well being. However, this
might lead X's partner to chastise X for having bought some but not all of the required items. In
other words (and more generally), the importance of a goal includesvarious factors that are
consequences of the goal (not) being satisfied. In social motivationthere are many examples of
adverse side-effects of not satisfying a goal.
There are intrinsic and extrinsic aspects to the importance of goals. The intrinsic aspects are
directly implicated in the goal state. These are the goals which are"good in themselves", e.g.,
producing something aesthetically appealing, performing a moral action,being free from pain,
enjoying something pleasant, etc.. What matters for the present purposeis not what humans treat as
intrinsically good, but what it means to treat something asintrinsically good. To a first
approximation, something is intrinsically good if an agent is willing towork to achieve it for its own
sake, even if the agent believes that the usual consequences of thething do not hold, or even if the
agent does not value the consequences of the thing. In other words,(1) intrinsic importance is
entirely determined by the propositional content expressed in the goaland not by any causal or other
implications of that content; (2) any goal with this same content willalways have some importance,
and therefore some disposition to be adopted, no matter what else is thecase (nevertheless, relative
importance will be context dependent). None of this implies that theagent's tendency to work for the
object will not eventually weaken if its usual consequences no longerhold. This would be analogous
to "extinction" in operant conditioning terms. An ontogenetic theorywould need to allow for the
possibility that an objective started out as extrinsically important,but then was promoted to being
important in itself. Intrinsic importance or value of a goal state issometimes referred to as functional
autonomy of that state (Allport, 1961). (See also (Boden, 1972,Ch. 6, pp. 206-207) ). The idea is
that even if the motivator is ontogenetically derived from some othermotivator, it functions as if its
value is inherent. An analogous case holds for phylogenetic derivationof value.
The extrinsic importance of a goal is due to the belief that itpreserves, furthers, or prevents
some other valued state. This subsumes cases in which one goal is asubgoal of another. Extrinsic
importance can be divided into two categories: goal consequences andplan consequences. (a) Goal
consequences are the main type of extrinsically valenced consequences ofgoals. These are the
valenced consequences that follow from the satisfaction of the goalregardless of what plan the
subject uses to achieve the goal. (b) Plan consequences are valencedside-effects of the plans used to
achieve a goal. (Plans are discussed in Section 184.108.40.206.) Differentplans can have different
consequences. The agent might need to distinguish between consequencesthat follow from every
plan to satisfy a goal—i.e., inevitable consequences—and those that only follow from a subsetof
plans (i.e. peculiar consequences). Inevitable plan consequences althoughlogically distinct from goal
consequences can be treated in the same way as goal consequences(unless the agent can learn new
plans). To illustrate, it might be an inevitable consequence of theplans to achieve a goal that some
babies risk getting injured (e.g., if we assume that the nursemaid can only depopulate a room by
hurling babies over the walls that separate the rooms). Plan contingentconsequences can be used to
select amongst different plans for a goal.
The concept of importance can be illustrated with an example from thenursery. If the
nursemaid detects that a room is overpopulated, it will produce a goal(call it G) to depopulate the
room. Assume that the nursemaid treats this goal as having a little"intrinsic importance", meaning
that even if the usual extrinsically important consequences of G were guaranteed not to hold, the
nursemaid would still work for G. As a colloquial description, "the nursemaid likes to keep the
population in a room under a certain threshold". Just how important thisis to the nursemaid is really a
matter of what other things it is willing to give up in order to satisfythis goal. G also has extrinsic
"goal consequences", for by preserving G, the nursemaid decreases the likelihood of a baby
becoming a thug. In fact, no baby will turn into a thug in a room where G is preserved (i.e., a non-
overpopulated room). In turn, preventing babies from turning into thugsis important because thugs
injure babies, and thugs need to be isolated. The importance of thesefactors in turn can be described:
injuries are intrinsically bad, and require that the nursemaid put theinjured babies in the infirmary
thus using up claw and the infirmary, both of which are limitedresources. The nursemaid should be
able to identify the importance of the goal in terms of one of these"plies" of consequences, without
producing an infinite regress of reasoning about the effects of effectsof effects. In an ideal
implementation of a nursemaid, it should be easy for the user tomanipulate the agent's valuation of
The valenced factors that an agent considers should be orthogonal, or ifthere is overlap
between them the agent should recognise the overlap. Otherwise, onemight overweigh one of the
factors. The worst case of a violation of the orthogonality constraintis when two considered factors
are actually identical (though they can differ in their names). Anexample of such an error is if the
nursemaid was considering the goal to move a baby away from the ditch.It might correctly conclude
that if it did not adopt this goal then the baby would fall into theditch and die (say from the impact).
Then it might also conclude that since the baby is irretrievably in theditch its battery charge would
eventually go down to zero, and die. The error, then, would be to factorin the baby's death twice
when comparing the importance of the goal to prevent the baby fromfalling into the ditch with some
other goal. The author conjectures that this kind of error is sometimesseen in human decision
making, especially when the alternatives are complex and the relationsamongst them can be muddled
because of memory load. Another case is when one factor is actually asubset of another. Another
case is if two factors partly overlap. For example, when assessing theimportance of G, the agent
might consider the consequence C1: "if I do not satisfy G then some baby might turn into a thug",
C2: "babies might get uncomfortable because of overcrowding", and C3: "babies might get injured
by the thug". C1 and C3 are not independent, because part of the reason why it is not good toturn
babies into thugs (C1) is that this might lead to babies being injured (C3).
There are many goal relations and dimensions which are implicit orimplicated in the assessment
of importance, such as hierarchical relations amongst goals. Some ofthese relations have been
expressed in imprecise or insufficiently general terms in related work.That is corrected here. Many
goals exist in a hierarchical network of goal-subgoal relations, where asupergoal has subgoals that
are disjunctively and/or conjunctively related.
A goal that is a subgoal to some other goal can derive
importance from its supergoal. Similarly, a goal that interferes withanother can aquire negative
importance. There are two complementary pairs of reciprocal dimensionsof hierarchical goal relations
that are particularly significant. The first pair of dimensions iscriticality and breadth of goals.
Criticality is a relation between a subgoal, G
, and its supergoal, G. The smaller the number of
subgoals that are disjunctively related to G, the more critical each one of these goals is to G. In other
words, if a goal G can be solved by executing the following plan:
... or G
is one of G
, ... G
is a subgoal of G, then the criticality of G
to G is equal
to 1/N. I.e., G
is critical to G to the extent that there are few other goals besides G
that can achieve
G. A more general notion of criticality would also consider the relativecosts of the alternative goals
as well as their probability of success. With the more general notion,the criticality of G
to G would
be inversely proportional to N, inversely proportional to the ratio of the cost G
to the cost of the
other subgoals, and inversely proportional to the ratio of theprobability of success of G
probability of success of the other goals. Other things being equal, if G
is more critical to G than G
is to G then G
should inherit more of G's importance than G
does. This notion of "criticality"
allows one to treat the relation of "necessity" (Ortony, et al., 1988)as a special case of criticality: i.e.,
ultimate criticality. Ortony and colleagues claim that a subgoal isnecessary to its supergoal if it must
Oatley (1992) points out that humans often lose track of goalsubgoal relations (i.e., they have fragmentary
plans). From a design stance, this is a fact that needs to beexplained and not merely assumed to be the case. Is
this fact a necessary consequence of requirements of autonomousagency?
be achieved in order for the supergoal to be achieved. This is thespecial case of criticality of G
where N is equal to 1.
The breadth of a supergoal G is simply the reciprocal of criticality of immediate subgoals of G.
That is, the breadth of G is equal to N. Thus the breadth of G is the number of goals that can
independently satisfy G. Thus a goal is wide if it can be satisfied in many ways, and narrow ifit can
be satisfied in few ways. It appears that children often produce goalsthat are overly narrow, as
Sloman (personal communication), and J. Kagan (1972) suggest. Forinstance, in the scenario
presented in the introduction where Mary took Dicky's toy, one mightexpect that if Dicky was
offered a different instance of the same kind of toy he would not besatisfied, he would want to have
that toy back. We might say that Dicky's motive is insufficiently broad, he does not realise that other
toys (other possible subgoals) could do just as well. (Of course, thesubjectivity of motivation
complicates the matter of imputing error upon a person's desires.) Theresearcher is left with the task
of producing a cognitive developmental explanation of the increase inbreadth of goals as children get
older. (This might somehow be related to variations in abstraction ofthe goals that are expressed.)
A second pair of dimensions, this time for the conjunction operator, isproposed: sufficiency
and complexity. Whereas Ortony, Clore, and Collins (1988) seesufficiency as a categorical notion, it
can be viewed as a dimension. Sufficiency is a relation between asubgoal, G
, and its supergoal, G.
The smaller the number of subgoals that are conjunctively related to G, the more sufficient each one
of these goals is to G. In other words, if a goal G can be solved by executing the following plan:
... and G
is one of G
, ... G
is a subgoal of G, then the sufficiency of G
to G is
equal to 1/N. Thus, the categorical notion of sufficiency is a special case, where N is equal to 1.
The complexity of a supergoal G is the reciprocal of sufficiency of immediate subgoals of G.
That is, the complexity of G is equal to N. Thus the complexity of G is the number of goals that are
required to satisfy G.
(4) An agent also must be able to form beliefs about the urgency of goals. In simple cases, the
notion of urgency is the same as that of a deadline: i.e., it indicates the amount of time left before it is
too late to satisfy the goal. This is called "deadline urgency" or"terminal urgency". A more general
notion of urgency is more complex: here urgency reflects temporalinformation about the costs,
benefits, and probability of achieving the goal (Beaudoin & Sloman,1991). For instance, urgency
information might indicate that the importance of satisfying a goalincreases monotonically with time,
or that there are two junctures at which action is much less risky orcostly. Hence urgency is not
necessarily monotonic, and urgency descriptors can be used tocharacterise some opportunities. An
even more general notion of urgency is not only indexed in terms ofquantitative time, but can be
indexed by arbitrary conditions: e.g., that executing the goal to recharge a baby will be less costly
when a new and more efficient battery charger is installed. In thisexample, the juncture is a condition
denoted by a proposition, not a quantitatively determined juncture.
Urgency can either be conceived in an outcome centred or an action (oragent) centred manner.
When urgency is outcome centred, it is computed with respect to the juncture of occurrence of the
event in question (e.g., when a baby will fall into a ditch). If it is action centred it iscomputed with
respect to the juncture at which an agent behaves (e.g., the latest time at which the nursemaid can
successfully initiate movement toward the baby heading for theditch).
The achievability of the goal is also relevant to estimates of urgency.In the example of saving
the baby, one is faced with an all-or-none goal, as well ascircumstantial constraints (the cost and the
likelihood of success) depending upon the time at which the action isundertaken. If the goal itself is
partially achievable, then the extent to which it is achieved can be afunction of the time at which
action is commenced. For instance, a baby that is being assaulted mightsuffer irreversible effects the
importance of which are monotonically related to the time at whichprotective action commences.
(5) For reasons described in the following chapter, it is sometimesuseful to associate measures
of insistence with goals (Sloman, 1987). Insistence can be conceived as heuristicmeasures of the
importance and urgency of goals. Insistence will be shown to determinewhether a goal is considered
by "high level" processes. Goals that are insistent over long periods oftime are likely to be frequently
considered, and hence are said to be "prominent" during that period.
(6) It is also often useful to record the original rationale for a goal. This indicates the reason
why the goal was produced in the agent. (Like other information it isoften possible to know this
implicitly, e.g., because of the goal's position in a goal stack.) Rationale is closelylinked to the
importance of a goal. The rationale might be that the goal is a subgoalof some other goal; and/or it
might be that some motivationally relevant fact is true. For instance, anursemaid that treats keeping
babies' charge above a certain threshold as a top level goal might seethe mere fact that babyA's
charge is low as the rationale of the new goal to recharge babyA. Issuesof recording reasons for
goals can be related to the literature on dependency maintenance, e.g., (Doyle, 1979). The task of
empirically identifying an agent's top level goals is discussed in(Boden, 1972 pp. 158-198).
(7) There is a record of the goal's dynamic state such as "being considered", "consideration
deferred", "currently being managed", "plan suspended", "plan aborted".The kind of dynamic
information that is required will depend on the agent's meta-levelreasoning capabilities. An important
dimension of the dynamic state is the goal's state of activation, thisis discussed in the next chapter,
once the notions of insistence based filtering and management have beenexpounded. Many of the
problems of designing an autonomous agent arise out of the fact thatmany goals can exist
simultaneously in different states of processing, and new ones can begenerated at any time,
potentially disturbing current processing.
Attributes (3) to (7) represent assessments of goals. These measureshave a function in the
agent—they are used to make decisions about goals. As is explained inCh. 4, autonomous agents
must be able to assess not only goals, but plans and situations aswell.
220.127.116.11 Decisions about goals
This section examines the four main kinds of decision about goals.
(8) Goals acquire a commitment status (or adoption status), such as "adopted", "rejected", or
Goals that are rejected or have not been adopted usually will not beacted upon. The
likelihood of commitment to a goal should be a function of itsimportance: i.e., proportional to its
benefits, and inversely proportional to its cost. However these factorscan be completely overridden
in the context of other goals of high importance. Processes which leadto decisions are called
"deciding" processes. The process of setting the commitment status isreferred to as "deciding a
goal". An example of a commitment status is if the nursemaid decides toadopt the goal to charge
(9) A plan or set of plans for achieving the goal can be produced. This comprisesboth plans
that have been adopted (as intentions), and plans that are candidatesfor adoption (Bratman, 1990).
Plans can be partial, with details left to be filled in at executiontime, or when more information is
available. The breadth of a goal is proportional to the size of the set of possible plans fora goal. That
is, a wide goal is a goal which can be satisfied in many different ways.A record of the status of
execution of plans must be maintained, and the plan must contain areference to the goal that
motivates it (compare the two-way process-purpose index in section 6.6of (Sloman, 1978), and Ch.
(10) Scheduling decisions denote when the goal is to be executed or considered. Thus one can
distinguish between physical action scheduling decisions anddeliberation scheduling decisions,
though many scheduling decisions are mixed (e.g., to the extent that action requires deliberation).
Scheduling decisions can be expressed in terms of condition-actionpairs, such that when the
conditions are satisfied mental or physical actions should be taken. Anexample of a scheduling
decision is if the nursemaid decides "to execute the plan for the goalto recharge babyA when there is
enough room in the nursery". The execution condition is partlystructural and relative as opposed to
being expressed in terms of absolute time. Scheduling is the subject ofmuch recent research in AI
Commitment in social organisms has additional complexity that is notexamined here.
(Beck, 1992; Fox & Smith, 1984; Gomes & Beck, 1992; Prosser, 1989;Slany, Stary, & Dorn,
(11) Finally, goals can be more or less intense. Intensity is a measure of the strength of the
disposition to act on the goal, which determines how vigorously it is tobe pursued (Sloman, 1987).
Intensity is a subtle concept which as yet has not been sufficientlyexplained. Intensity is not a
descriptive measure. In particular, it is not a descriptive measure ofcurrent or past performance, nor
of the sacrifices that an agent makes to pursue the goal. Ratherintensity is a prescriptive measure
which is used by an agent to determine the extent of the goal'spropensity to drive action to satisfy it.
The word "should" here does not denote a moral imperative; instead, it has mechanistic interpretation,
in that whatever mental systems drive action will be particularlyresponsive to intensity measures.
The links between measures of intensity and action are stronger than thelinks between
measures of importance and action, and between urgency and action. Anactual design is required to
specify more precisely how intensity is computed and the precise way inwhich it directs action. Still,
it can be said that although on a statistical basis the intensity ofgoals should be highly correlated with
their importance and urgency, especially if the cost of achieving themis low, this correlation is not
perfect. Sometimes, important goals cannot be very intense, because of arecognition of the negative
impact which any behaviour to achieve it might have. Furthermore, it isa sad fact about human nature
that some goals can be evaluated as having low or even negativeimportance and yet be very intense.
A person who regretfully views himself as intemperate can usually partlybe described as having a
goal which is very intense but negatively important. Whereas obsessions,in the clinical sense,
involve insistent goals and thoughts that are not necessarily intense,compulsions involve intense
goals (Barlow, 1988). (Obsessive-compulsive disorder is described inCh. 7.) Explaining how
intensity can be controlled is a particularly important psychologicalquestion, because of the
directness of its links with behaviour.
Elaboration of the theory may try to define and explain the terms"pleasure" and "displeasure",
which possibly refer to dimensions of goals.
Most of the information about goals can be qualitative or quantitative,conditional, compound
and gradually elaborated. For instance, the commitment status of a goalmight be dependent on some
external condition: e.g., "I'll go to the party if I hear that Veronica is going". And an agentmight be
more or less committed to a goal (Hollenbeck & Klein, 1987). Moreinformation would need to be
recorded in an agent that learns. The information about goals willfurther be discussed below as the
goal processes are specified.
In Ch. 4 information about goals is elaborated in the context ofmanagement processes that
produce these assessments. Simplified examples of goals are provided inCh. 5, which contains a
scenario and a design of an autonomous agent.
3.3 Competing interpretations of goal concepts
There has long been an uneasiness with intentional concepts in general,and with the terms "goal" and
"motive" in particular. M. Boden (1972) has dealt convincingly withthe arguments of reductionists
and humanists who, for different reasons, reject the possibility of apurposive mechanistic
Readers who are comfortable with the concept of goals provided in theprevious sections are
advised to skip this section on first reading, as it merely defends theconcept in relation to the work of
Some authors note the paucity of clear definitions of goals and thediversity of relevant
definitions, e.g., (Heckhausen & Kuhl, 1985; Kagan, 1972). For instance, H.Heckhausen and J.
Kuhl (1985) write "goal is a notoriously ill-defined term inmotivation theory. We define goal as the
molar endstate whose attainment requires actions by the individualpursuing it" (1985 pp. 137-138).
Although apparently valid, there are a number of problems with thisdefinition. Firstly, the definition
does not circumscribe an intentional state, i.e., it is not written in terms of "a representation (or
proposition) whose attainment ...". Secondly, it leaves out anessential component which
distinguishes goals from representations of other states, such asbeliefs, namely a motivational
attitude toward the state. Thirdly, it leaves out an important kind ofgoal namely "interest goals"
(Ortony, et al., 1988), i.e., states which the agent cannot bring about but would like to see true(such
as wanting a certain team to win a football game, but not being able tohelp it). This can be allowed in
various ways in the goal concept used here. For instance, there could bea motivator with a "make
true" attitude, and plan information showing that there was no feasibleplan to make it true. There is a
fourth problem, which is closely related to the third: the definitionexcludes those states which an
agent wishes to be true but which do not require action on his behalfbecause someone else will
achieve them. Fifth, the definition encompasses some things which arenot goals: i.e. all of those
things which require action by an agent but which are not his goals.This faulty definition suggests
that it is not a trivial task to provide an acceptable definition ofgoals.
3.3.1 Formal theories of "belief, desire, intention" systems
Many psychology and AI papers use the term "goal" without defining it.For instance, a seminal
paper on planning does not even define the term goal, though theimplicit definition was: a predicate
or relation to be made true (Fikes & Nilsson., 1971). Nevertheless,the most formal attempts to
define goals are to be found in recent AI and philosophical literature(Cohen & Levesque, 1990; Rao
& Georgeff, 1991). Indeed, H. A. Simon (1993) and M. Pollack (1992)note the apparent "theorem
envy" of some AI researchers in recent years.
P. R. Cohen and H. J. Levesque (1990) provide a specification levelanalysis of belief-desire-
intention systems (though not a design). A formal specificationcomprises a syntax, definitions, and
axioms. The Cohen and Levesques specification is meant to provideprinciples for constraining the
relationships between a rational agent's beliefs, desires, intentions,and actions. They cite (Bratman,
1987) as providing requirements of such a specification, such as: (1)Intentions are states which an
agent normally tries to achieve (though they will not necessarilyintend to achieve all of the side-
effects of these attempts) and the agent monitors its attempts toachieve them, retrying if the attempts
fail. (2) Intentions constrain what future goals are adopted asintentions: intentions must not be
incompatible. (3) Intentions must be states which the agents believeare possible. Although the aim of
providing formal specifications is apparently laudable, an unfortunateproblem with them is that they
are usually overly restrictive. Two of the constraints of (Cohen &Levesque, 1990; Rao & Georgeff,
1991) are particularly troubling. (1) They require that goals beconsistent. However, this requirement
is too harsh for modelling agents such as human beings, because it isknown that not only can goals
be inconsistent, but so can intentions. A common experience is to havetwo different incompatible
intentions for a lunch period. (2) In order to be able to propose auniversal formula, the authors
assume that the agent knows everything about the current state of theworld. However, this
assumption violates a requirement of autonomous agents, namely that theyshould be able to cope
with incomplete and possibly erroneous or inconsistent world knowledge.
Aristotle's injunction that virtue lies in the mean between a vice ofexcess and a vice of defect is
applicable here. Theories that are too stringent outlaw knownpossibilities, whereas those that are
insufficiently clear fail to distinguish between known possibilities.Formal theories tend to be too
stringent, and psychological theories tend to be insufficiently clear.
3.3.2 Arguments against viewing goals as mental states
An old but still contemporary question about the interpretation of"goals" is whether or not they are
best characterised as internal states or external attributes of anagent. This debate dates from the early
days of behaviourism in psychology. Most AI researchers and cognitivescientists had until recently
espoused the view that intentional states (like belief and desire)usefully could be represented in
computers. See (Brachman & Levesque, 1985; Dennett, 1978 Ch. 7;Dennett, 1987 Ch. 6). This
view contradicted many of the tenets of behaviourism.
Over the last decade, however, there have been renewed criticisms ofreceived notions of
representation in general and of goals and plans in particular. Forinstance, some connectionists have
argued that the use of pointer referenced data-structures must be kept toa strict minimum (Agre, 1988
pp. 182-188). However, the author knows of no cogent argument to theeffect that no internal
representation is used. For instance, although Brooks (1991b) entitledhis paper "Intelligence without
representation", he later says that he merely rejects "traditional AIrepresentation schemes" and
representations of goals. Hence he is merely suggesting differentrepresentational schemes. Brooks
and his colleagues emphasise the importance of interaction between anagent and its environment in
determining behaviour, as if this was not obvious to everyone else. In asimilar vein, R. W. White
Dealing with the environment means carrying on a continuing transactionwhich gradually
changes one's relation to the environment. Because there is noconsummatory climax,
satisfaction has to be seen as lying in a considerable series oftransactions, in a trend of
behavior rather than a goal that is achieved. (p. 322)
These are words that one would expect to find in recent texts on so called"situated activity". (But see
(Maes, 1990b) for apostasy within this community.)
A stance needs to be taken in relation to such arguments, since they dobear on the
representation of goals. However, this thesis is not the place for asurvey of these fundamental
arguments. Instead, one of the clearest positions on these matters isdescribed and evaluated:
(Dennett, 1987). Dennett's work is chosen instead of that of AIresearchers such as (Agre, 1988;
Agre & Chapman, 1987; Brooks, 1991a), because in my opinion hisarguments are much more
sophisticated than theirs. However, his work and that of hisphilosophical sparring partners ( e.g.,
Fodor, Churchland, and Clark) are very technical and intricate. Dennetthimself characterises the
literature as follows: "the mix of contention and invention in theliterature [on propositions] [...] puts
it practically off limits to all but the hardy specialists, which isprobably just as well. Others are
encouraged to avert their gaze until we get our act together."(Dennett, 1987 p. 205). I nevertheless
succumb to the temptation of having a cursory glance at thisliterature.
The Intentional Stance contains a particular class of arguments concerning (1) theinterpretation
of intentional terminology and (2) the different ways information canbe stored, manipulated, and
used in a system. It is important to view these two classes of argumentas potentially standing
separately. G. Ryle (1956) argued that motives are a particular sortof reason for acting (based on a
kind of disposition), and neither an occurrence nor a cause of action. Dennett (1987) has developed
Dennett claims that intentional terms in general are simply used bypeople as tools to predict and
interpret behaviour on the basis of knowledge of their beliefs anddesires, and not as terms referring
to internal mental states, events, or processes. His claim is partlybased on the belief that people do
not have access to (nor, presumably, theories about) the design ofeach others minds, and hence that
lay people cannot adopt a "design stance" with respect to one another. Itis also based on analogies
between intentional terms and physical "abstracta", things that are notreal but useful for prediction
(e.g., gravity). Just as to be five foot tall is not to be in a particularinternal state, to believe that Jon is
happy is not to be in a particular state either. Yet either concept canbe used predictively.
Dennett further argues that (propositional) representations should notbe used to model
psychological mechanisms, but to model the worlds in which they shouldoperate. One of Dennett's
main justifications of this claim is that he believes thatrepresentationalist theories cannot cope with
inconsistencies in beliefs. In particular, he thinks it is difficult forthem to explain behaviour when it
breaks down, when it appears irrational. For in such cases, it oftenseems as if a person believes
things which are inconsistent. Some of Dennett's more technical argumentshave to do with
philosophical difficulties in specifying the relationship betweenintentional structures—which are in
the mind—and their referents—which may be external to the mind(Dennett, 1987). Dennett takes the
example of a calculator which though it embodies rules of mathematics,it does not refer to them or
use symbols (except in the input and output stages). He claims thatmuch mental processing might be
of "that nature".
Dennett's arguments provide a useful reminder that one should not assumethat there is no
problem in using intentional representations when designing cognitivesystems. A related but distinct
thesis, which is in some respect more general than Dennett's, is thatthe concepts of ordinary
language are often both imprecise and inconsistent and that they must beused with caution. For
instance, our concepts of personal identity and life do not permit us to decide whether tele-
transportation—the process of copying a person's molecularcomposition, destroying it, and building
a "new" one—involves killing the individual or not. However, thisdoes not imply that we cannot
benefit from progressively reformulating these terms. The reformulationscan be judged more on the
basis of practical scientific usefulness than consistency with previousterminology (compare Kuhn).
Dennett is well aware of the role of conceptual analysis; nevertheless,as is argued below, his
proposal to eradicate intentional constructs from designs of systemsseems premature.
In principle, Dennett could give up his proposal to eradicateintentional constructs from designs
while maintaining the thesis that intentional constructs can beinterpreted behaviouristically, on the
basis that they buy one predictive power, and even that they have somemeasure of "reality".
(Dennett, 1988 pp. 536-8, argues that his view is not strictlyinstrumentalist.) For, it does not follow
from the fact that behaviouristic interpretation of terms is very usefuland that it is in a sense real
("abstracta") that representationalist interpretations are empiricallyfalse, philosophically untenable, or
that they lead to poor designs: i.e. the two tenets need not be mutually exclusive.
R. S. Peters (1958) critically notes that Ryle lumps together amultifarious compilation of
concepts under the dispositional umbrella term "motive". Dennett positsan even broader category of
"intentional idioms". Dennett motivates his intentional stance not onlyas an account of beliefs,
desires, and intentions, but of folk psychology in general, includingpreferences, goals, intentions,
interests "and the other standard terms of folk psychology (Dennett,1987 p. 10). What regroups
these terms together? Previous philosophical work answered this questionby saying that they (or at
least some of them) were intentional in that they had components thatreferred to something. Dennett
does not allow himself the luxury of grouping these terms in theconventional way, yet he refers to a
category that is co-extensive with the traditional one, and it does notseem clear that he has a proper
category which encompasses them. Intentional now means "folkpsychological", which means
"useful for predicting and interpreting behaviour". But what aboutmoods, attitudes, personality
traits, and other categories classified above? Although Dennett does notprovide an analysis of these
categories, he boldly assumes that they are all to be distinguishedstrictly in terms of how they are
used to predict behaviour. Yet, conceptual analysis suggests that someof these terms are not even
"intentional" in the sense of previous philosophers. For example,currently some researchers believe
that moods have little or no semantic content but can best be understoodin terms of the control they
effect (Oatley, 1992; Sloman, 1992b).
As was suggested by Sloman (1992b) and noted above,
control states differ in the precision or extent of their semanticcontent.
Moreover, although Dennett claims that taking the intentional stancebuys one predictive power,
he does not provide us with rules to make these predictions, nor does helist this as a topic for future
It is not evident that models which use intentional constructs cannotaccount for inconsistencies
in beliefs. For instance, in a society of mind theory (Minsky, 1986),it is not impossible for two
agents to have different and incompatible beliefs and desires. It is notbecause many theories require
that beliefs or preferences be consistent that representationalist AIneeds to be committed to the
assumption of consistency. Even within a single module, preferences canbe intransitive or
inconsistent. Dennett is very familiar with work in AI. Yet he onlyconsiders a small number of
possible explanations of agent level inconsistency (Dennett, 1987 Ch. 4). He provides aninsufficient
basis for making sweeping statements about all possible designs. Forinstance, he does not do justice
to the broad thesis, developed in (Clark, 1989; Sloman, 1985b), thatit is possible to explain mental
phenomena in terms of a number of virtual machines, which use many formsof knowledge
representation, some of which can be described adequately in technicallydefined terms of belief and
However, "moods" are notoriously very difficult to define, and itis possible that the concept is peculiar to
English speaking cultures. In Québecois French, the closest termis "humeur" and it has a much narrower
extension; in that language, there are only two linguisticvariations of 'mood': good mood and bad mood.
This line of argumentation suggests that an important problem withDennett's view is that it
does not offer a very practicable methodology for cognitive scientists.Dennett believes that a lot of
our knowledge of ourselves uses intentional constructs. Yet he does notwant to allow cognitive
scientists to try to tap this knowledge (except in their statement ofthe requirements of the system).
This constraint is easy for a philosopher to obey, if he is not in thebusiness of building models; but
this is not so for a cognitive scientist. Even if the completeeradication of intentional terminology from
cognitive models were ultimately needed—and that is by no meansobvious—it does not follow that
cognitive scientists ought not gradually to try to refine and extendintentional constructs in their
models. For it is possible that this gradual refinement can lead morerapidly to good models than the
alternative which Dennett proposes. In other words, part of thedifficulty with Dennett is that he
criticises "folk psychology" writ large on the basis of its purportedinability to give accurate accounts
of mental processes. He unjustifiably assumes that the choice is betweena complete rejection of folk
psychological categories at the design level and a complete acceptanceof folk psychology at that
level. But why make such a dichotomy? Is it not possible to improve someof the categories? After
all, scientific physics has progressed by using and improving folkcategories such as space and time.
One of the most important difficulties with using folk psychologicalterms is that people use them in
different ways. However, this does not prevent a theoretician fromanalysing these concepts and then
defining the terms technically. In this thesis an illustration of thispoint is made: progress is made by
providing a technical definition of the concept "goal". This definitionis not a lexical one (Copi, 1986
p. 173); i.e., it is not meant accurately to reflect the meaning of the term "goal" asused by laymen.
In this chapter the concept of goal was expounded. A provisionalhierarchy of control states was
described. Goals are a subclass of motivators, and motivators. Thishierarchy needs to be improved,
and ways of doing this were suggested. An elaborate notion of goals waspresented. The analysis
suggests a richer concept of goal than has been previously supposed.Related work on purposive
explanations was reviewed.
In the following chapter, processes that operate on goals areexpounded.
Chapter 4. Process specification
In the present chapter, the processes that operate on goals aredescribed. A process specification
determines which state transitions are possible. This specificationbuilds upon the concept of goal
given in the previous chapters, since many processes are concerned withtaking decisions or
recording information about goals in terms of the dimensions andcomponents that were given in the
conceptual analysis. The present discussion is in terms of partialstate-transitions rather than total state
transitions. State-transitions of goals can be seen as "decisions"concerning goals, in the large sense
of decision, i.e., the result of an effective decision procedure. The decisions can be ofvarious types,
including decisions that set the fields of goals, that assess thegoals, or that manage the decision-
making process itself. Each postulated process serves a function for theagent. This does not preclude
the possibility, however, of emergent processes or locally dysfunctionalprocessing.
Rather than bluntly presenting the specification, this chapterincrementally introduces
processes. This is reflected in a succession of state-transitiondiagrams. This didactic subterfuge is
useful for explaining the justification for the theoretical postulates.Section 4.1 distinguishes between
goal generation and "management" processes, and analyses them. Section4.2 presents an
outstanding problem regarding the control of managementstate-transitions. Section 4.3 raises and
attempts to answer the question "What limitations should there be on management processing?"
Section 4.4 presents Sloman's notion of insistence filtering, which ispredicated on there being
limitations to management processing, and expands upon this notion.Section 4.5 summarises the
states in which goals can find themselves. Ch. 4 can be read asproviding requirements for an
architecture. Discussion of an architecture is deferred to Ch. 5.
4.1 Goal generation and goal management
In order to expound the difficulty of the requirements of goalprocesses, the following process
specification is given in a few stages of increasing sophistication.However, for the sake of
conciseness, many of the possible specifications of intermediatecomplexity are not mentioned.
A simple autonomous agent might process goals according to thespecification depicted in
Figure 4.1. Such an agent responds to epistemic events where it noticesproblematic situations or
opportunities by producing appropriate goals or reflex-like behaviourthat bypasses normal purposive
For example, if it perceived that a baby was dangerously close to aditch, it might
produce a goal to move the baby away from the ditch. This goal wouldthen trigger a "goal
Reflex-like behaviours can be purely cognitive or overtly behavioural,innate or acquired. Acquired reflexes are
generally called "automatic". Since this thesis is mainly concernedwith goal processing, the important
conceptual and design issues concerning automaticity are notinvestigated. See Norman & Shallice (1986) and
Uleman & Bargh (1989).
expansion" (i.e., "planning") process which determines how the system is to execute thegoal. This
planning could take the form of retrieving an existing solution (say ifthe system should happen to
have a store of plans) (Georgeff & Lansky, 1986), or it might involveconstructing a new plan in a
combinational fashion (Cohen & Feigenbaum, 1982 part IV). Combinational planning involves
considering a succession of combinations of operators until one is foundthat will satisfy the goal in
question. Once a plan has been retrieved or constructed, the agent wouldexecute it.
Act on plan
Figure 4.1. State-transitions for goals (1).
Such an agent, however, is too simplistic to meet the requirements ofautonomous agency set
out above. This is because, among other shortcomings, (1) it is notcapable of postponing
consideration of new goals; (2) it necessarily and immediately adoptsgoals that it produces; (3) it is
not capable of postponing the execution of new goals—hence new goalsmight interfere with more
important plans currently being executed; (4) it executes its plansballistically, without monitoring or
adjusting its execution (except to redirect attention to a new goal).Thus, a more sophisticated
specification is required.
A state-transition diagram along these lines is depicted in Figure 4.2.When this agent produces
goals, it does not automatically process them, but performs a"deliberation scheduling" operation
which aims to decide when to process the goal further. (A more general notion of deliberation
scheduling is presented below in terms of "meta-management".) If a morepressing processing task is
underway, or if there does not exist enough information to deal with thegoal at the time, the new goal
will not continue to interfere with current tasks; instead, itsconsideration will be postponed. (Notice
that this assumes that goal processing is resource limited. Comparesection 3.2.) If the goal is to be
considered now, the agent starts by determining whether the goal is tobe adopted or not. Thus, if the
goal is rejected the agent will have spared itself the trouble offurther processing an undesirable goal.
If the goal is adopted, the agent will find a way to satisfy it (as thesimpler agent did). But this
solution will only be executed at a convenient juncture—for the agentschedules its goals.
Execute plan now
Consider later or never
Figure 4.2. State-transitions for goals (2).
Before evaluating and improving this process specification, it is usefulto propose a taxonomy
of goal processes, including some new terminology.
• Goal generation refers to the production of new goal control states.There is no requirement (yet)
that goals be represented as data-structures. All that is required isthat the system have states that
can support the goal attributes given in the previous chapter.
• Goal activation is a process that makes the goal control state acandidate for directing management
processes (see below). It is assumed that whenever a goal is generatedit is necessarily activated.
• Goal generactivation refers to the generation of a goal, if it does notalready exist, or the activation
of that goal if it does exist.
• Goal management refers to those processes involved in taking decisionsabout goals or
management processes. The main kinds of decisions were described in theprevious chapter: i.e.,
decisions proper, expansion, and scheduling. Taking these decisions isreferred to here as the
"main function" of management processing. In order to take thesedecisions, the system needs to
be able to perform various other processes (this is referred to here asthe "auxiliary functions" of
management processes), including gathering information about theattributes of particular goals
(importance, urgency, etc. as per Ch. 3), and assessing situations.Two other functions are part
of management: the control of action and management of managementprocesses.
Assessment of goals was discussed in Section 18.104.22.168; however, theassessment of situations,
in the environment or in the agent, has not yet been discussed. Anautonomous agent should be able
to assess situations in order to select management strategies that aresuited to the occasion and control
their execution. B. Hayes-Roth (1992; 1993) presents some of therelevant situational dimensions of
autonomous action: i.e., the degree of uncertainty of the environment, constraints oneffective
actions, availability of run-time data, and availability of a model. Itis of course important to
distinguish between the objective fact of the matter (e.g., what the constraints on effective action
really are) and the agent's perception of these facts.
Another important dimension is the busyness of the situation. Objectively, busyness is the
extent of the adverse consequences of spending a certain amount of timeidling. In principle, one
could characterise busyness as a temporal (possibly qualitative)function which describes the effects
of idling for various periods of time. For example one could figure thatif one spent 1 minute idling
one might risk missing the chance to pass a message to one's friend(who is about to leave); 20
minutes idling and one would not be able to finish a letter before ameeting; and with 30 minutes
idling one would be late for a meeting. Since one can be idling eitherin processing and/or in physical
action, there may be two or three conceptions of busyness: managementbusyness, action busyness,
or unqualified busyness. However, since management involves physicalaction the distinction
between mental action and physical action is less relevant. Busyness canbe high even if no physical
action is required for one of the alternatives. For instance, one mightneed to decide quickly whether
or not go to a banquet. If one decides not to go, no action is requiredof one; otherwise, immediate
action might be necessary.
Specifying how information about busyness can be generated is no trivialmatter. An agent will
have some heuristic measures that are roughly related to objectivebusyness but which do not match it
exactly. A subject might treat busyness as a measure of the extent towhich there are important,
urgent, and adopted unsatisfied (but potentially satisfiable) goalsthat require management—and/or
action—relative to the amount of time which is required to manage thegoals. No definitive function is
provided in this thesis for busyness, but the concept is illustrated. Oneof the dimensions of
busyness, in this sense, is the number of goals that are contributing tothe busyness of the situation.
A situation can be perceived as busy because there is one very importantand urgent goal requiring
attention, or because a number of urgent and more or less important goalsrequire attention. A more
dynamic measure of busyness is the rate at which goals appear inrelation to the rate at which they can
be processed. For instance, the problems or opportunities might appearall at once, or in a rapid
Information indicating high busyness can have multifarious effects. Hereare three examples.
(1) It can lead to an increase in "filter thresholds", in order todecrease the likelihood of further
distraction and increase the likelihood of satisfaction of currentgoals. See Section 22.214.171.124. (2) It can
lead to an increased sensitivity to problematic management conditions,and thereby an increase in the
likelihood of meta-management processes being spawned. See Section 4.2.(3) It can lead the agent
to relax its criteria for successful completion of tasks and selectstrategies that render faster but
possibly less reliable results. The third state may be called one of"hastiness". Which of these
consequences follow might depend on the nature of the busynessinformation.
(Beaudoin and Sloman (1993) used the term "hastiness" to denote asimilar concept to what is
now called "busyness". A. Sloman (1994a) later remarked that the term"hastiness" is more
appropriate as a definiendum of the resulting psychological state (inwhich an agent does things
quickly without being very careful). The term "busy" has both apsychological state interpretation and
an "objective" one, and is therefore more suitable than "hastiness".Moreover, like hastiness, it is
neutral as to whether the goals involved are desirable or undesirable. Ofcourse, the definition of
busyness is technical and does not completely capture the tacitunderstanding of anglophones.)
It was said above that the control of action is a management function.That is, management
processes are involved in the initiation, modulation, and termination ofphysical actions. The
specification does allow for non-management processes to be involved incontrolling actions ( e.g.,
situation-action reflexes), though the details of this distinction areleft for future research.
The specification of Figure 4.2 denoted goal management processes. Oneof the functions was
particularly narrow. The agent was assumed to ask the question "Whenshould this goal be
processed?" This is a form of deliberation scheduling. Now that thenotion of "goal management" has
been introduced, this question can be rephrased as "When should this goalbe managed?" Answering
this question and implementing the answer is a form of"meta-management." However, meta-
management has a broader function than deliberation scheduling alone;for meta-management is
concerned with the control of management processing. Meta-management isdefined as managing
management processes (some of which might be meta-managementprocesses). That is, a meta-
management process is a process whose goal refers to a managementprocess. The following are
meta-management objectives: to decide whether to decide whether to adopta goal; to decide when to
execute a process; to decide when to decide when to execute a goal; todecide which management
process to run; to decide which management process to apply to aparticular goal; to decide whether to
decide whether to adopt a goal; etc. The notion of meta-managementprocesses leads to the discussion
of management control in the following sub-section. (Having introducedthis notion, the
"deliberation-scheduling" node in Figure 4.2 should be replaced by theterm "meta-management.)
(It is useful (but difficult) to a draw a distinction between (1)meta-management, which
involves making "deliberate" decisions about how management shouldproceed, and (2) "decisions"
that are implicit in control structures used by management processes.The second type of control
"decisions" are decisions in the very general computer science sense ofeffective decision procedure.
It is easier to make such a distinction when faced with a particulararchitecture that embodies these
4.2 The control of management processing
The process specifications depicted in the previous figures haveimportant flaws, most of which
pertain to how processing is controlled. Seven such flaws are discussedhere. (1) One problem is that
in human agents the order in which management decisions are taken isflexible and not necessarily the
same as that given in Figure 4.2. For example, goal generactivation doesnot necessarily lead to meta-
management—it might lead to any of the management processes, e.g., scheduling, expansion,
assessment, etc. Moreover, an agent might be in midst of scheduling agoal when it decides to
postpone considering it and to work on another goal instead. All this,of course, raises the question
"What determines the kind of management process that follows goalactivation?" More generally,
"What determines the kind of management process that is dedicated to agoal at any time?" There does
not appear to be a straightforward answer to these questions. The issuesinvolved here do not seem to
be addressed in the psychological literature on goal processing, whichimplicitly assumes a fixed
order of processing of goals (e.g., Bandura, 1989; Heckhausen & Kuhl, 1985; Hollenbeck & Klein,
1987; Lee, et al., 1989). The questions are considered in more detailbelow.
(2) A closely related and equally important problem is that given amanagement process, it is
not clear what determines the conclusion to which it comes. Someprinciples for deciding,
scheduling, and expanding goals were proposed in the previous chapter,where it was said that
information about importance, urgency, and instrumentality of goals(respectively) should be
gathered to make decisions. However, these principles are quiteabstract. The question arises whether
more specific principles can be proposed.
(3) Another problem with Figure 4.2 is that it does not allow for onemanagement function to
implicate another. Whereas the various functions of management processeswere described
separately, they are in fact often inextricably linked. For instance, how a goal is expanded might
depend on when it can be acted upon, as well as on how important it is;and when a goal is pursued
might affect the chances of the endeavour succeeding. Often the processof deciding whether to adopt
a goal requires planning—at least in order to assess the cost of thegoal. Therefore, executing any
particular management function might involve pursuing the others.Furthermore, there is no
requirement that a process be dedicated to one type of decision only.
(4) Similarly, the specification seems to imply a degree of serialityin decision-making that is
not necessary. The trade-offs involved in serial vs. parallel managementprocessing ought to be
investigated. Compare section 3.2 below.
(5) The specification does not illustrate interruptability ofmanagement processes nor their
termination conditions. Since management processes are to be designed asanytime algorithms (cf.
Ch. 1), there need to be provisions for determining when to interruptthem and to force them to come
to a conclusion.
(6) The figures do not accommodate many other types of managementprocess that were posited
as required: such as assessing situations and goals.
(7) Finally, there is an assumption that all management processes aregoal directed. This
assumption is subtle because goals are doubly involved. Most managementprocesses are goal
directed in the sense that they are meant to manage goals. Nevertheless,the specification allows for
some processes to process other things besides goals. The processspecification is goal directed in
another sense: every process was described as being directed toward a type of conclusion (e.g., a
scheduling decision or an assessment), as opposed to being datadirected and non-purposive. This
restriction is too narrow. It is sometimes useful to take high leveldecisions in a data-driven fashion.
Indeed, people seem to use both methods, and it is convenient for theengineer to combine them
(Lesser, et al., 1989). In the general case, if every process weregoal directed, there would be an
infinite regress and nothing could ever get done.
An improved state transition diagram is presented in Figure 4.3, whichstates that goal
activation should lead to management processes but does not specify theorder of processes, and is to
be interpreted according to the requirements mentioned in this section.Whereas this view of goal
processing is much more general than the previous one, it implies thatquite a few control issues need
to be addressed. Indeed, the difficulty of the control problems that areto be solved should be
underscored. There is both an empirical problem, in knowing whatdetermines the course of
processing in humans, and an engineering problem, in knowing what are themost promising
methods for directing management processing.
Figure 4.3. State-transitions for goals (3). This indicates that goalgeneractivation leads to a
management process without specifying the type of m-process. The moreabstract expression "goal
generactivation" is used rather than "goal generation".
4.2.1 Heuristic meta-management
B. Hayes-Roth (1985) speaks of the "control problem" which is for asystem to decide which of the
currently possible computational actions to perform next. Solving thecontrol problem is especially
important for autonomous agents, because they must reach their decisionsin good time, given the
urgency and multiplicity of their goals. Now an agent cannot at everymoment proceed in a decision
deliberate manner, surveying the space of possible management actions totake, predicting
their consequences, computing their expected "utility", and selectingthe one with the highest utility.
Even if the infinite regress implied by this manner were halted, thismanner is too time consuming
and knowledge intensive. (An exposition of decision theory is given inCh. 6.) Instead, an agent that
is capable of meta-management should only engage its meta capabilities at timely junctures where a
shift in processing is required—or at least should be considered.(Compare the discussion of demon
systems in (Genesereth, 1983).) Roughly speaking, these junctures canbe divided into two classes:
management opportunities and management problems.
Thus, there is a set of internal and external situations that can arisewhich require that the
current management processing be redirected in some way, becauseotherwise time and effort will be
wasted, an opportunity will be missed, or stagnation will ensue, etc. Inorder to make the task of
managing management tractable, it is useful for an agent to be able torecognise and respond directly
to such states. Some of the types of problems in management processingwhich an agent should be
able to detect and correct are expounded below in this section. Anautonomous agent that lacks the
ability to respond to the following situations will perform"unintelligently" under the said conditions.
Decision theory was originally developed to control external behaviour,but it has recently been applied to guide
internal processing (Boddy & Kanazawa, 1990; Dean & Boddy, 1988; Doyle,1989; Good, 1971b; Haddawy &
Hanks, 1990; Russell & Wefald, 1991). An attempt is made to designagents which make optimal choices in a
population of cases. Decision theory states that an agent aims totake control decisions which have the highest
utility in that situation. Compare Simon (1959).
Sensitivity to related situations is being examined by psychologistsunder the subject headings "self-
regulation" and "meta-cognition" (Brown, 1987; Kanfer & Stevenson,1985; Miller, 1985).
By being sensitive to certain key problems in processing (oropportunities) an autonomous
agent need not intensively monitor and analyse its managementprocessing. I.e., its meta-management
facilities need not be controlling every lower level action but needonly respond to a limited set of
conditions. When the problems or opportunities are detected,meta-management processing should be
invoked to determine whether there really is a problem, in which caseremedial responses might be
elicited. The idea is progressively to identify possible problems, andfor intensive verification and
computation to be performed only if initial screening suggests it isneeded.
Here follow five problematic conditions that should lead tometa-management. Opportunities
are not covered here.
• Oscillation between decisions. This is when over a period of time management processes
take decisions that are incompatible and that cancel or contradictprevious decisions. For instance,
faced with the choice between wearing a green tie and a red tie, aperson might select a green tie,
then change his mind and select a blue tie, and change his mind againrepeatedly. Such a situation
needs to be detected and resolved by some arbitration, which ameta-management process can
command. In order for the decisions to be implemented some control overmechanisms that
dispatch management processes needs to be exercised. This category ofprocess should subsume
cases in which physical action commences and is interrupted for somegoal only to have action for
the latter goal interrupted for some other goal which is possibly thesame as the one driving the
• Ongoing disruption by an insistent goal that has been postponed orrejected but
nevertheless keeps "reappearing". This situation corresponds to a manifest state of
perturbance. (See Ch. 7). Such disruption might interfere with themanagement of important
goals, and if it is detected then various means might be taken to dealwith this, such as analysing
the situation leading to the perturbance, satisfying the perturbinggoal, or trying to prevent it from
being reactivated. Neither of these solutions is necessarilystraightforward. For instance, an agent
who is disturbed by a motive to harm another, but who decides to rejectthis, might need to devise
some strategies to stop considering the spiteful goals. This is ameta-management objective
because the objective is produced in order to exercise control over themanagement. So called
"volitional strategies" are expounded in (Kuhl & Kraska, 1989; Mischel,1974; Mischel,
Ebbesen, & Zeiss, 1972).
Detecting both of these kinds of problematic management conditionsrequires storing records of
the goals that appear, and the decisions taken about them. Like (Oatley& Johnson-Laird, to appear),
this theory implies that a perturbance can be detected while remainingnon-analysed ( i.e., the agent
does not necessarily know the cause of the perturbance that isdetected).
• High busyness. When the busyness of a situation is high, it is particularlyimportant for
prioritisation of goals to take place, and for the management toschedule its deliberation
appropriately, deferring consideration of those goals that can wait, andconsidering the more
pressing ones. This might require detecting conflicts amongst goals, andarbitrating amongst
them. Thus the system should become more likely to respond to theappearance of a goal by
engaging a meta-management process whose objective is to decide whetherit would be best to
manage this goal now, or at a future time. If the busyness is very high,it may be necessary to
accelerate the process of meta-management and increase the bias towardpostponing goals.
Below, a notion of goal filtering is expounded and it is suggested thatfilter thresholds should
be high when the busyness is high. The effect of this is to keep thedisruptability of management low.
• Digressions. A digression occurs when a goal is scheduled for deliberation,deliberation
commences, but the agent loses sight of the fact that the deliberation waspursued as a means to
an end, rather than for itself, or the deliberation aims to achieve ahigher level of detail than is
necessary. Whether a train of management is to be considered as adigression, of course, requires
an evaluation of the extent to which it contributes to relevantdecision-making. How is this to be
• Maundering. Maundering is similar to digressing, the difference being that whenone is
maundering one is managing a goal for some length of time without everhaving properly
decided, at a meta-management level, to manage it. If an agent discoversthat it is managing goals
that are not urgent or important, but other goals are pressing, then itought to tend to postpone
consideration of the former goals.
For computational economy, heuristic ways of detecting theaforementioned problems need to
be used. I.e., one cannot usually expect a system to be able to detectevery occurrence of a problem;
and there will sometimes be "false positives". Nonetheless, often thecritical part of the work of meta-
management comes not in answering the question "When should I thinkabout this?" but in actually
realising that "perhaps I should not be thinking about this". Forexample, it might take a person a few
minutes unconsciously to realise that he might be digressing, but once he comes to ask himself "Am I
digressing?" the question usually can be quickly answered. This mightbe because in a human being
the demon for detecting digressions is not always active.
In order to be able to execute meta-management processes, a systemrequires a language in
which to express management objectives that has in its lexicon termsreferring to management
processes. Organisms which do not possess such languages, which cannotproduce such terms, or
which do not have the mechanisms to combine them, are not capable ofmeta-management.
Interesting empirical questions could be formulated along the lines of"What species are capable of
managing their own management processes?", "What are the mechanisms thata given class of
organisms has for meta-management?", "What formalisms best match theirlanguage?", "How do the
meta-management mechanisms they use develop?", "What kind of variabilityis there within the
human species?", "What pathologies of meta-management can develop?" etc.These questions might
improve upon the less precise questions concerning whether otherorganisms have a language at all,
or whether they are capable of self-reflection. The questions areparticularly relevant to researchers
interested in studying the space of possible designs (Sloman, 1984;Sloman, 1994c) and the relations
between requirement space and design space (Sloman, 1993a).
Of course, the author has not solved the control problem. Some controlconditions have been
identified, but there are many other control conditions to study—e.g., opportunities. Moreover, more
needs to be said about how to make the control decisions themselves.
4.3 Resource-boundedness of management processing.
It is usually assumed in AI that real agents have important "limits" onthe amount
of "high level"
processing in which they can engage (e.g., Simon, 1959). The expression "autonomous resource-
bounded agents" is gaining currency, as is the expression "resource-bounded"reasoning. A large
variety of implications is said to follow from the requirements ofautonomous agency. Typically, they
involve assuming the use of "heuristic" algorithms, as opposed toalgorithms that are proven to be
correct. Limits in processing play a crucial role in many theories ofaffect, e.g., (Frijda, 1986; Oatley
& Johnson-Laird, 1987; Simon, 1967; Sloman, 1987; Sloman & Croucher,1981). They are also
said to imply that an agent should to a large extent be committed to itsplans (Bratman, 1987); for by
committing itself to its plans an agent thereby reduces the amount ofprocessing it needs to do—those
possible behaviours which are incompatible with its intentions can beignored.
The issue of limits in mental resources is addressed in this thesis fortwo reasons. One is that
resource limits have implications for designing autonomousagents—including the need for an
"insistence" based goal filtering process (Sloman, 1987). See section4.4. The author is not
committed, however, to expounding the precise nature of the constraints:a general characterisation
suffices for this thesis. The other is to stimulate discussion on anissue that has not been
systematically explored from a design-based approach.
The expression "amount" is just a short-hand way of referring toconstraints on processing. In fact there are
qualitative constraints on parallelism that can't be capturedquantitatively.
Two important questions need to be asked. The first one is "What mentalprocesses can go on
simultaneously in humans?" In Ch. 2, where some literature concerningattention was reviewed, this
coarse factual psychological question was broken down. It was noted thatpsychologists tend to
assume that there are processing and memory constraints, and thatempirical research must ascertain
what those constraints are. A prominent empirical psychologist of attention,(Allport, 1989), upon
reviewing the literature on attention, which he claims is makingprecious little theoretical progress,
concludes that more research is needed on the function of attention as opposed to on where this or
that particular bottle-neck lies. This leads to our second question,which is posed from a design
stance: What limits ought or must there be on the amount of mental processing that can go on
simultaneously in an autonomous agent.
In this section, an attempt to refine and answer this vague
question is made; however, the speculative and tentative nature of thediscussion needs to be
underscored. The problems involved here are some of the most difficultones in this thesis.
In order to make the question more tractable, we will focus on aparticular kind of process,
namely management processing. (Requirements of management processes arepresented above. A
design for management processes is given in Ch. 5. ). So, if one weredesigning an agent that
embodied the processes described so far in this chapter, to what extentshould management processes
be allowed to go on in parallel? We are not concerned withmicro-parallelism here but with coarser
parallelism, where different tasks are involved. Neither are we concernedwith the distinction between
real and simulated parallelism. We are concerned with at least virtualparallelism of management
processes. This merely requires that one management process can commencebefore another finishes,
and therefore that two management processes have overlapping intervalsof execution.
If there were no constraint, then whenever a goal was generated amanagement process could
simultaneously attempt to decide whether to adopt it and if so, to whatextent it should satisfy it, how
it should proceed, and when to execute it. With no constraint, no matterhow many goals were
generated by the system, it could trigger one or more processes tomanage them, and these processes
could execute in parallel without interfering with each other (e.g., by slowing each other down or
corrupting one another's results). In the case of our nursemaid,whenever it discovered a problem it
would activate processes to deal with them. For instance if itdiscovered within a short period of time
that one baby was hungry, one was sick, and two others were fighting,the nursemaid could, say,
prioritise these problems and then simultaneously plan courses ofactions for each one of them. If
there were constraints on these processes, the nursemaid might have toignore one of the problems,
and sequentially expand goals for them.
There is a general way of expressing these issues. It uses the notion ofutility of computation
expounded in (Horvitz, 1987). Assume for the sake of the argument thattheoretically one can
A related question that is sometimes asked is: Should there be anylimit at all in mental processing?
compute probabilistic estimates of the costs and benefits of managementprocessing, which are
referred to as the "utility of computation". One could then ask how thetotal utility of computation
increases as management parallelism increases. One hypothesis is thatthe utility of computation
increases monotonically (or at least does not decrease) as the amountof management parallelism
increases. Another is that, beyond a certain threshold, as the amountincreases the total utility of
computation decreases. There are, of course, other possible relations.This framework provides us
with a convenient theoretical simplification. And it is a simplification since in practice it is usually not
possible to quantify the utility of computation. Moreover, as alreadymentioned, there are some
constraints on management processing that cannot adequately be describedin terms of a quantity of
The rest of this section reviews a number of arguments that have beenproposed in favour of
limiting management parallelism. The review is brief and more research isrequired for a definitive
solution to this analytical problem.
The first constraint that is usually mentioned, of course, is that anagent necessarily will have
limited physical resources (chiefly effectors and sensors). Some management processes requireat
some point the use of sensors or effectors. For instance, in order toascertain the urgency of dealing
with a thug a nursemaid would need to determine the population densityaround the thug—which
requires that it direct its gaze at the thug's current room. Twomanagement processes can
simultaneously make incompatible demands on a sensor (e.g., looking at one room of the nursery vs.
looking at another). This implies that one of the processes will eitherneed to do without the
information temporarily, wait for a while for the sensor to becomeavailable, or wait for the
information opportunistically to become available. One can imagine thatin some circumstances, the
best solution is to wait for the sensor to be available (e.g., because the precision of the sensor is high,
and the required information cannot be obtained by inference). Thisimplies the need to suspend a
process for a while.
Now if many waiting periods are imposed on management processes, thenthe utility of
computation might fall, to the extent that some of the suspendedprocesses are dedicated to important
and urgent tasks, since waiting might cause deadlines to be missed.Clearly, some prioritisation
mechanism is needed. And in case the prioritisation mechanism should beaffected by the sheer
number of demanding processes, it might even be necessary to preventsome processes from getting
started in case they should make demands on precious resources. Thisargument does not apply to
processes that do not require limited physical resources. But if forsome reason some severe limits are
required for internal resources (e.g., memory structures with limited access) then the number of
management processes requiring them also might need to be constrained.
This argument can be extended. A. Allport (1987) argues that onlyprocesses that make direct
demands on limited physical resources actually need to be constrained innumber. However, his
criterion excludes from consideration management processes that mightmake indirect demands on
physical resources, through "subroutines". The extension, then, is thatan important aspect of
management processes is that they might make unpredictable demands on physical resources. That is,
it might not be possible to know before a process starts whether it willneed an effector or not. For
example, a person might start evaluating the urgency of a problem anddiscover that he has to phone a
friend in order to find some relevant information. Hence one cannoteasily decide to allow two
processes to run together on the assumption that they will not makeconflicting resource demands.
This is because management processes—being fairly high level—areflexible and indeterminate and
can take a variety of "search paths", and deciding which branch to takewill depend on the situation.
(The design of management processes in Ch. 5 will illustrate thispoint.) The implication, then, is that
(at least in some architectures) it might be necessary to prevent thespawning of management
processes in case they should claim a limited physical resource andinterfere with more pressing
management processes. Thus limited physical resources (and a few otherassumptions) imply the
need for limiting management processing.
An obvious constraint is that whatever processing hardware supports the management
processes, it will necessarily be limited in speed and memory capacity,and therefore will only be able
to support a limited number of management processes simultaneously. Forexample, there will be a
finite speed of executing creating, dispatching and executing newprocesses, and given external
temporal constraints, this might imply a limit on managementparallelism. Similarly, there might not
be enough memory to generate new processes. However, one could alwaysask of a given finite
system "If it were possible to increase the speed and memory capacity ofthe system, would it be
profitable to allow it to have more management parallelism?"
A more general argument than the latter is that there might beproperties of the mechanisms—at
various virtual or physical levels—that discharge the mechanisms thatlimit the amount of parallelism
that can be exhibited. There are many possible examples of this. Oneexample that falls in this class
is, as A. Allport (1989) has argued, that an important constraint onbiological systems which use
neural networks is to avoid cross-talk between concurrent processes implemented on the same neural
network. One can suggest, therefore, that as the number of managementprocesses using overlapping
neural nets increases beyond some threshold, the amount of interferencebetween these processes
might increase, and this might adversely affect the total utility ofcomputation. However, since the
present section is concerned with design principles (rather thanbiologically contingent decisions), for
Allport's point to be weighty, it would need to be shown that in orderto meet the requirements of
autonomous agents it is necessary (or most useful) to use neuralnetworks or hardware with similar
cross-talk properties. Otherwise one could simply assume that neuralnets are not to be used. Another
set of examples of such constraints is used in concurrent blackboardsystems that face problems of
"semantic synchronisation" or the corruption of computation (Corkill,1989). See Corkill (1989) for
examples. One solution that has been proposed is temporarily to preventregions of the blackboard (or
particular blackboard items) to be read by one process during thelifetime of another process that is
using it (Corkill, 1989). This is referred to as "memory locking". Inother words, it is sometimes
useful for regions of a memory structure to be single-read—processeswanting to read information in
the region would either have to wait or redirect their processing.
Another constraint concerns the order of management processes. One might argue that some
decisions logically must precede others and hence so must the processesthat make them. For
instance, one might claim that before deciding how to satisfy a goal oneneeds to decide the goal. And
one might also need to know how important the goal is (so that themeans not be disproportionate to
the end). However, as was noted above there does not seem to be an a priori order in which
management decisions must be taken. For instance, it is often (but notalways) necessary to consider
plans for achieving a goal before deciding whether or not to adopt it.The lack of a universal order
does not imply that it is reasonable to pursue every kind of managementdecision simultaneously; nor
does it imply that no order is more appropriate than another in aparticular context. B. Hayes-Roth
and F. Hayes-Roth (1979) have argued that problem solving shouldproceed opportunistically. This
would imply that processes that can contribute to the management ofgoals in a given context should
be activated and those that cannot should not. This is fairly obvioustoo. Many reasoning systems
have procedures, methods, or knowledge sources that have conditions ofapplicability attached to
however, most of them also have mechanisms which select amongstmultiple applicable
procedures. The abstract question which we are dealing with here is "Whycouldn't all applicable
procedures run in parallel?"
It seems to be the case that the more management parallelism is allowed,the more difficult it is
to ensure the coherence of management decisions, and this in turn adversely affects the utilityof
computation. The notion of "coherence" would need to be spelt out. Itinvolves taking decisions that
are not incompatible with other decisions (in the sense thatimplementing one decision does not
reduce the likelihood of being able successfully to implement anotherdecision, or increase the cost
thereof); or that if such incompatibilities are engendered, they willbe noted. For instance, consider a
process, P1, that is meant to decide when to pursue a particular goal. If P1 is operating serially it is
easier to ensure that its output will be coherent with respect to otherdecisions if it is not running
simultaneously with another scheduling procedure. (Note that assuring"coherence" can be difficult
even without asynchronous management processes—e.g., because of the frame problems—and
A general notion of "opportunity" must cope with cases of gradedopportunity and costs and benefits of
Coherence is a particularly important criterion for managementprocessing. That parallelism
poses a problem for coherence is well known (Booker, Goldberg, &Holland, 1990). It has been said
in (Baars & Fehling, 1992; Hayes-Roth, 1990; Simon, 1967) to imply theneed for strict seriality at
some level of processing. However, one could counter that there areexistence proofs of systems that
effectively do embody "high level" coarse-grained parallelism (Bisiani& Forin, 1989; Georgeff &
It would seem, therefore, that the coherence argument needs to be madein terms of
trade-offs between deliberation scheduling policies allowing differentdegrees of parallelism, rather
than between "strict" seriality and an indefinite amount ofparallelism.
One may counter that the risk of incoherence due to parallelism is notvery severe, for there are
already two important cases of asynchrony that are required forautonomous agents and that at least in
humans are resolved in some not completely incoherent manner. One caseis between management
processing, perception and action. (This is taken for granted in thisthesis.) The other is between
management processes. That is, the system will necessarily be able (atleast part of the time) to
commence managing one goal before having completely managed another. Theargument is that if the
system has to deal with these cases of asynchrony, then it might also beable to deal with higher
degrees of management parallelism. This is an implication of the kind ofinterruptability assumed in
the requirements. Therefore, the counter to the coherence argument goes,a proper design of an
autonomous agent will need to be based on a theory, T, of how to prevent or cope with problems of
"incoherence due to management parallelism". For instance, the fact thatan agent perceives changes
in the world as it reasons implies that the basis for its decisionsmight suddenly be invalidated. This is
obvious. The counter to the coherence argument then is that it is notyet clear that T will imply a need
for severe constraints on management parallelism. It might be that quiteminor constraints are
sufficient for dealing with the various kinds of asynchrony (e.g., synchronising reads and writes,
and establishing demons that detect inconsistency). In any case, oneneeds to develop such theories
as T, and analyse their implications for management parallelism which may ormay not be severe.
A final noteworthy argument has been proposed by Dana Ballard (Sloman,1992a). In a
nutshell, the argument is that in order for an agent to make its task oflearning the consequences of its
actions computationally tractable, it should limit the number of mentalor physical actions that it
performs within a period of time. The requirement of learning theconsequences of one's actions is
assumed to be essential for autonomous agents. The complexity oflearning the consequences of
one's actions can be described as follows:
D. Dennett and M. Kinsbourne (1992) deal with philosophical issuesarising from viewing the mind as a
coarse-grained parallel processing system.
(1) An agent is designed to learn which of its actions are responsiblefor some events— i.e., to
learn the consequences of its actions. Let A be the set of the agent's actions performed in the last T
minutes and let C be the set of events which are possible consequences of elements of A.
(2) In principle an event in C might be an effect not only of one action in A, but of any subset
of the elements of A.
(3) Therefore, the complexity of the learning task is equal to thepower set of A, i.e., 2 raised
to the power A.
Since the learning function is exponential, A must be kept reasonably small. Sloman proposed
a few methods for doing this: one may abstract the features of A, group elements of A together, or
remove elements of A (e.g., by reducing T, or eliminating actions which for an a priori reason one
believes could not be implicated in the consequences whose cause onewishes to discover). Ballard's
method is to reduce the number of actions that are performed inparallel—a rather direct way of
Although Ballard's argument is not without appeal, for indeed complexityproblems need to be
taken quite seriously, it is not clear that his solution is the bestone, or even that the problem is as
severe as he suggests. Firstly, one could argue that reducing managementprocessing is too high a
price to pay for the benefit of learning the effects of management. Suchan argument would need to
expound the importance of learning, and the effectiveness of the othermethods of making it tractable.
It might suggest that abstracting the properties of the actions is moreuseful than reducing their
number. And it would also suggest that there are some management actionswhich can be ruled out as
possible causes (i.e., as members of A); compare (Gelman, 1990).
Secondly, one could argue that in most cases, causal inference is (orought to be) "theory-
driven" (or "schema driven") rather than based on statisticalco-variation, as Ballard's argument
supposes. This involves an old debate between David Hume and ImmanuelKant on the nature of
causation and the nature of causal attribution (Hume, 1977/1777; Kant,1787/1987). Hume believed
that, metaphysically, there is no such thing as causalrelations—there are only statistical relations
between events. Kant, on the other hand, believed in generativetransmission of causal potency.
Psychologically, Hume believed that "causal inference" is illusory, andbased mainly on perceptions
of covariation. Kant believed that human beings can intuit causalrelations. These two views have
been at odds in philosophy as well as psychology, and have generated alarge fascinating literature . It
appears, however, that causal inference is often based on other factorsbesides covariation. In
particular, it does not seem reasonable to assume that a causalattribution need consider (even in
principle) the power set of the actions preceding an event, as Ballard's argument (axiom 2) states.
Instead, the agent can use "causal rules" or interpretation mechanismsto postulate likely causes
(Bullock, Gelman, & Baillargeon, 1982; Doyle, 1990; Koslowski, Okagaki,Lorenz, & Umbach,
1989; Shultz, 1982; Shultz, Fischer, Pratt, & Rulf, 1986; Shultz &Kestenbaum, 1985; Weir, 1978;
White, 1989), and eliminate possible combinations thereof. However, theliterature is too voluminous
and complex to be discussed here. It suffices to say that Ballard'sargument relies on a debatable
assumption (axiom 2).
Occam's criterion of parsimony is directly relevant to the discussion of this section. One may
argue that if a system can meet the requirements with less concurrencythan another then, other things
being equal, its design is preferable. Occam's razor cuts both ways,however, and one might want to
try to demonstrate that increased parallelism is necessary or that itcan give an edge to its bearers. But
that is not an objective of this thesis.
The preceding discussion expounded analytical or engineering (asopposed to empirical)
arguments for limiting the amount of management processing in autonomousagents. This exposition
does suggest that there are reasons for limiting management parallelism,but the counter-arguments
raised do not permit one to be quite confident about this conclusion.The discussion did not specify or
determine a particular degree of parallelism that forms a thresholdbeyond which utility of reasoning
decreases. Such thresholds will undoubtedly depend on the class ofarchitectures and environments
that one is discussing. Despite the cautious conclusions, this sectionhas been useful in collecting a
set of arguments and considerations that bear on an important issue.
If we accept that there are limits in management processing in humans,and if we believe that
they are not necessary for meeting autonomous agent requirements, theymight be explained as
contingent upon early "design decisions" taken through phylogeny. (Cf.(Clark, 1989 Ch. 4) on the
importance of an evolutionary perspective for accounts of humancapabilities. R. Dawkins (1991)
argues that evolution can be seen as a designer.) The auxiliaryfunctions of management processes
(particularly those involved in predicting the consequences of possibledecisions and actions) might
be heavily dependent upon analogical reasoning mechanisms (cf. Funt,1980; Gardin & Meltzer,
1989; Sloman, 1985b) that cannot be dedicated to many independent tasksat once. Analogical
reasoning might itself use an evolutionary extension of perceptualprocesses which although powerful
are restricted in the number of concurrent tasks to which they can bededicated because of physical
limits and needs for co-ordination with effectors. Therefore managementprocesses might have
inherited the limitations of vision and analogical reasoning. However,these constraints might be
beneficial, if more verbal ("Fregean") ways of predicting would havebeen less effective. This
evolutionary argument is merely suggestive and needs to be refined.
None of the above provides specific guidelines for constrainingmanagement processes. More
research is required to meet that objective. In particular, it has notbeen shown that at most one
management process should be active at a time. Nevertheless, there doesseem to be a need for some
limits on management processes; hence, the design to be proposed in thenext chapter will assume
that there must be some restrictions, but not necessarily strictseriality.
4.4 Goal filtering
It is assumed that not all goals that are generated or activated willnecessarily be immediately
considered by management processes, but might be suppressed (filteredout). An important rationale
for goal filtering has been proposed by Sloman. In this section,Sloman's notion of filtering is
described, while particular care is taken to dispel some commonmisconceptions about it. In the next
section, some other roles which the filtering mechanism can play areproposed.
Sloman assumes that when a goal is generated (or activated) and isconsidered by a
management process this may interrupt and at least temporarily interferewith current management
process(es) and physical actions that they may be more or lessdirectly controlling. This causal
relation is supposed to follow from (a) the need for immediateattention, and (b) limits in management
processing (the rationale of which was discussed in the previoussection). This interference can have
drastic consequences. For instance, if a person is making a right turnin heavy traffic on his bicycle
and he happens to "see" a friend on the side of the road, this mightgenerate a goal to acknowledge
the friend. If this goal distracted his attention, however, it mightlead him to lose his balance and have
For such reasons, Sloman supposes a variable-threshold goal filteringmechanism that
suppresses goals that are not sufficiently important and urgent,according to some rough measure of
importance and urgency. Insistence is defined as a goal's ability topenetrate a filter. The filter
threshold is supposed to increase when the cost of interruptionincreases. Suppressing a goal does
not mean that the goal is rejected. It only means that the goal is temporarily denied access to "higher-
order" resource-limited processes.
When is goal filtering required? A. Sloman (1992b) says:
This mechanism is important only when interruption or diversion ofattention would undermine
important activities, which is not necessarily the case for allimportant tasks, for instance those
that are automatic or non-urgent. Keeping the car on the road whiledriving at speed on a
motorway is very important, but a skilled driver can do it whilethinking about what a
passenger is saying, whereas sudden arm movements could cause a crash.However, in
situations where speed and direction of travel must be closely relatedto what others are doing,
even diverting a driver's attention could be dangerous. So our theory'sfocus on diverting or
interrupting cognitive processing is different from the focus in Simonand the global signal
theory on disturbing or interrupting current actions. (Section 10)
An entire paper could be dedicated to elucidating this example andconsidering alternative explanations. The
notion of suppression of motivational tendencies has a historicalprecedent in psychoanalysis (Erdelyi &
Goldberg, 1979; Erdleyi, 1990) and is accepted by some theorists ofpain (Melzack & Wall, 1988 Ch. 8 and 9).
Colby (1963) describes a computer model of defence mechanisms. (Seealso Boden 1987, Ch. 2-3).
A subset of the cases in which preventing distraction might be importantis when a rare and important
opportunity requires attention (such as when a thief suddenly gets tosee someone typing in a
password to an expense account).
The notion of filtering calls for a new term referring to a goalattracting attention from a
management process. This is called "goal surfacing". That is, a goal issaid to "surface" when it
successfully penetrates a filtering process. If the goal isunsuccessful, it is said to be "suppressed".
Goal suppression is different from goal postponement. Goal postponementis a type of meta-
The requirement of filtering critically rests on limitations in"resources", where initiating one
mental process might interfere with some other mental process. Adetailed specification of how to
know whether and when one process will interfere with another is needed.This would require
proposing a computational architecture of goal processing. It isprobably not the case that every
design that meets the requirements of autonomous agents will be equallyvulnerable to adverse side-
effects of goal surfacing. One can imagine designs in which a system canperform complex speech
analysis while driving a car in acutely dangerous circumstances. If anarchitecture allows some
management processes to be triggered in a mode that guarantees that theywill not interfere with
others, then under circumstances where diverting a management processmight be dangerous, non-
pressing goals that appear could trigger non-interfering managementprocesses or processing by
dedicated modules (e.g., the cerebellum in humans?). Such goals would not be suppressed in a
The situations in which Sloman says filtering would be useful all havethe characteristic that
even brief interruption of management processes could have importantadverse consequences. Since
the goal filters have the purpose of protecting management processes, itis crucial that they cannot
invoke the management processes to help decide whether a goal should beallowed to be managed
(that would defeat the filters' purpose). Filters must make theirdecisions very rapidly. This is
because if the goals that are attempting to penetrate are very urgent,they might require attention
Sloman (personal communication) points out that none of this impliesthat computing insistence
should not use highly complex processing and powerful resources. Theonly requirement is that the
insistence-assignment and filtering mechanisms (which may be the same)act quickly without
interfering with the management. Consider vision in this respect, ituses very sophisticated and
powerful machinery, but it can also produce responses in a relativelyshort period of time (compared
to what might be required, say, for deciding which of two goals to adoptor how to solve a peculiar
problem). Sloman therefore emphasises that insistence and filteringmechanisms can be
It is easy to misunderstand the relation between insistence andfiltering. A reason for this is that
a system which is said to have goals that are more or less insistent,and that performs filtering, might
or might not actually produce insistence measures. Consider two modelsinvolving filtering. In the
first, a two stage model, one process assigns an interrupt prioritylevel to a goal (this is the insistence
assignment process) and another process compares the priority level tothe current threshold, and as a
result of the comparison either discards the goal or else puts it into amanagement input queue and
interrupts the management process scheduler so that it receives somemanagement processing. For
instance, suppose that when our nursemaid hears a baby wailing, itcreates a goal to attend to the
wailing baby. Suppose that the nursemaid has a simple rule that assignsan insistence level to such
goals: "the insistence of the goal to attend to a wailing child isproportional to the intensity of the
wail". Suppose that the rule contains an explicit function that returnsa number representing an
insistence priority level. So, in this model insistence assignment andfiltering are different processes.
In the second model, filtering (i.e., the decision of whether or not a particular goal should surface) is
based on rules that may be particular to every "type" of goal (if thereare types of goal), and no
explicit priority level representing the importance and urgency of agoal is computed. For instance,
one such rule might be embodied in our nursemaid who responds to theintensity of wailing of
babies. The system might filter out any goal to respond to a wailingbaby if the wailing is below a
certain intensity. In such a system, it might still be possible to talkabout the goal's insistence; the
insistence, however, is not computed by the system, nor is it explicitlyrepresented.
Sloman also writes "Attention filters need not be separate mechanisms:all that is required is that
the overall architecture ensures that the potential for new informationto interrupt or disturb ongoing
perceptual or thinking processes is highly context sensitive" (Sloman,1992b p. 244). Therefore not
only insistence but also filtering can in a sense be "implicit".
There is a subtle difference between the intentional aspect of "insistence measures", and the
propensity concept of insistence as such. The intentional aspect of insistence thatis typically
mentioned is one which heuristically represents importance and urgency.This applies also to
qualitative "measures" of importance and urgency. Such measures can inprinciple play other roles in
a system besides determining insistence as a propensity; and they mightbe evaluated as more or less
correct (in this respect they are at least implicitly factual). It isnot correct to define insistence as a
heuristic measure of importance and urgency. As was said above, somesystems can have goals that
can be said to be more or less insistent even if they do not produceinsistence measures. Information
is given the attribute "insistence" because of the role that it plays.
Sloman's actual definition of insistence is "the propensity to get throughattention filtering
processes and thereby divert and hold attention" (Sloman, 1992b). Withthis dispositional notion of
insistence one can make counter-factual conditional statements regardinga goal, by saying for
instance that "the goal was very insistent and it would have surfacedhad it not been for the fact that
the threshold was high". The dispositional notion of insistence can bevery subtle in complex
systems, and might require (for an adequate characterisation) that onemove beyond speaking in terms
of a goal being "more or less insistent" to describing the factors thatwould have contributed to its
management in slightly different conditions, and the reason why it didnot surface. One might also
refer to the likelihood that the filter be in a state in which a goalwith the given "insistence profile" can
surface. For instance, consider a system that evaluates all goals ondimensions A, B, C, and D
which might be said to comprise "insistence measures". The goal mighthave high measures on all
dimensions but D; suppose it was suppressed because the filter has a rule R that "the goal must have
high measures on dimension D". The system might also have a number of other rules which express
requirements along the other dimensions. One might say that "this goalwas very insistent". Since
insistence is a dispositional notion, this statement is valid, for oneunderstands that if only R had
been relaxed (and perhaps only slightly), the goal would have surfaced(other things being equal).
However, if it so happens that in the system in question R is always operative, then one might say
that the goal was not insistent, because it could not have surfacedunless its measure on D was much
higher. (Or R might be mutable in principle, but provably immutable in practice.) Atheorist who
desires in depth knowledge of the behaviour of such a system willrequire a language to describe
insistence that reflects the components that are involved.
Figure 4.4 contains a state-transition diagram which indicates that goalfiltering precedes goal
Figure 4.4. State-transitions for goals (4). Same as Figure 4.3, except thatgoal filtering follows
In order to distinguish the role of filtering described in this sectionfrom other roles, the former
will be referred to as "acute management protection", because the ideais that filtering should prevent
drastic side-effects that can happen if a goal surfaces if only briefly.The processes involved in
generactivating goals asynchronously to management processes, assigninginsistence, and
performing insistence filtering are called "vigilational processes", in contrast with management
processes. The term "vigilation" is used because in effect theseprocesses imply a readiness to redirect
attention in agents that have them.
It is expected that as different designs that support insistence andfiltering are developed, these
concepts will be modified and improved.
4.4.1 Other functions of filtering
Related requirements can be served by filtering. All of them have incommon the idea that when the
cost of interruption of management by a goal is high, the filterthreshold should be high. Designs that
satisfy the following requirements must still satisfy all of therequirements mentioned above,
especially that filtering should be done quickly and without disruptingmanagement processing. It
should be said about the following requirements that like otherrequirements, they are hypothetical.
As such they are subject to refutation and qualification. Moreover, thefollowing requirements are not
completely independent and might overlap.
126.96.36.199 Busyness filter modulation
One requirement is that when the busyness of a situation is high, thesystem should become more
inclined to suppress consideration of goals that are "trying" to surfaceunless it has reason to believe
that some overriding problem is likely to surface. (Busyness wasexplained in section 4.1.) Let us
call this "busyness filter modulation". Recall that a situation is busyto the extent that there are urgent
and important goals that are being processed that require more time thanis available. These conditions
are different from the "acute" ones, in which a split second distractioncould have drastic
consequences. This is because when the busyness is high, the systemmight not be likely to suffer
major consequences from engaging management processes for currentlyirrelevant goals; the
management simply can itself decide to postpone consideration of thegoal. Nevertheless, the system
might suffer from repeated distraction from many irrelevant goals. Byincreasing its resistance to
distraction, the system is taking the gamble that other goals that mightbe generated during this period
of high busyness are not as likely to be relevant, and that if they arerelevant that they will be
sufficiently insistent to surface.
Recall that apart from the importance and urgency of the current goals,there is another
dimension of variation of busyness, namely the number of current orpending goals. For example, a
situation can be busy because there is one very important and urgentgoal or because there are many
moderately important and moderately urgent goals, etc. For the samelevel of busyness (in terms of
importance and urgency of the contributing goals), the fewer the goalsthat are contributing to the
busy situation, the less likely it is that a more important goal thanone currently being considered will
surface (other things being equal). This is because the goals beingconsidered will be relatively
important and urgent; whereas, for the same level of busyness if manygoals are being considered
then it is more likely that a goal that surfaces will be more pressingthan one of the goals contributing
to the busyness of the situation. Therefore, a potentially useful ruleis that for the same level of
busyness, busyness should have a greater effect on thresholds insituations where the number of
urgent goals is smaller.
A simpler rule to use, which was suggested by A. Sloman (1994a), isthat as the rate at which
new goals arrive in relation to the rate at which they can be processedincreases, the filter threshold
should increase. This has the advantage that "detecting that thefrequency of interrupts by new goals
has exceeded some threshold may be easier than detecting otherdimensions of [busyness]". In
particular, this does not require computing the importance of thecurrent goals. Analysis and
simulations are required to determine how best to allow busyness tomodulate filter thresholds.
The author does not mean to imply that the main effect of beliefs aboutcurrent or expected
busyness should be to modulate filter thresholds. Indeed, this is arelatively minor function of
knowledge about busyness. There are difficult issues to addressconcerning how busyness should
affect the system's management, such as the time windows that it givesitself for managing goals
(how it controls its anytime algorithms), how it controls itsperceptual processes to scan for possible
problems which its beliefs about "expected busyness" imply could arise,whether it should favour
quick plans for action over slower ones which might otherwise bepreferred, etc.
188.8.131.52 Filter refractory period
A principle that is implicit in the previous section is that it might beproblematic for the
management processes to be interrupted too frequently. This might causeerratic processing and
"instability". In order to decrease the likelihood of this, it might beuseful briefly to increase the
resistance of the filter after a goal surfaces. This is analogous to therelative refractory period of
neurones, during which stimulation of a higher intensity than the normalthreshold is required for
triggering an action potential. The intention is not for the refractoryperiod to involve complete
intransigence to potential distractions (which is referred to as an"absolute refractory period"),
although implementation issues might well imply the need for an absoluterefractory period.
Applying the concept of refractory periods to psychological processes isnot without precedent.
M. M. Smyth et al. (1987) review literature concerning psychological refractory periodsin
"attentional" processes. Smyth and associates mention a variety of typesof refractory periods (and
reasons for them) that have been proposed. In a generalisation of ahypothesis presented by Smyth
and associates, one assumes that there is a "decision-making process"that is serial and comprises
successive non-interruptable sequences of processing (interruptions aredelayed until the end of the
current sequence). When decision-making starts, its first sequence isexecuted. The refractory period
of the decision-making process varies as a function of the length ofeach component sequence. Such
hypotheses have been investigated empirically in domains in whichsubjects are given tasks that they
must commence upon presentation of a stimulus. Response to a stimulus isdelayed by a predictable
amount if the stimulus occurs soon after the commencement of anothertask. Existing psychological
hypotheses are different from the current one in that (1) they assumean absolute refractory period
rather than a relative one. (They do not even distinguish betweenabsolute and relative refractory
periods.) (2) They seem to assume that refractory periods areunintended side-effects of a design
rather than functional aspects of a design.
184.108.40.206 Meta-management implementation
As was said in a previous section, the management ought to be able totake decisions to the effect that
the consideration of a goal should be postponed, or that a goal is to berejected and no longer
considered. An example of this is if a nursemaid realises that it cannotrecharge a baby because its
battery is broken and it has no way of fixing it. (In fact, the currentnursemaid scenario does not
allow batteries to break.) The nursemaid might therefore decide no longerto try to find ways to
satisfy the goal, or even that it should not try to manage it anyfurther. The question arises, however,
"How can such decisions be implemented in an agent?" In particular, anagent might want the goal to
become less insistent, for if the goal remains insistent, then it willkeep surfacing even after its
consideration has been postponed—the management's decision topostpone it will have been
ineffectual. In our example, this goal might keep resurfacing andthereby activate management
processes to try to satisfy it. This might interfere with the processingof other goals which are equally
important but which are much more pertinent since they can be solved.
Therefore it appears that there needs to be a link between managementprocesses and vigilation
mechanisms. For instance, the mechanisms that determine how a goalshould be processed once it
surfaces could be biased so that when this goal surfaces it triggers ameta-management process that
examines information about the decisions that have been taken about thesaid goal and if that
information indicates that the goal is "fully processed" then it shouldloop indefinitely, or simply
terminates once it starts. So long as this management process does notinterfere with other
management processes, then this mechanism would work. However, not allarchitectures will offer
this option, particularly if the user of the model feels that managementprocesses need to be limited in
number (for reasons mentioned above). An alternative response ofcourse is to increase the filter
threshold and hope that the generated goal simply is not sufficientlyinsistent. But this method is too
indiscriminate, since it will affect all other goals across the board.Yet another method is to
(somehow) ensure that this goal does not get generated or activatedanymore in the first place.
A better method (call it M) is to allow management processes to tell the filter to suppress—orin
other circumstances, be less resistant to— particular goals orclasses of goals. If there is a mechanism
that assigns numeric insistence measures, then an equivalent method to M is to get this mechanism to
vary the insistence of the goal whose consideration has been postponedshould it be activated. In our
example, the filter could be made to suppress the goal to recharge thebaby in question. Even if some
process assigned high insistence measures to it, the filter mightcontain a special mechanism to
prevent this particular goal from surfacing. A system that learns couldtrain itself to refine the way it
determines insistence of goals such that eventually meta-managementinput to the filter is no longer
required. For example, an actor or ceremonial guard whose job does notpermit sneezing or
scratching at arbitrary times might somehow train the sub-systems thatgenerate itches or desires to
sneeze not to assign high insistence in situations where that would becounter-indicated. (One would
have to determine how suitable feedback could be given to the vigilationmechanisms to evaluate its
The concept of meta-management control of goal filters can beillustrated by a metaphor of a
human manager with a secretary. The secretary can be seen as the filter.The manager might give
various filtering instructions to her secretary. For instance, she couldtell him that she does not want
to take any calls unless they concern today's committee meeting; or thatany advertisement letters
should be put in the bin; or that if person X comes to see her he should be let in immediately. These
instructions might turn out to allow some irrelevant distractions (e.g., X comes in but merely wants
to chat); or filter out some relevant information (e.g., an advert for very affordable RAM chips which
the manager needs to purchase). Some of this might lead to finer tuningof the filter in the future
(e.g., the manager might tell the secretary next time "Only let X in if he has information about Y").
And the secretary might have some other basic rules of his own; e.g., if the caller is a reliable source
saying that there's a life threatening emergency, then let them through.Notice that all of the rules
given here are qualitative. Filtering need not be based on quantitativemeasures of insistence.
Meta-management filter control appears to suit the purpose at hand, butthere are a number of
possible objections and caveats that must be considered. One caveat isthat since the basis for the
meta-management's decision might be invalidated (e.g., because an opportunity arises) the system
ought not to become totally oblivious to goals that it wants to besuppressed. This is not incompatible
with the idea of selectively increasing the threshold for a particulargoal (or goal type).
At first glance it might seem that meta-management filter control defiesthe purpose of filtering
since it involves the use of management processes, and managementprocesses are exactly the ones
that need to be protected by the filter. It is true that this methodinvolves the input of management;
however, it is crucial to note that this input is not requested by the filter. That is, the filter does not
call a management process—say as a subroutine—in order to decidewhether a goal should surface or
not. Instead, the filter merely consults information that has alreadybeen stored in it. If no information
concerning this goal is available to the filter, then the decision ismade on the basis of numeric
insistence measures (or whatever other bases are normally used).Therefore, not only is the
management not invoked, but the filter does not have the functionalitythat is required of the
The proposed filtering mechanism is not suitable for all designs. Insimple designs it will be
relatively "easy" to determine that a goal that is being filtered is ofthe type that the management has
asked to suppress. In more complex designs, two difficulties arise. Thefirst occurs in systems that
can express the same goal descriptor in a variety of ways but that donot use a standard normal form
for descriptors. For instance, in the design presented in the nextchapter, seeing a baby close to a
ditch generates goals of the standard form "not(closeTo(Ditch,Baby))". A different system with
greater expressive flexibility might respond to the same situation byproducing goals such as
"farFrom(Ditch, Baby)", "closeTo(SafeRegion, Baby)", etc. Whereas these goals are
syntactically different they might be considered by the managementprocesses (given its knowledge of
the domain) to be semantically the same. The problem is that the filtermight not be able to recognise
this identity. Notice that the problem of recognising identity of a"new" goal and one that has already
been processed also applies to some management processes; the differenceis that vigilational
mechanisms have fewer resources to use. The second source of difficultyis that some systems might
respond to the same situation by producing a number of goals. In thiscase, the goals are not simply
syntactically different, they are semantically different but have thesame functional role in the system.
For instance, in the scenario in which a baby's batteries are brokenthis might generate a wide variety
of sub-goals, e.g., goals that are different means of fixing the batteries. However, itmight be beyond
the capabilities of the vigilation processes to recognise the functionalequivalence between goals.
At this juncture, it is important to note another rationale andrequirement for insistence filtering:
separating different functional components. It is important for goalgenerators and insistence
mechanisms to be somewhat independent from management. The vigilationmechanisms need to be
able to increase the insistence of certain classes of goals regardlessof whether the management
processes want them to be suppressed. This is often (but not always)useful for categories of
important goals, where the designer (possibly evolution and/orlearning) knows the circumstances
under which they are likely to be relevant and urgent, but where themanagement processes might err
in assessing them along these dimensions. Obvious examples of this arethe "primary motives" of
hunger, thirst, sex, etc. A person might decide that he will not eat orthink about eating for a month.
But he will not be able to implement this decision: the goal to eat willbe activated with increasing
insistence as time goes on. This might not prevent him from fasting, butthe goal to eat will not be
suppressed effectively. According to P. Herman and J. Polivy (1991),when people fast they engage
in "obsessive thinking about food [...] their minds, as a consequence,[come] to be monopolised by
thoughts of food, including fantasies of gourmet meals past and to come,and plans for their future
career as chefs" (p.39). This holds whatever training people use (e.g., meditation is not effective). If
people have goal filters, it seems that they cannot control them aseasily, say, as they can move their
arms. Evolution has discovered that it is best to make it increasinglydifficult for management
processes to postpone the goal to eat as a function of time since thelast meal and other variables. So,
not only should the goal generators and filters operate withoutdisrupting management or performing
the same kinds of processes that the management executes, they should beresistant to some forms of
direct manipulation by the management. (The same can be said of paingenerators, and other sources
The task of the designer is to discover a satisfactory (but notnecessarily optimal) compromise
between hard and fast rules and the ability of the management throughits "higher level powers" to
by-pass and possibly inhibit or modify them. The designer's decisionneeds to be based on the
requirements that the system has to satisfy. There is no absolute rulethat holds for all environments
and all designs concerning the ways in which filtering mechanisms can becontrolled by management
processes. Nevertheless, researchers should try to refine the rules thusfar presented. If their efforts
fail, it could be argued that only learning mechanisms can solve theproblem of finding suitable
compromises for individuals in specific environments. If this were so,theoreticians would
nevertheless have an interest in studying the compromises produced bylearning mechanisms, in the
hope that principles—of various degrees of generality, to besure—could be extracted from what on
the surface appear to be idiosyncratic solutions.
So far in this section the focus has been on engineering considerations.Sloman argues that
even if it were good in some engineering sense for human beings to havegreater control of insistence
processes than they do, it might be that because they evolved at different junctures the vigilation
processes are separate from management processes. That is, thisseparation might have evolved
contingently, without offering an evolutionary advantage.
Why can't my tongue reach my left ear? It's just too short. I can't saythat evolutionary and
survival considerations explain why my tongue isn't much longer.Similarly if an architecture
happened to evolve with certain limitations, that need not be because itwould have no value to
overcome those limitations. I think some things have limited access tohigher level information
simply because they evolved much earlier, and originally needed onlyaccess to particular sub-
mechanisms. E.g. detecting shortage of fluid and sending a signal to thebrain may be done by
a primitive mechanism that simply can't find out if the correspondinggoal has previously been
considered and rejected or adopted. (Personal communication, 25 Nov.1993)
That is, not all extant (or missing) features of an architecture arethere (or absent) for a good
engineering reason, some are just side-effects of the way it developedphylogenetically. (Compare
Clark, 1989 Ch. 4).
The empirical example of hunger was given above as an instance of auseful inability to control
a module. However, there are other examples where the inability does notseem to be that useful.
States that are described as emotions often have the characteristic thata goal (or a cluster of goals and
"thoughts") tend to surface even if the management would prefer to notbe distracted by them.
(Sloman and Beaudoin refer to these states as "perturbance".) One mayconsciously and accurately
believe that the goal is causing more damage than it can possibly causegood. Consider for example
the hypothetical case in which a tribal man, M1, covets a woman who is married to a man who is in a
much higher social stratum than he. M1 might accurately believe that if he acts on his desires, he will
run a severe risk of being executed, say. For the sake of the argument,we can suppose that the man
has a choice of women in relation to whom he does not run the risk ofpunishment (so a simple
argument in favour of selfish genes fails). Thus M1 decides to abandon his goal and to stop thinking
about the woman; in practice, however, there is no guarantee that hismeta-management intention will
be successful, even if his behavioural intention is. It might be thatthis disposition does not favour the
individual but favours his genes. (Compare Dawkins, 1989).
In sum, some measure of management control of vigilation processes isuseful for
implementing meta-management decisions. But in autonomous agents suchcontrol is not (or should
not be) unconstrained. Most meta-management decisions do not need to beimplemented by
modulating the goal filter. Yet most research on meta-level reasoning hasnot even used the concept of
4.5 Summary of goal state specification
Given the above process specification, it is now possible to providemore terminology to describe
goal processes, and some constraints on goal processes. This will beparticularly useful for the
discussion of architectures in future chapters.
In this process theory, activation is a qualitative attribute of a goal's dynamic state that
expresses a relation between the goal and processes that operate on it.A goal, G, might be a focal or
contextual object of a management process. G is said to be a focal object of a management process,
P, if P is trying to reach one of the management conclusions regarding it. G is a contextual object of
P if P has some other goal(s) as its focal object(s), and if G figures in the deliberation of this
management process. For instance P might be a process of deciding whether to adopt a goal. This
goal would be the "focal goal" of P. The goals with which it is compared would be contextual goals.
Goals can dynamically change state between being focal and contextualwhile a process is executing
(typically this would be through invocation of subprocesses).
The theory allows for a goal to be in one or more of the followingstates of activation at a time
(these are predicates and relations, not field-accessing functions):
• filtering-candidate(Goal). By definition a goal is a filtering candidate if it is about togo
through a process of filtering, or is actually being filtered (asdescribed above).
• asynchronously-surfacing(Goal). A goal that is surfacing has successfully passed the
filtering phase and is about to be actively managed (this subsumes thecase of a "suspended" goal
being reactivated e.g., because its conditions of re-activation have been met). This is alsocalled
• synchronously-surfacing(Goal). Such a goal has arisen in the context of a management
process's execution (e.g., it is a subgoal to one of the management processes' goals). Thisis also
referred to as "top-down" surfacing.
• suppressed(Goal). A goal is prevented from surfacing by a filtering process.
• actively-managed(Goal, Process). A goal is actively managed if it is the focal object of a
currently executing (and not suspended) management process.
• inactively-managed(Goal, Process). Since management processes can be suspended, it is
possible for a goal to be a focal object of a suspended managementprocess. In this case the goal
is said to be inactively managed by the process.
• managed(Goal, Processes). A goal is managed if it is actively or inactively managed by a
• off(Goal). By definition a goal is "off" if the aforementioned predicates andrelations do not
hold in relation to it.
Goals that become an object of a management process without beingfiltered are said to be "recruited"
by that process. This is referred to as a top-down process. It isassumed that a goal cannot jump from
the state of being "off" to being managed, unless it is recruited by amanagement process. Goals that
surface and trigger or modify a management process are said to "recruit"that management process.
This is referred to as a bottom-up process.
The picture of goal processing provided in this chapter points towardsan architecture with a
collection of abilities of varying degrees of sophistication. Theseabilities span a range of areas in AI,
such as prediction, causal reasoning, scheduling, planning,decision-making, perception, effector
processing, etc. The picture is not complete, however. In particular, itis not yet clear how
management processing can best be controlled. Moreover, whereas a highlevel explanation was
given of the links between concepts such as importance and deciding, andurgency and scheduling,
the management functions have not been specified in a completelyalgorithmic fashion: we have
general guidelines but no complete solution to goal processing. Thismakes the task of designing an
agent difficult: we may be able to specify the broad architecture andthe kinds of processes that it
should be able to support—in this sense we are providingrequirements—but many of the details of
the agent (particularly its decisions rules) are not yet theoreticallydetermined. Thus, the architecture
will be broad but shallow. Nevertheless, it is instructive to try todesign such an agent, as it suggests
new possibilities and it demonstrates limitations in our knowledge. Thisis the task of the following
Chapter 5. NML1—an architecture
This chapter describes a proposed design of a nursemaid (called NML1)which is meant to operate in
the nursemaid scenario described in Ch. 1, and to meet the requirementsdescribed in the previous
chapters. Some of the limitations of the design are discussed in thefinal section of this chapter, and in
5.1 NML1—Design of a nursemaid
There are many ways to build a model that attempts to meet therequirements and specification.
NML1 is a particular design proposal that embodies a collection ofdesign decisions with different
types of justification. Many of the decisions were based on the groundsof effectiveness; others were
based on an attempt to explore Sloman's extant theoretical framework. Afew others were motivated
by empirical conjectures; however, justifying such hunches is not easy,because any particular
mechanism only has the implications that it does given assumptions aboutthe rest of an architecture.
Some decisions were simply arbitrary. And some are decidedlyunsatisfactory (usually because they
amount to postulating a black box) and were taken simply because somemechanism needed to be
proposed for the model to work at all. All the decisions areprovisional; mathematical and
implementation analyses are required to judge their usefulness (somehigh level analyses are reported
in the following chapter).
Early prototypes of NML were implemented in order to help design a morecomprehensive
system. However, most of the design as described here has not been implemented by the author,
since much of it derives from analysis of shortcomings of what wasimplemented. Ian Wright of the
University of Birmingham is currently implementing the NML1specification. Since we are concerned
with a proposed system, the current chapter is written in the simplefuture tense.
Although some of the alternative ways in which NML1 could have beenbuilt and their
implications are discussed in the present chapter, a more systematicexposition of the surrounding
design space is relegated to Ch. 6.
As discussed in Ch. 2, procedural reasoning systems (Georgeff &Ingrand, 1989) are worthy
of further investigation for meeting the requirements of autonomousagents, though there is a need to
improve them and explore alternatives. For this reason, it is proposedthat NML1 be designed as a
procedural reasoning system. Some of the similarities and differencesbetween NML1 and PRS are
discussed throughout and summarised in Ch. 6.
The overall architecture of NML1 is depicted in Figure 5.1. It will havea simple perceptual
module that will record information about the babies and stores it inthe World Model, which will be
distinct from the program that will run the nursery. There will be a Perceptual Control module that
will direct the camera to a contiguous subset of rooms, based onperceptual control strategies and
current activities. The number of rooms that can be simultaneouslyviewed will be a parameter of the
system. There will be Goal Generactivators that will respond to motivationally relevant information in
the World Model (such as a baby being close to a ditch) and the Goal Database by producing or
activating goals (e.g., to move the baby away from the ditch). The interrupt Filter will be able to
suppress goals, temporarily preventing them from disrupting themanagement. The Interpreter will
find management procedures that are applicable to goals and will select some for execution, and
suspend or kills others. Management procedures will be able to causephysical action through the
Management Procedure Library
Procedure Activation Records
(several active concurrently)
(procedures for expansion, assessing
Descriptor- Goal Index
System Procedure Library
Figure 5.1 Proposed architecture of NML1. (Some of the links
between modules are not displayed.)
5.2 The Perceptual Module and the World Model
Since the World Model will be distinct from the program that will runthe nursery there is a
possibility of information being dated and erroneous, actions havingunintended consequences, etc.
Every baby actually has the following features which will be recorded inthe World Model:
• A position. This will indicate the room and (x,y) co-ordinates of the baby. (According to the
current design proposal, there will only be one level of positioninformation. Approximate
positions will not be represented in NML1. Still, this would be usefulbecause information about
positions quickly becomes out of date, but in principle one could havean idea of approximate
location of a baby—e.g., through knowing that it would not have had enough time to move outof
a part of a room.)
• Life Status. This will indicate whether the baby is dead or alive.
• Age. This will be an integer denoting the baby's age in "cycles".(Cycles are the unit of time used
by the nursemaid and the simulation of the world.)
• Charge. This will be a real number between 0 and 1.
• Speed. This will represent the maximum number of steps per unit of timewhich a baby can take.
• IdentificationNumber. Every baby will be unambiguously identified by aninteger.
• Illnesses. This will be a possibly empty list of terms denoting thebaby's illnesses. There are three
possible illnesses: shakes, melts, and memory-corruption.
• Injuries. This will be a list of body-parts which can be injured,possibly including the head, right
or left arm, and right or left legs.
• isThug. This will be a boolean field indicating whether the baby is athug.
• PickedUp. This will be a boolean field indicating whether the baby ispicked up by the claw.
The World Model will also keep track of the co-ordinates of the claw,and its contents. The World
Model will be a multiple read, multiple write data base. It will beaccessed mainly by the Goal
Generactivators and the management processes.
The second form of perception is a sensor attached to the claw and usedlocally and only by the
Execution Device. No matter what the size of the room the claw sensorwill only detect items that are
within a 9 unit square centred on the claw. (One unit is the spacetaken by a baby and/or a claw.)
Within this area, the sensor will be able to determine theidentification number and locations of the
babies and the contents of the claw. The distinction between the twoforms of perception is useful
because the Execution Device requires accurate information in order todetermine whether actions are
successful or not.
5.3 The Effector Driver
The Effector Driver (ED) will interface between the NML1 cognitivearchitecture and its two
effectors: the claw and the camera. It will receive inputs(instructions) from the management
processes. (Management processes are discussed below. In this sectionthey will simply be referred
to as "controlling processes".) The ED will also have access to sensorinformation of the claw in
order to detect failure or success of primitive instructions. On the basisof the instructions it receives
the ED will cause claw actions and camera translation movement. Thecontrolling processes will
sequentially give out instructions to the ED. Sequences of instructionscan be thought of as "plans" at
the level of the processes, though the ED will only know about singleinstructions. Thus control of
the ED is useful to achieve their goals and direct behaviour.
The ED will be made of two channels. A channel will contain an inputport, a processor, and an
effector. One channel will be dedicated to the claw, the other to thecamera. This division will allow
claw and camera actions to execute in parallel.
The core information of instructions will have the following form:
instructionName(Argument1, ..., Argument N)
The arguments will be data-structures or pointers to them. There willalso be a port number and
an identification tag for the instruction. The port number will be usedto determine whether the
instruction is for the camera or the claw; the identification numberwill be used in records of success
or failure of instructions.
Here follow the instructions that will be available and theirspecification. Each specification has
two parts: a description of the action (if successful) andpreconditions. If the pre-conditions of an
instruction are violated then the action will fail, and the ED willstore an error message with the
identification tag in the World Model, which will be accessible to theprocess that initiated the
instruction. This information could be used by controlling processes forerror recovery.
• pickUp(Baby). Pre-conditions: (1) Baby is immediately adjacent to the claw; (2)the claw is
empty. Action: This will cause the claw to pick up Baby.
• deposit(Baby). Pre-conditions: (1) the claw is holding Baby; (2) there is anunoccupied
position that is immediately adjacent to the claw. Action: This willdeposit Baby in an adjacent
• moveTo(Position). Pre-condition: (1) claw is immediately adjacent to Position.Action: This
will cause the claw to move to Position.
• enter(). Pre-conditions: (1) the claw is immediately adjacent to a curtain;(2) the position
immediately in front of the curtain in the adjacent room is unoccupied.Action: This will cause the
claw to pass through the curtain and thereby to enter the adjacent room.(A curtain connects
exactly two rooms. See Figure 1.1.)
• plug(Baby). Pre-conditions: (1) Baby must be adjacent or on the recharge point;(2) the claw
must be adjacent or beside the recharge point. Action: This will causethe claw to plug Baby into
the recharge outlet. The claw will still be left holding the babyafterward.
• dismiss(Baby). Pre-conditions: (1) The claw must be holding Baby; (2) theclaw must be
adjacent to or on the dismissal point. Action: This will cause the babyto be removed from the
• moveCamera(Room). Pre-condition: The camera is in a room that is adjacent to Room.Action:
This will cause the camera to move to Room and thereby direct its gaze atit.
At any one time a channel of the ED will either be executing aninstruction or not. While executing an
instruction, it will be uninterruptable. (The actions are sufficientlybrief that this does not imply that
there will be long periods of not being interruptable.)
It will be up to the processes that control the ED to make sure thatprimitive actions are
combined in such a way as to direct the effectors coherently and recoverfrom whatever failures might
arise. For example, the controlling process might test for whether anaction, such as
pickUp(babyA), was successful and if it was not to decide what to do next on thebasis of the error
message. For example, if the error is that the claw is not adjacent tothe baby then the controlling
process might (re-) establish the goal to become adjacent to baby.Examples of "plans" (actually
management procedures) that will be used to drive the effectors via theED are given below.
5.4 Goals and Goal Generactivators
There will be two kinds of goal generactivators. The first kind aremanagement procedures
(abbreviated as "m-procedures"). They will be goal generators in asmuch as they will be able to
expand a solution to a problem, and thereby produce a collection ofgoals. These goals will typically
be means of achieving other explicit goals. (An explicit goal is a goalfor which there corresponds an
extant goal data-structure.) The second kind are programs runningasynchronously to the
management programs, which will respond to their activation conditionsby producing or activating
goals. (These can be thought of as reflex mechanisms based onperception of internal or external
states and events.) When a goal generactivator will produce goals, itwill set their descriptor fields,
and their insistence. If there already exists a goal whose descriptorcorresponds to the one that it
would produce, then, rather than produce a new goal, the generactivatorswill "activate" the extant
goal, i.e., they will make it a filtering candidate (hence the state of that goalwill no longer be "off").
This is because, in NML1, goals will be unique and they will beidentified by their descriptors (see
this section, below). Table 5.1 contains the main domain top-levelgoals that NML1 will be able to
produce, and the factors that will be used to compute their insistence.In NML1, a goal, G1, is
considered as a top-level goal if there does not exist another goal (orset of goals) G2, such that G1
is strictly a subgoal of G2.
NML1's goals, and their insistence functions
!( not(closeToDitch(Baby)) A function of the distance between the baby and the ditch
!( not(lowCharge(Baby))) An inverse function of the charge
!( not(thug(Baby))) A function of the number of babies in the room
A function of the population of the nursery
!(not (inNursery (Baby )))
A function of the population of the room and the time
during which this problem has been present.
A function of the number of injuries that the baby has
A function of the difference between the population of the
room and the threshold number of babies in the room
This goal can occur for different reasons. In this case therationale is that age(Baby) >ageThreshold.
Rationale for this goal is that dead(Baby).
Rationale for this goal is that injured(Baby).
The term Room unifies with an integer representing the room that isoverpopulated
The specification of NML1 goals differs from the one provided in Ch.3—as a simplification,
information about goal intensity is not computed. This is because it isnot yet clear precisely how to
determine intensity, nor how to use the measure in conjunction withother dimensions of goals. In
other respects, the Ch. 3 requirements hold for the nursemaid.
It was said above that asynchronous goal generators whose conditions ofactivation are met will
verify whether the goal that they would generate is present in thesystem, and if it is then rather than
generate a new goal they will activate the existing one. This willprevent the system from generating
different versions of the "same" goal. The need for such a mechanism wasdiscovered when an early
version of this architecture was implemented, and it was found that the"same" environmental
contingency (e.g., seeing a baby that is close to a ditch) repeatedly triggered theconstruction of
similar goal data structures. Comparison will be made with all goals, inparallel. Two goals will be
considered as identical if they have the same descriptor
. Descriptors will be expressed in a rapidly
obtainable canonical form to facilitate identity comparison.
Since goals will be individuated by their descriptors, the level ofdetail that exists in the
descriptor will be quite important. For instance, if the descriptormerely states that there is "a baby
close to a ditch", then this will express less information than isavailable if it states that "babyB is
close to a ditch", and therefore more dispositions will be consideredequivalent to it. The human mind
allows progressive refinement of the descriptor of goals, whereas NML1will not.
Goal generactivators must have access to parameters for determining whento generactivate
what goal. These data will be contained within the generactivators. Themain data will be: the
dismissal age for babies, the critical charge below which NML1 shouldconsider recharging a baby,
the maximum number of babies in a room (above which babies startturning into thugs), and the
maximum safe distance to a ditch. As an example of all of this, notethat a certain goal generator will
respond to the fact that a baby is older than the dismissal age bygenerating the goal to dismiss the
Many other goal generators will be required. For instance, after a goalhas been scheduled for
execution the system might detect that it is less urgent than previouslythought. This would cause a
goal to be generated which has as its objective to reschedule the goal.If a dependency maintenance
scheme were implemented for all types of decisions, the system could setup monitors which detect
when the reason for a decision is invalidated, and that would create agoal to reassess the decision. A
few other types of goal generators are mentioned below.
It is debatable whether the goal descriptor is a sufficient basis foridentity. One might argue that the rationale
field ought also be included: thus, two goals with the same descriptorbut different rationales would be embodied
in different data structures. Philosophical aspects of the identity ofmotivational constructs are discussed by Trigg
(1970, section V).
If a new goal does not get past the filtering phase, it will be storedin the New Pre-Management
Goals database, and removed from the system when its insistence is 0.Insistence of goals in this
database will decay steadily if not activated. However if the filterthreshold falls faster than the
insistence, then the goal may be able to surface. If a goal does getthrough the filtering phase, and if it
is new, it will be put on the bottom of a new goal stack in the GoalDatabase (described below). It
will be removed from there only if it is satisfied or otherwise deemedto be "inapplicable" (these are
judgements that can only be made by management processes).
5.5 Insistence assignment
Insistence assignment will be performed on a cyclical basis. Insistenceheuristics were abstractly
described in Table 5.1. NML1 will need to be prepared for thepossibility that more than one
generactivator generates the same goal at any one moment. Then howshould insistence be
computed? There are many alternatives. For experimental purposes, it wasdecided that
generactivators should contribute a suggestion for a goal's numeric insistence, and that more than one
generactivator could contribute such a suggestion. If a goal only hasone insistence suggestion, then
that will determine the insistence; if a goal has more than onesuggestion, its new insistence will be at
least equal to the maximum suggestion, while the other suggestions willbe factored into the equation;
if it has no suggestion, then its insistence will be decreased by theproduct of its previous insistence
and the insistence decay rate. Usually, there will only be one source ofinsistence per goal.
In the current state of the specification, the user of the model willhave to tweak the insistence
assignment functions so that they yield "sensible" values, based on anarbitrary set of utilities. A less
arbitrary set of assignments could result from a learning process or anevolutionary mechanism, both
beyond the scope of this research.
5.6 Goal Filter
NML1 will use an explicit filtering mechanism, which will take acollection of goals as input, and
allow at most one of them to surface at a time. It is designed accordingto a winner-take-all
mechanism which will allow for the possibility that no goal wins(surfaces). A variable numeric
threshold will be set for filtering goals. Most filtering candidateswill be subject to this threshold; but
certain specific goals will have their own threshold.
The Filter will have three independently variable components: (1) aglobal threshold ( i.e.. a
threshold that will apply to most goals), (2) idiosyncraticthresholds (3) and a management efficacy
parameter. The global threshold will be a real number between 0 and 1.The "idiosyncratic
thresholds" will be a collection of two item collections which willcontain (a) a pattern that can be
unified with a goal descriptor, and (b) a real number between 0 and 1representing a filter threshold.
The management efficacy parameter will weight the management's abilityto set idiosyncratic
Filtering will be performed according to the following three stagealgorithm. Firstly for all goals
that are "filtering candidates" the filter threshold will be found inparallel. If the descriptor of a
candidate goal does not unify with a pattern in the idiosyncraticthreshold ratios, then the global
threshold will be used for it. Otherwise, the pattern's associate willbe used as its threshold. Thirdly,
if (and only if) there are supraliminal goals (resulting from thefirst and second stages), then the most
insistent one will be allowed to penetrate the Filter, though astochastic function will be used in order
to prevent highly insistent goal from continuously overshadowing others(an inhibitory mechanism
could also have been used that would inhibit the more insistent goals).In order to promote stability,
multiple goal surfacing will not be permitted.
In NML1 only two parameters will drive the global filter threshold. Thismakes it different from
the specification of the previous chapter. In particular, in this domainthere is no need for "acute
management protection". The parameters are interval busyness measures andrefractory periods.
Busyness measures will be computed by management processes. Intervalbusyness measures are
rough estimates of the importance of the effects of the managementprocess remaining idle for a
certain time. The length of the period that will be used in this contextis an estimate of the time it
would take for a meta-management process to detect that a goal is notworth managing currently and
postpone it. The user of the model will need to determine on an a priori basis the particulars of the
function that takes busyness as an input parameter and returns athreshold value. This needs to be
done on the basis of knowledge of the utility that corresponds to giveninsistence measures. For
instance, if an insistence measure of 5 can be generated when the effectof non-surfacing is that a
baby dies, then (ideally) the filter threshold should only be above 5if the effect of interruption is
worse than a baby dying (e.g., if it causes two babies to die). Recent work on decision theory
(Haddawy & Hanks, 1990; Haddawy & Hanks, 1992; Haddawy & Hanks,1993; Russell &
Zilberstein, 1991) might be relevant for determining expedient numericfilter thresholds.
Management processes will be able to determine idiosyncratic filterthresholds indirectly. This
will be a method for meta-management processes to implement decisions topostpone the
consideration of goals by selectively increasing or decreasing thelikelihood that a goal surfaces. This
will be achieved as follows. A management process will inform the Filterthat it would like to add an
item (i.e., a pattern and a value) to the idiosyncratic filter thresholds. TheFilter will accept any such
request; however, it will weight the value by multiplying it by themanagement efficacy parameter.
This parameter will allow the system conveniently to control the extentto which an m-procedure can
control the Filter (and thereby control its own processing). If theparameter is zero, then the
management process cannot directly increase or decrease its sensitivityto particular goals. The need
for parameterised management filter control was discussed in Ch. 4.Idiosyncratic filter thresholds
will persist for a fixed number of cycles, and then will be deletedautomatically.
The state of activation of a goal that penetrates the Filter will be setto "asynchronously
surfacing". If the goal does not figure in a goal stack then a new goalstack will be created; on top of
this (empty) goal stack a new meta-goal
will be pushed. (Goal stacks are described below.) The
objective of this meta-goal will be to "manage" the surfacing goal. If agoal stack does exist, and its
associated m-process is suspended, then its m-process will beactivated.
5.7 M-procedures and associated records
Four kinds of data that are relevant to m-processing are described inthis section. (1) M-procedures
(m-procedures) are structures that will discharge the managementfunctions described in Ch. 4. As
described in a following section on the Interpreter, m-procedures thatare "applicable" to a surfaced
goal can be selected by the Interpreter. (2) Procedure activation records are temporary records formed
as a substrate for the execution of m-procedures, in response to thesurfacing of goals. (They are
analogous to call stack frames in procedural programming languages.)(3) Process records will
contain procedure activation records (they are analogous to Processrecords in Pop-11). (4) S-
procedures are implementation level procedures. These four types of datastructures are described in
M-procedures will contain information used to determine whether theyought to be executed,
and to construct procedure activation records for themselves ifnecessary. They will have the
• Applicability detector. Normally m-procedures will be applicable to agoal if the goal's descriptor
matches the procedure's goal descriptor, and some conditions that arespecific to that procedure
are met. However, unlike in PRS, the user of the model will have theliberty to allow a procedure
to be applicable to a goal even if it is not meant to satisfy it. Theapplicability detector, will tell the
Interpreter whether or not its m-procedure is applicable to a goal. Whenthe applicability detector
for a particular m-procedure (described below) will execute, it will"know" about the context in
which it is operating (through links with the World Model). It willtherefore be convenient to
allow the applicability detector to be responsible for setting the inputparameters of the procedure
activation record as well as other individuating information (theseparameters are described later in
• Goal pattern (intended outcome). This is the outcome which them-procedure aims to achieve.
This can be unified with the descriptor of a goal on a goal stack. Thegoal will often be the
This is termed a "meta" goal because the argument of the predicate ofits descriptor is a goal.
achievement of a management result, such as deciding whether to adopt agoal, or when to
execute it, etc.
• Body. The body will contain the instructions that will be executed whenan m-procedure is run.
These instructions may cause goals to surface (and thereby trigger morem-procedures), they may
read and manipulate information throughout the system, and they may sendcommands to the ED.
• Outcome predictor. This field will be reserved for "expansion"procedures that direct physical
action. It will contain a special purpose procedure that returns acollection of collections of
descriptors of possible consequences of the m-procedure. Some of theseconsequences will
actually represent failures of the m-procedure. This field will be usedby other m-procedures
which must decide which m-procedure to use to attain a goal. Generalpurpose predictive m-
procedures are described below, as are the difficulties of prediction.(NML1 will have to deal
with variants of the frame problem.)
• Activation revision procedure. Procedure activation records will have anactivation value (see
below). Each m-procedure will know how to compute the activation of itsactivation record. This
activation value will be used by the Interpreter to prioritise multipleprocedure activation records
that are applicable to the same goal. Activation procedures need to bedesigned to reflect the
relative efficacy of the procedure.
• Interruption action. Procedures that use interruptable anytimealgorithms will be able to store a
procedure which when applied yields the currently best solution. (Notethat this field will not
contain intermediate results of computation. For example, it will not bea process stack.)
Here is an abstract example of an expansion (management) procedurethat is meant to satisfy the goal
to recharge a baby (as in the scenario described above). Its applicability detector will respond to any
situation in which its goal pattern matches a surfaced goal, and wherethe goal is scheduled for
current execution. Its goal pattern will have the following form:
where Baby is an identifier that will be unified with a data structure containinginformation about a
baby, as described above. The body of the m-procedure could be definedin terms of the following
procedure, which uses the PRS goal expression syntax described in Ch. 2,within a Pop-11 context
! position(baby) = rechargePoint /*rechargePoint is a global variable*/
# hold(baby) and ! recharged(baby)
As in PRS, expressions preceded by an exclamation mark (!) denote goals to be achieved, and the
pound symbol (#) denotes a goal of maintenance. Either symbol will cause a goalstructure to be
created and pushed onto the goal stack of the process record in whichthe procedure activation record
is embodied. This particular m-procedure body will assert the goal tomove the baby to the recharge
point. Then it will assert the goal to plug the baby into the batterycharger. It will then assert a goal to
hold the baby until it is recharged.
In order for an m-procedure to be executed, the Interpreter must createa procedure activation
record for it. Procedure activation records are temporary activations ofa procedure. It will be possible
for there to be many concurrently active procedure activation recordsfor the same m-procedure. The
following information will be associated with procedure activationrecords.
• An m-procedure, with all its fields (expounded above). In particular,the body of the procedure
will be used to drive execution.
• Input parameters. These are data on which the process will operate,(-baby- in the example in
Procedure 5.1) and which will be provided by the applicabilitydetection procedure.
• An activation value. This will be the strength of the procedureactivation record. It will be
determined by the activation revision procedure contained in them-procedure. It will be used to
prioritise m-procedures when more than one m-procedure applies to agoal.
• A program counter indicating what to execute next.
• Focal goal. This will be the goal that triggered the m-procedure.(Unlike the goal information in
the procedure, this field can contain literals.)
• Contextual goals. These will be the goals in relation to which the focalgoal is being examined.
Procedure activation records will either be stored within an invocationstack of a process
record, or within a temporary collection of candidate records from whichthe Interpreter will choose
one to be applied to a goal.
There is a need for process records. These structures will contain the following information.
• An invocation stack, which will be a stack of procedure activationrecords.
• A pointer to the goal stack on which the process record's procedureactivation records will put
• Dynamic state information, indicating whether the process is shallowlysuspended or not, deeply
suspended or not, and live or dead. A process (P) is shallowly suspended if it is suspended by
the Interpreter while the Interpreter is doing its book-keeping; P can be deeply suspended by m-
processes—for instance, if an m-process (M) determines that two processes are interfering with
each other, M might suspend one of them. A process is dead if it has completed itslast
instruction or has been killed by some other process. Dead processeswill be removed from the
collection of process records.
It is expected that future versions of NML will have richer processrecords, possibly including
"strengths of activation" measures, which will be used to resolveconflicts between processes (this
will be analogous to contention scheduling in (Norman & Shallice,1986)). (Cf. next chapter.)
S-procedures are procedures that will be invoked in the same way asprocedures in the
implementation language (e.g., Pop-11, Smalltalk, Pascal), i.e., they will be "called", and will not
use the dispatching mechanism. (Dispatching is a function of theInterpreter and is described below.)
Of course, there are important differences between implementationlanguages in how they handle
procedure application. In principle they could make use of connectionistnetworks. But the important
thing about s-procedures is that the Interpreter's dispatching mechanism(described below) is not
used. That is, they will allow NML1 to perform actions that by-pass itsregular dispatching
mechanism (see the section on the Interpreter, below). This will allowprimitive actions to be taken
which can make use of the ED. This might later prove useful forimplementing cognitive reflexes.
Although s-procedures cannot be invoked by the Interpreter, it will bepossible for s-procedures
to be called within the body of m-procedures or actually be the body ofm-procedures. One needs to
know, however, if a particular s-procedure does make calls tom-procedures, in which case it can
only be applied within the scope of an m-procedure (otherwise goalassertions will fail).
5.8. Databases of procedures
The architecture will contain separate databases of m-procedures,s-procedures, and process records.
There will be an m-procedure database and an s-procedure database.Procedure activation records will
be stored within process records.
Some of the algorithms for m-procedures used by NML1 are described insection 5.12.
5.9 The Goal Database
The Goal Database (GD) will contain instances of goals—as opposed to goal classes. (Goal classes
will be implicitly within goal generators and management procedures.)Decisions and other
information concerning goals will be recorded in the GD. Some of theinformation about goals will be
stored in temporary data-structures that are not mentioned here. Inparticular, information about the
importance and urgency of goals will be implicit in procedures that doscheduling and arbitration
amongst goals. Nevertheless, it will be useful to have the followingseparate stores of information
about goals, within the GD. Decisions recorded in the database will betaken by management
processes. The information will be read by management processes, theInterpreter, and some goal
• New Pre-Management Goals. When goals are first generated, before they gothrough the filtering
phase they will be put in this database, and will be removed wheneverthey surface or their
insistence reaches zero, whatever happens first.
• Goal Stacks. These are structures which contain dynamic information forthe execution of m-
processes (See section 2.2.3). A goal that surfaces asynchronously forthe first time will be
moved from the Pre-Management Goal Database, to the bottom of a goalstack. The goal stacks
will contain stacks of goals. On any stack, if goal B is above goal A, then goal B is will be
considered to be a means of achieving A (i.e., a subgoal of A). There is no specific limit to length
or number of goal stacks. Goals stacks will also be components ofprocess records.
• Descriptor-Goal index. This will be used for mapping descriptors togoals. (Goal descriptors are
described in section 3.2). Every goal in the system will have an entrythere. Before a new goal is
produced, the system will check this index to make sure that there is noother goal with the same
descriptor. If there is, that goal will be used or activated, ratherthan allowing two goals to have
the same descriptor. (Notice that this will not preclude thepossibility of ambivalence, which can
occur if the importance field stores both positive and negativeinformation about the goal.)
• Overlapping Goals. The system will attempt to discover which (if any)of its goals have
overlapping plans. These are opportunities to "kill two birds with onestone". M. Pollack (1992)
refers to the satisfaction of overlapping goals as "overloadingintentions". This information is
stored in the Overlapping Goals database. This information can be usedin determining whether a
goal should be adopted or not (overloaded goals might be favoured overother ones). Note that
according to the ordinary sense of "opportunity", not all opportunitiesare best described as
overlapping goals: the opportunity might involve a new goal, such as whena motorist sees a
flower shop and decides to purchase roses for his partner.
• Goal Conflicts. The system also will record goals that it concludes tobe incompatible. This may
trigger an m-process to resolve the conflict (e.g., by selecting between the incompatible goals).
Representing goal relations (e.g., conflicts and opportunities) and reasoning about them raises
difficult unsolved questions, such as "How can we discover and representwhich particular part of
one or more plans interfere with one another?", and "When are utilitymeasures of incompatibility
useful and when are they not?" (Compare Hertzberg & Horz, 1989; Lesser,et al., 1989; Peterson,
1989; Pryor & Collins, 1992b; Sussman, 1975; Wilensky, 1983).
Although there will be no separate database equal to a two wayprocess-purpose index
(Sloman, 1978 Ch. 6), which maps goals to processes, and processesto goals, this indexing
information will be accessible to the system. The reasons for actionswill be recorded in the goal field
of procedure activations. And goals themselves will have a plan fieldcontaining information about
procedures for satisfying them, along with the status of execution ofthe procedures ( e.g., if the
procedure will have been activated, a pointer to the procedureactivation will be available.)
The schedule will be a multifaceted part of the Goal Database. It willcontain the different types
of decisions that can be taken regarding when certain goals should beexecuted.
• The sequential schedule. This will contain a list of goals which will beexecuted one after
another, with no other goals executed in between. Hence, this willcontain an expression of the form:
(Goal1 Goal2 ... GoalN), where GoalN is to be executed immediately after Goal(N-1).
Unless explicit information within the goal itself indicates that thegoal is suspended, it will be
assumed that the first goal in this list is always executable (i.e., an execution m-procedure might be
activated to get it going). If such a goal is suspended, its activationconditions must be recorded
somewhere else in the schedule, otherwise its entry might be removedfrom this part of the schedule.
The reason for this is to allow the rapid processing of goals in thispart of the schedule.