The Workload Curve

Abstract

Objective: In this paper I begin looking for evi- dence of a subjective workload curve.
Background: Results from subjective mental workload assessments are often interpreted linearly. However, I hypothesized that ratings of subjective mental workload increase nonlinearly with unitary increases in working memory load.
Method: Two studies were conducted. In the first, the participant provided ratings of the mental difficulty of a series of digit span recall tasks. In the second study, participants provided ratings of mental difficulty associ- ated with recall of visual patterns. The results of the second study were then examined using a mathematical model of working memory.
Results: An S curve, predicted a priori, was found in the results of both the digit span and visual pat- tern studies. A mathematical model showed a tight fit between workload ratings and levels of working mem- ory activation.
Conclusion: This effort provides good initial evi- dence for the existence of a workload curve. The results support further study in applied settings and other facets of workload (e.g., temporal workload).
Application: Measures of subjective workload are used across a wide variety of domains and applications. These results bear on their interpretation, particularly as they relate to workload thresholds.

Full text

Running head: THE WORKLOAD CURVE 1
PRE-PRINT VERSION. CITE Estes, S. (2015). The workload curve: Subjective mental 1
workload. Human Factors: The Journal of the Human Factors and Ergonomics Society, 57 2
(7), 1174–1187. 3
The Workload Curve: Subjective Mental Workload 4
Steven Estes 5
(703) 983-5716 6
sestes@mitre.org 7
The MITRE Corporation 8
9
Extended Multi-Phase Study 10
5355 words 11
12
13
Author Note 14
This work was produced for the U.S. Government under Contract DTFA01-01-C-00001 15
and is subject to Federal Aviation Administration Acquisition Management System Clause 3.5-16
13, Rights In Data-General, Alt. III and Alt. IV (Oct. 1996). 17
The contents of this document reflect the views of the author and The MITRE 18
Corporation and do not necessarily reflect the views of the FAA or the DOT. Neither the Federal 19
Aviation Administration nor the Department of Transportation makes any warranty or guarantee, 20
expressed or implied, concerning the content or accuracy of these views. 21
© 2015 The MITRE Corporation. All Rights Reserved. 22
Approved for Public Release; Distribution Unlimited. Case Number 14-2766 23

The Workload Curve 2
24
25

The Workload Curve 3
Abstract 26
Objective: In this paper I begin looking for evidence of a subjective workload curve. 27
Background: Results from subjective mental workload assessments are often interpreted 28
linearly. However, I hypothesized that ratings of subjective mental workload increase non-29
linearly with unitary increases in working memory load. Method: Two studies were conducted. 30
In the first, the participant provided ratings of the mental difficulty of a series of digit span recall 31
tasks. In the second study, participants provided ratings of mental difficulty associated with 32
recall of visual patterns. The results of the second study were then examined using a 33
mathematical model of working memory. Results: An s-curve, predicted a priori, was found in 34
the results of both the digit span and visual pattern studies. A mathematical model showed a 35
tight fit between workload ratings and levels of working memory activation. Conclusion: This 36
effort provides good initial evidence for the existence of a workload curve. The results support 37
further study in applied settings and other facets of workload (e.g., temporal workload). 38
Application: Measures of subjective workload are used across a wide variety of domains and 39
applications. These results bear on their interpretation, particularly as they relate to workload 40
thresholds. 41
keywords: Mental Workload, Working Memory, Mathematical Models, 42
précis: When we rate subjective mental workload, are we grading on a curve? In this 43
article I examine the shape of mental workload ratings and find that, in fact, we do. 44

The Workload Curve 4
The Workload Curve: Subjective Mental Workload 45
I find – and perhaps this is your experience as well – that an easily managed mental task 46
can become, with just the slightest amount more mental demand, decidedly unmanageable. The 47
relationship between unitary increases in cognitive load and the subjective experience of mental 48
demand seem nonlinear; perceived mental workload is hardly impacted at all by increases in 49
demand under low cognitive load, but rises quickly and disproportionally as the limits of the 50
cognitive system are approached. In the context of subjective rating scales, it would appear that 51
sometimes five is closer to six than to four. That is, the cognitive load required to move a 52
subjective workload rating from five to six is less than that required to move the rating from four 53
to five. This relationship, which is the basis of my central hypothesis, should take the shape of an 54
“s” or sigmoid curve, as notionally depicted in Figure 1. 55
56
Figure 1. The subjective workload curve 57
The hypothesized asymptote at the top of this “s” curve is the predictable result of using a 58
finite scale (subjective mental workload) to evaluate a conceivably infinite quantity (mental 59
load). Once workload is rated a ten on a one to ten scale, it does not matter whether task load 60
proceeds to double or triple or quadruple. In each instance, subjective workload shares the basic 61

The Workload Curve 5
quality of being “too much” and is therefore a ten. At the lower end of the scale, however, there 62
is a finite beginning to the scale and the workload. There the relationship between perceived and 63
actual load could be linear, a power function, exponential, and so on. I propose that, as seen in 64
Figure 1, where subjective ratings of workload are low or moderate something resembling a 65
power function will be observed. With very low subjective workload, unitary increases in 66
cognitive load will result in modest increases in subjective ratings. As subjective workload 67
increases, however, the rater will become more sensitive to their diminishing resources and 68
unitary increases in cognitive load will result in increasingly large jumps in subjective ratings. 69
Throughout the paper I will refer to my hypothesized relationship between cognitive load and 70
subjective ratings of mental workload as the workload curve. 71
The hypothesized workload curve is unique within the literature. It is commonly 72
assumed that equal intervals in ratings equate to equal intervals of imposed workload (Reid & 73
Nygren, 1988; Young et al., 2015). If one is of a mind to find them, there are examples of curves 74
in studies of mental workload (Eggemeier et al., 1982; Berka et al., 2007), but they uniformly go 75
uncommented upon and are never the topic of study. There is no mention of a curve in subjective 76
ratings of workload in the seminal references for the most common subjective workload rating 77
tools (Hart & Staveland, 1988; Wierwille & Casali, 1983). Hart and Staveland acknowledge the 78
possibility of a proportionality between observed ratings and the magnitude of the rated 79
phenomena, though they make no hypothesis as to how that might impact ratings or whether a 80
curve may result. Yet, a curve in subjective ratings of mental workload would have a significant 81
impact on the interpretation and perception of a subjective workload rating. 82
In this paper I review the evidence for a curve in the most commonly used subjective 83
measures of mental workload and present the results from two studies. Those results are then 84

The Workload Curve 6
evaluated in a mathematical model of working memory activation decay (subjective ratings of 85
mental workload are strongly influenced by working memory). I begin with a brief review of 86
mental workload and the impacts of working memory on subjective workload ratings. 87
Subjective Mental Workload 88
Workload, to oversimplify, is complex. It is multidimensional and its magnitude is the 89
result of interactions between the human, the task, and the environment (Hart & Staveland, 1988; 90
Simon, 1969; Wickens, 2008). Ultimately, documentation of the workload curve must take into 91
account all of these variables. But, I require a starting point and the evaluation of the mental 92
dimension of workload is a reasonable place to begin if for no other reason than it is difficult to 93
quantify and there is some appeal in dealing with the most difficult elements of a problem first. 94
A universally accepted definition of mental workload has been elusive. For this paper, 95
mental workload is defined in the strictest sense: the work done by the mental system. Somewhat 96
less recursively, mental workload is the cognitive and perceptual processing expended in the 97
course of completing a task (Eggemeier & Wilson, 1991) where processing includes the storage, 98
maintenance, manipulation, and retrieval of information within working memory and long-term 99
memory as accomplished through control of the locus of attention. 100
The measurement of workload has been a topic of interest in the applied community since 101
at least the 1950s. By the late 1960s workload had become an area of significant research with a 102
variety of techniques being developed to measure it. Wierwille and Williges (1978) classified 103
these techniques into three categories: performance measures, psychophysiological measures, 104
and subjective assessment. Those categories still accurately classify the vast majority of mental 105
workload assessment techniques (Gawron, 2008) used today. 106
Of the many dozens of methods within those categories proposed for measurement of 107

The Workload Curve 7
workload - including dual task tests, performance measures, heart rate, respiration, pupil 108
dilation, fMRI and IR spectrometry - subjective workload measures have assuredly been the 109
most widely used. This is likely attributable to their usability and face validity. As Moray et al. 110
put it, "If the person feels loaded and effortful, he is loaded and effortful whatever the behavioral 111
and performance measures may show” (Moray et al., 1979, p. 105). 112
Representative subjective workload measurement techniques such as NASA-TLX (Hart 113
& Staveland, 1988), SWAT (Reid & Nygren, 1988), and Cooper-Harper (Cooper & Harper, 114
1969) all produce a scalar rating of workload. Cooper-Harper’s scale is ordinal and NASA-TLX 115
and SWAT’s continuous. Several studies have shown strongly correlated workload ratings across 116
these and other subjective measures (Rubio et al., 2004; Hess, 1971; Vidulich & Tsang, 1985). 117
Many measures, like NASA-TLX and SWAT, are multi-dimensional and make allowances for 118
distinguishing between different sources of workload, including mental workload. 119
While there are many subjective techniques, including open-ended scales, there is a very 120
limited set that see consistent, applied use. The most popular, if the frequency of study is any 121
indicator, is by far NASA-TLX. In her retrospective on its use, Hart (2006) found over 550 122
studies of NASA-TLX. To be clear, this is not just studies that made use of NASA-TLX, but 550 123
studies of NASA-TLX. Because of their overwhelming popularity, closed, bipolar rating scales 124
for mental workload are of particular interest for this paper. 125
Consciousness, Working Memory, and Subjective Mental Workload 126
When mental workload is being measured subjectively, one may reasonably ask “What is 127
it that is being measured?” It does not seem, for example, that we sense the workload involved 128
in visual perception; it is not effortful to see although an incredible amount of neural processing 129
is required. Instead, our perception of workload is influenced almost solely by processes of 130

The Workload Curve 8
which we have some conscious awareness (Vidulich, 1988; Yeh & Wickens, 1988). 131
In cognitive psychology, consciousness is thought to reside in working memory (Baddeley, 132
2007; Hassin et al., 2009). As the location of consciousness in the cognitive system, working 133
memory has been attributed a central role in subjective ratings of mental workload (Gopher & 134
Braune, 1984; Ericsson & Simon, 1980). 135
Yeh and Wickens (1988) found that the majority of variables found to affect subjective 136
workload are related to working memory demands. Those variables include capacity (Hauser, 137
Childress, & Hart, 1982), presentation rate (Daryanian, 1980), processing rate (Tulga & 138
Sheridan, 1980), attention allocation, and decision alternatives. 139
Judgement of Mental Workload 140
The variables catalogued by Yeh and Wickens (1988) are a product of the capacity and 141
durability limitations of working memory. One prevalent theory as to why those limitations exist 142
is decay theory. According to decay theory, the strength of a memory, determined by its level of 143
activation, fades over time (Baddeley, 1975; Brown, 1958). Further, the pool of activation is 144
limited and must be spread across all chunks in working memory (Just & Carpenter, 1992). In 145
order to be recalled, a chunk’s activation must exceed a threshold (Barrouillet, Bernardin, & 146
Camos 2004) and therefore the initial strength of the memory trace and the decay resultant from 147
the amount of time the trace has been held in memory are critical to determining the probability 148
of recall. 149
While decay is a critical element of working memory, it seems unlikely that, in 150
generating an estimate of mental workload, we directly measure decay of memory activation. 151
More probable is the hypothesis that someone asked to rate their mental workload, lacking a 152
direct measure of working memory activation, bases their rating on the effects of working 153

The Workload Curve 9
memory activation and decay. 154
One could theorize many mechanisms by which activation influences judgements of 155
mental demand. For instance, metamemory and learning research has documented our ability to 156
estimate remaining working memory capacity and rates of forgetting as determined by activation 157
(Amichetti et al., 2013; Kornell et al., 2011; Halamish, McGillivray, & Castel, 2011; Bunnell, 158
Baken, & Richards-Ward, 1999). It may be that the accuracy of judgements about available 159
capacity increases as available working memory capacity decreases and that this in turn gives 160
rise to the workload curve. Whatever the precise judgement process, it is my contention that 161
they are based on the effects of working memory activation. Evidence for this hypothesis is 162
discussed later in the modeling section of the paper. It is worth noting both that judgements in 163
ratings of workload more have been discussed before in the literature (Hart & Staveland, 1988) 164
and that they are thought to be relative to prior experience rather than absolute (Sheridan & 165
Simpson, 1979). 166
In summary, I hypothesize that effects of working memory activation decay are critical to 167
the assessment of mental workload: 168
• When subjective mental workload rated is plotted as a function of a measure of the 169
imposed workload the result is curvilinear 170
• The relationship between subjective and imposed workload takes the shape of an s-curve 171
(the workload curve) 172
• The workload curve results from judgement of the effects working memory activation 173
decay 174
Study 1 175
To test these hypotheses, I performed two web-based recall studies and one modeling 176

The Workload Curve 10
exercise. Study 1 required serial recall of a digit span in order of presentation. After each trial, 177
participants were asked to rate the mental demand of the recall task. 178
Participants 179
Study 1 included 102 participants. All participants were employees of the MITRE 180
Corporation and participated voluntarily and anonymously. The study was deemed exempt by 181
MITRE’s Institutional Review Board (IRB) under the provisions of 45 CFR 46. Participants 182
were recruited via an internal newsletter. MITRE is a technical company and this was reflected 183
in the demographic information provided by participants, 52% of whom described their job as 184
some form of engineering. Other job descriptions included computer scientists (7%), managers 185
(6%), IT professionals (5%), and administrators (4%). No further demographic information was 186
collected as additional details like age and gender would, in some cases, allow identification of 187
the participant. 188
Procedure 189
Participants accessed an internal web site to complete the study. On the welcome page 190
they were given a high level description of the task. Participants then completed two practice 191
problems representing the easiest and most difficult recall spans. After the practice problems, 192
participant completed 24 digit span recall trials. 193
In each trial the participant was presented with a digit span of varying length. 194
Presentation time was determined by multiplying the span length by 500ms. Once the 195
presentation time elapsed, the span was removed and the participant was provided an open text 196
field for entering the digit span as they recalled it. There was no time limit on recall. 197
After entering each span, the participant was asked to rate the mental difficulty of the task 198
on a scale of one to ten. Not unlike NASA-TLX, mental difficulty was described as a rating of 199

The Workload Curve 11
the mental effort required to complete the task. Decimal ratings were allowed. The scale was 200
bipolar with adjectival labels of low and high at each end. Once all trials were completed, 201
participants were asked to provide some information on recall strategies used during the study. 202
Manipulations 203
A repeated measures design, the study was block ordered and included a No Chunks and 204
a Chunks condition, each with 12 levels of digit span to be recalled (resulting in a total of 24 205
trials). In the No Chunks condition each digit span level corresponded with the number of digits 206
actually presented to the participant (e.g., at level 10 the participant was presented with 10 digits 207
to recall). A random number generator was used to create the digit spans. Each span was then 208
inspected manually to ensure that no obvious patterns were included. Commonly chunked digits 209
such as local area codes or ZIP codes were replaced. Where possible (levels 1 to 9), an integer 210
was used only once. Integers appeared, at most, twice in the span. Although care was used to 211
ensure no obvious chunks were included, this did not preclude the participants from developing a 212
chunking strategy based on non-obvious, personal information. 213
The second condition is referred to as the Chunks condition. As with the No Chunks 214
condition, participants were presented a digit span, recalled that span, and provided a rating of 215
mental difficulty. However, as part of the Chunks condition one digit in the string was replaced 216
with a five-digit chunk based on an obvious pattern (either 12345 or 54321) such that a condition 217
span of six would contain five digits followed by a five-digit chunk. Participants were told to 218
expect these patterns in some spans. To ensure participants recognized them as chunks, those 219
digits, otherwise black, were colored blue. I have hypothesized that working memory decay, and 220
therefore chunk activation, plays a central role in ratings of mental workload. This condition is 221
therefore included to verify workload ratings were based on the number of chunks in the span 222

The Workload Curve 12
rather than the number of digits or apparent length of the span. For my hypothesis to be 223
supported, a 10 digit number composed of 6 chunks (as in the center of Figure 2) should be rated 224
more like the 6 digit number containing no chunks on the right of Figure 3 than the 10 digit 225
number containing no chunks on the left. 226
227
Figure 2. Chunks versus digit spans 228
The spans used for all twenty for trials are provided in Table 1. 229
Table 1 230
Study 1 Digit Spans 231
Length
No Chunks
Chunks
1
5
12345
2
73
6 54321
3
185
94 12345
4
4967
296 54321
5
43795
3172 12345
6
154859
21531 54321
7
5439156
981564 12345
8
15362084
2451386 54321
9
945132708
73016829 12345
10
1594862730
953472680 54321
11
38170649527
5203147968 12345
12
715039428603
16428530920 54321
232
Results 233
The primary concern was the shape of subjective mental workload: will an s-curve be 234
produced when plotting workload ratings as a function of the number of chunks in the span? 235
Qualitatively, as seen in Figure 3, the answer is yes. The shape was confirmed quantitatively 236
when the data was fit to a 4PL logistic model (Baud, 1993) which produced an R2 value of 0.99 237

The Workload Curve 13
(RMSE = 0.16). 238
239
Figure 3. Ratings as a function of chunks in digit span with confidence intervals 240
On the secondary question of equivalence I found a largely favorable result. A repeated 241
measures ANOVA showed, as would be expected, no statistically significant difference between 242
the two chunk conditions (F1, 101 = 2.9, p = 0.09). Likewise, Eta Squared showed chunk 243
condition accounted for none of the total variance (η² = 0.00). Testing for equivalence using 244
Inferential Confidence Intervals (Tryon, 2001) with a criterion of a 0.75 scale degree, seven of 245
the twelve span lengths were shown to be statistically equivalent (p < .05) when compared across 246
chunk conditions (Table 2). 247
Table 2 248
Inferential confidence intervals 249
Span
1
2
3
4
5
6
7
8
9
10
11
12
HCI
0.33
0.55
0.59
0.61
0.48
1.35
0.56
1.19
1.01
0.90
0.89
0.65
250
Discussion 251
Study 1 showed that, indeed, participant’s judgement of mental workload took the shape 252

The Workload Curve 14
of an s-curve as nonlinear increases in ratings of workload occurred with unitary increases in the 253
number of chunks in the span. And, with regard to the number of chunks in the span, the data, 254
both in practical interpretation and statistical testing, favors the conclusion that mental difficulty 255
was judged by the number of chunks rather than the number of digits in the span. This latter 256
point will be important in the modeling section of the paper, but for now we can turn to a second 257
test of the hypothesized workload curve. In this second study, a different type of demand will be 258
placed on working memory to see if the sigmoid shape remains. 259
Study 2 260
The digit span study relied on working memory in the central executive and articulatory 261
rehearsal loop (Kahana, 2012). Capacity and durability in the rehearsal loop, however, differs 262
from that of the visuospatial sketchpad (Card, Moran, & Newell, 1983). To determine if the s-263
curve persists when processing load is placed on the visuospatial sketchpad subsystem, a visual 264
pattern was used for the second study. For this study, I hypothesized the s-shaped workload 265
curve seen in the digit span would be retained, though steeper due to the higher demands placed 266
on working memory. 267
Participants 268
Thirty volunteers participated in Study 2. As with the first study, all participants were 269
employees of the MITRE Corporation and completed the study voluntarily. Limited 270
demographic information was collected to ensure anonymity. The study was deemed exempt by 271
MITRE’s Institutional Review Board (IRB) under the provisions of 45 CFR 46. Participants 272
were recruited via an internal newsletter. 54% of participants described their job as engineering. 273
Other job descriptions included IT professionals (6%), computer scientists (5%), managers (3%), 274
and administrators (3%). 275

The Workload Curve 15
Procedure 276
Participants accessed an internal web site to complete the study. The welcome page 277
provided a high level description of the task. This was immediately followed by training for the 278
first of three sets of trials. The training included instructions on how the trials should be 279
completed, example problems with solutions, and three practice problems. Each participant 280
completed all levels of every condition. Conditions were presented in blocks and block order was 281
varied. Prior to each block, participants were given instructions on how to proceed and then 282
completed three practice trials. The three practice trials represented the easiest, moderate, and 283
most difficult levels of the condition. 284
The visual pattern was shown to the participant using a wheel consisting of 6 colored 285
buttons (grayscale example in Figure 4). In each trial buttons were highlighted for one second in 286
a predetermined pattern. The pattern was repeated back by the participant by pressing the buttons 287
on the color wheel. The entire pattern was shown only once. Response time was unlimited and 288
after each trial the participant was asked to rate the mental difficulty of the task with the same 289
bipolar scale used in the first study. In addition to collecting the scale rating, correctness of the 290
response and the total response time was calculated. Upon completion of the study, participants 291
completed a brief exit survey. This procedure was modeled after the electronic game Simon, 292
variants of which have been widely used to test visual memory span (Cleary, Pisoni, & Geers, 293
2001; Gendle & Ransom, 2006; Humes & Floyd, 2005). 294
295

The Workload Curve 16
Figure 4. Visual pattern study button wheel 296
Manipulations 297
Instructions for a set of trials and the length of the visual span were manipulated. Trial 298
instruction conditions included: As Seen, Reverse, and Offset. In the As Seen condition, the 299
participant repeated the pattern back in the order it was presented. The Reverse condition 300
required the participant to repeat pattern back in the opposite order, beginning by pressing the 301
button highlighted last, first. This condition necessitated manipulation of information in working 302
memory, but not the creation of new chunks for storage. The Offset condition required both 303
manipulation and the creation of new working memory chunks. In this condition, the participant 304
repeated the pattern back in the order presented, but with a one position clockwise offset as 305
illustrated in Figure 5. Span length varied from one item in the visual pattern to ten items. 306
307
Figure 5. Visual pattern study instructions by condition 308
Results 309
As expected, the workload curves found in Study 2 were even more pronounced than 310

The Workload Curve 17
those seen in the digit span (Figure 6). Using the As Seen curve as an example, ratings at the 311
bottom of the curve increased less than half a point between spans, jumped to nearly 2 full points 312
between spans in the middle of the curve, and then subsided back to increases of half point or 313
less near the top. Though the increases happened more quickly in the other conditions, the basic 314
pattern remained. Using 4PL logistic functions the As Seen (R2 = 0.99, RMSE = 0.24), Reverse 315
(R2 = 0.98, RMSE = 0.24), and Offset (R2 = 0.99, RMSE = 0.08) showed strong fits as sigmoid 316
curves. When sigmoid model curves were applied to individual data the RMSEs increased across 317
the board but were, nonetheless, within one scale degree of the observed value (As Seen RMSE = 318
0.75, Reverse RMSE = 0.49, Offset RMSE = 0.98). 319
320
Figure 6. Visual span subjective ratings as function of span length with confidence intervals 321
The ordering of the results was also as expected with the As Seen condition rated the 322
easiest and Offset the most difficult. A repeated measures ANOVA showed those differences to 323
be significant (F2, 28 = 63.7, p < .05). There was a significant main effect for span (F6, 24 = 181.6, p 324

The Workload Curve 18
< .05) as well. According to Cohen’s (1988) guidelines for interpretation of the Eta squared test, 325
the effect sizes for the instruction condition and span were both large (η² = 0.34 and η² = 0.80 326
respectively). 327
Discussion 328
Once again we see distinct s-curves. With small visual spans, unitary increases in 329
workload correspond to modest increases in workload ratings. This is followed by noticeable 330
jumps in ratings as the participant becomes more sensitive to diminishing working memory 331
resources, and an asymptote as those resources become overwhelmed. While these results need 332
to be tested further in ecologically valid environments, they provide a strong initial argument for 333
the workload curve and rethinking how we interpret subjective workload ratings collected via 334
popular methods like NASA-TLX. 335
The results also uncover an unfortunate, if logical, interaction between our limited 336
working memory capacity – estimates range from 7 chunks (Miller, 1956) to as low as 3 or 4 337
chunks (Cowan, 2001) – and perceived mental workload. In most situations, larger jumps in 338
subjective ratings with unitary increases in load would be preferred – particularly if boredom is 339
of concern – when working memory demand is low. Smaller jumps would be preferred when 340
working memory is moderate and we begin to approach the limits of our capacity. However, the 341
workload curve shows the opposite to be true. Given that the practitioner is often trying to 342
manage user workload, knowledge of the curve is valuable as is gives guidance on the magnitude 343
of mental demands necessary to increase a user’s engagement and emphasizes how carefully 344
workload must be managed passing the midpoint of the subjective rating scale. 345
A Model of the Subjective Mental Workload Curve 346
In the section on judgement I hypothesized that the subjective workload curve arises from 347

The Workload Curve 19
judgements about the availability of information in working memory. This does not mean that 348
other parts of the cognitive system do not impact subjective ratings of mental workload. Rather, I 349
am hypothesizing that availability plays the central role in these judgements and that knowledge 350
of availability alone is sufficient replicate the workload curve. If this is true, then a model of 351
working memory availability (i.e., activation decay) should predict ratings of mental difficulty in 352
conditions where pure recall plays a lesser role (the Offset condition of the visual span study) just 353
as accurately as it does ratings in near pure recall conditions (the As Seen condition). 354
Activation Based Model 355
In his famous paper, “You can’t play 20 questions with nature and win" Allen Newell 356
(1973) argued that experimental psychology, while adept at answering binary questions about 357
psychological phenomena, was not advancing us towards a unified understanding of the mind. 358
Newell believed that a unified theory required the development of cognitive architectures: 359
software that implemented human cognitive capacities and constraints such that they could be 360
used to test a theory’s plausibility within the broader cognitive system. 361
One of the many cognitive architectures that arose as a response to Newell’s challenge is 362
ACT-R (Anderson, 2007). Relevant to the work at hand, research on memory using the ACT-R 363
software and formulations has been extensive, with hundreds of published papers on the topics of 364
memory activation, decay, or interference (ACT-R related research is archived at http://act-365
r.psy.cmu.edu/). For this particular paper, my objective was to provide plausible support for the 366
hypothesis that the workload curve demonstrated in the first two studies arises from judgements 367
based on the effects of working memory activation and decay. As such, I relied on memory 368
activation and decay functions that have been well established within the ACT-R community 369
(Altmann & Schunn, 2002; Altmann & Trafton, 2002; Böhm & Mehlhorn, 2009; Anderson, 370

The Workload Curve 20
Reder, & Lebiere, 1996; Pape & Urbas, 2008; Sohn et al., 2004). First, the activation of a 371
memory trace was calculated using the following equation taken from the literature (Altmann & 372
Trafton, 2002): 373
𝑎 = 𝑙𝑛%( 𝑛
𝑇)
(1)
where a is activation, n is the number of times that chunk is rehearsed, and T is the total time the 374
trace is held in memory (and the determinate of decay). Next, in order to mimic the division of 375
activation across all working memory chunks I reduced the activation as a function of the 376
number of chunks in the problem span: 377
𝑑 = 𝑎 +%1
𝑐− 1
(2)
where d is the divided activation, a is activation, and c is the number of chunks the activation 378
must be divided amongst. The idea of limited activation source pools and their distribution 379
amongst all the chunks held in working memory has been previously documented in the 380
literature (Anderson, Reder, & Lebiere, 1996). 381
These two basic equations allowed me to model a relationship between the number of 382
chunks to be memorized (visual pattern span), decay over time, and a subjective rating of mental 383
demand. The first equation requires a reasonable estimate of the number of times each chunk is 384
rehearsed between storage and final recall. In order for the model to have explanatory power, we 385
should set free parameters like rehearsal once and use those settings for modeling the results of 386
all three visual pattern conditions. The model could not be applied to the digit span study as 387
response time – required to set T – was not collected. For this model, rehearsal (n) was set to 3 388
retrievals of the chunk (a plausible level of rehearsal that provided the best overall fit). 389
With the parameters set, an activation value was calculated for each participant response 390

The Workload Curve 21
using the condition to determine number of chunks and response time for the decay period. The 391
data was then collapsed across all three visual pattern conditions and an average activation 392
generated for each degree of the subjective rating scale. It was necessary to round each raw 393
subjective rating to the nearest whole number in order to ensure a sufficient number of 394
observations at each point in the rating scale. A logarithmic curve, seen in Figure 7, was then fit 395
to the data from study 2 (the model curve). This allowed evaluation of the fit of averaged data for 396
each condition to the model curve. The best trend and magnitude of fit (Schunn & Wallach, 397
2005) were found for the As Seen condition (R2 = 0.99, RMSE = 0.27). This makes intuitive sense 398
as it was closest to what may be called a pure working memory test (i.e., place information in 399
working memory and repeat it back verbatim). The Reverse condition equaled the trend of the As 400
Seen condition but showed a slightly larger error value (R2 = 0.99, RMSE = 0.35). The values for 401
the Offset condition, though lower still, were not noticeably worse (R2 = 0.97, RMSE = 0.47) and 402
the estimated subjective workload ratings were, on average, within half a point of the observed 403
workload ratings. So, while increasing recruitment of differing cognitive resources does seem to 404
impact the accuracy of the model, in these tests working memory activation levels alone were 405
able to provide a very good prediction of the observed subjective ratings of mental difficulty and, 406
more to the point at hand, account for the subjective workload curve. 407

The Workload Curve 22
408
Figure 7. Model fit for problem rating as function of activation by condition 409
Conclusions 410
With hindsight it makes sense that, when it comes to mental workload, five is sometimes 411
closer to six than to four. In psychology many, if not most well established effects exhibit 412
curvilinear relationships. Fitts’s law (Fitts, 1954), Hick’s Law (Hick, 1952), the Yerkes-Dodson 413
Law (Yerkes & Dodson, 1908), Steven’s power law (Stevens, 1957), subitization (Jevons, 1871), 414
and the power law of practice (Newell & Rosenbloom, 1981) are all curvilinear processes as is 415
activation decay (Anderson, 2007; Byrne & Bovair, 1997; Oberauer et al., 2012) and the 416
probability of memory recall over time (Mueller & Krawitz, 2009; Bachelder, 2000; Taatgen, 417
2000). Those curves exist for a variety of reasons. Cognitive processing, for example, is thought 418
to happen along a curve for expediency; it is unnecessary and inefficient to process sensory input 419
linearly (Burns, 2014). In the case of workload, the modeling exercise conducted in this paper 420
supported the idea that the mental workload curve may bend as a result of the subject reflecting 421
on the difficulty they have maintaining information in working memory (i.e., availability). 422

The Workload Curve 23
Though perhaps intuitive, the existence of a subjective workload curve has, until now, 423
never been formally documented. And while this research needs to be replicated with 424
ecologically valid tasks, it acts as a starting point for refactoring our interpretation of subjective 425
ratings of workload. The activation model gives credence to the theory proposed here that the 426
judgements that give rise to the workload curve are based on the effects of activation and decay. 427
This suggests that practitioners who wish to manage the mental workload imposed by a tool must 428
manage not only the total amount of information the user must hold in working memory, but 429
how long it must be held there. 430
Where subjective measures are used to compare workload under differing conditions, the 431
workload curve indicates that the true magnitude of the difference is dependent on where on the 432
scale the ratings lie. The position of the rating on the scale may likewise tell us something about 433
the stability of the rating, as ratings in both studies became more sensitive under increasing 434
loads. 435
With regard to the impact of the workload curve, consider how widespread the use of 436
subjective workload ratings are in safety critical domains like aviation. The Journal of Aviation 437
Psychology, for example, publishes applied research related to aviation safety. Using their 438
online search capabilities I found that, of 566 articles, approximately 25% included measures of 439
subjective workload. One in ten included NASA-TLX specifically. For these practitioners who 440
so commonly use subjective workload ratings to ensure system safety a more thorough 441
understanding of subjective workload ratings is always of value. 442
Moving forward, several questions need to be answered. First, can the workload curve be 443
found in more complex mental work? Ecologically valid tests will be required to answer this 444
question, but they will be difficult as they require formulating and modeling working memory 445

The Workload Curve 24
processing in complex environments. To that end, the formulations found in the modeling 446
section here will be integrated into the Cogulator cognitive modeling tool (http://cogulator.io). 447
Second, does the curve seen in subjective ratings of mental workload exist in other 448
dimensions of workload? Some evidence of a temporal workload curve can be found in existing 449
studies (e.g., Dijksterhuis et al., 2013) and temporal workload will likely be the next dimension 450
of workload I investigate. Until those studies take place, I hope that others find the workload 451
curve useful in their applied work and add to it with their own research. 452
Acknowledgments 453
My thanks to Dr. Ronald Chong for his help in reviewing the experimental design, data, and 454
document. Thanks to Christopher DeSenti for brainstorming the concept and reviewing the 455
document. Thanks to John Helleberg for his thoughts on the concept, data, and design. Finally, 456
my thanks to Dr. Valerie Gawron and Dr. Kevin Burns for their feedback on the results. 457
Key Points 458
• Found evidence for the existence of a workload curve 459
• S-curves characterize the relationship between working memory load and subjective 460
ratings of workload 461
• We hypothesize subjective mental workload is driven by the availability of working 462
memory traces (activation) and our models support that as a plausible theory 463
464

The Workload Curve 25
References 465
Altmann, E. M. & Schunn, C. D. (2002). Integrating decay and interference: A new look at an 466
old interaction. In W. Gray & S. Schunn (Eds.), Proceedings of the 24th annual 467
conference of the Cognitive Science Society (pp. 65-70). Fairfax, Virginia: Cognitive 468
Science Society. 469
Altmann, E. M. & Trafton, J. G. (2002). Memory for goals: An activation-based model. 470
Cognitive Science, 26(1), 39-83. 471
Amichetti, N., Stanley, R., White, A., & Wingfield, A. (2013). Monitoring the capacity of 472
working memory: Executive control and effects of listening effort. Memory & Cognition, 473
41(6), 839-849. 474
Anderson, J. (2007). How can the mind occur in the physical universe? New York, New York: 475
Oxford University Press. 476
Anderson, J., Reder, L., & Lebiere, C. (1996). Working memory: Activation limitations on 477
retrieval. Cognitive Psychology, 30(3), 221-256. 478
Bachelder, B. (2000). The magical number 4 = 7: Span theory on capacity limitations. 479
Behavioral and Brain Sciences, 24(1), 87-185. 480
Baddeley, A. (1975). Word length and the structure of short-term memory. Journal of Verbal 481
Learning and Verbal Behavior, 14(6), 575-589. 482
Baddeley, A. (2007). Working memory, thought, and action. New York, New York: Oxford 483
University Press. 484
Barrouillet, P., Bernardin, S., & Camos, V. (2004). Time constraints and resource sharing in 485
adults’ working memory spans. Journal of Experimental Psychology: General, 133(1), 486
83–100. 487

The Workload Curve 26
Baud, M. (1993). Data analysis, mathematical modeling. In R.F. Masseyeff, W. Albert, & N.A. 488
Staines (Eds.). Methods of immunological analysis volume 1: Fundamentals. New York, 489
New York: VCH Publishers. 490
Berka, C., Levendowski, D.J., Lumicao, M.N., Yau, A., Davis, G., Zivkovic, V.T., Olmstead, 491
R.E., Tremoulet, P.D., & Craven, P.L. (2007). EEG correlates of task engagement and 492
mental workload in vigilance, learning, and memory tasks. Aviation, Space, and 493
Environmental Medicine, 78(5), 231-244. 494
Böhm, U. & Mehlhorn, K. (2009). The Influence of Spreading Activation on Memory Retrieval 495
in Sequential Diagnostic Reasoning. In A. Howes, D. Peebles, & R. Cooper (Eds.), 496
Proceedings of the 9th International Conference on Cognitive Modeling. Manchester, 497
United Kingdom. 498
Borst, J. P., & Anderson, J. R. (2013). Using model-based functional MRI to locate working 499
memory updates and declarative memory retrievals in the fronto-parietal network. 500
Proceedings of the National Academy of Sciences U.S.A., 110(5), 1628-1633. 501
Brown, J. (1958). Some Tests of the Decay Theory of Immediate Memory. Quarterly Journal of 502
Experimental Psychology, 10(1), 12-21. 503
Bunnell, J., Baken, D., & Richards-Ward, L. (1999). The effect of age on metamemory for 504
working memory. New Zealand Journal of Psychology, 28(1), 23-29. 505
Burke, B (2013). On the effect of coaching. Retrieved from 506
http://www.advancedfootballanalytics.com/2013/10/on-effect-of-coaching.html. 507
Burns, K. (2014) Entropy and optimality in abstract art: An empirical test of visual aesthetics. 508
Retrieved from http://www.ask-how.org/papers/Entropy%20and%20Optimality%20Pre-509
Print.pdf 510

The Workload Curve 27
Byrne, M, & Bovair, S. (1997). A Working memory model of a common procedural error. 511
Cognitive Science, 21(1), 31-61. 512
Card, S., Moran, T, & Newell, A. (1983). The psychology of human computer interaction. 513
Hillsdale, New Jersey: Lawrence Erlbaum Associates. 514
Cleary, M., Pisoni, D. B., & Geers, A. E. (2001). Some measures of verbal and spatial working 515
memory in eight- and nine-year-old hearing-impaired children with cochlear implants. 516
Ear & Hearing, 22, 395–411. 517
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, New Jersey: 518
Lawrence Erlbaum Associates. 519
Cooper, G. & Harper, R. (1969). The use of pilot rating in the evaluation of aircraft handling 520
qualities (Technical Report TN D-5153). NASA. 521
Cowan, N. (2001). The magical number 4 in short-term memory: a reconsideration of mental 522
storage capacity. Behavioral and Brain Sciences, 24(1), 87-114. 523
Daryanian, B. (1980). Subjective scaling of mental workload multi-task environment 524
(Unpublished master's thesis). MIT, Cambridge, Massachusetts. 525
Dijksterhuis, C. de Waard, D., Brookhuis, K., Mulder, B., & de Jong, R. (2013). Classifying 526
visuomotor workload in a driving simulator using subject specific spatial brain patterns. 527
Frontiers in Neuroscience, 7(149). Retrieved from 528
http://journal.frontiersin.org/article/10.3389/fnins.2013.00149/full 529
Eggemeier, F.T., Crabtree, M.S., & Reid, G.B. (1982). Subjective workload assessment in a 530
memory update task. Proceedings of the Human Factors Society Annual Meeting, 26, 531
643-647. 532
Eggemeier, F.T., & Wilson, G.F. (1991). Performance-based and subjective assessment of 533

The Workload Curve 28
workload in multi-task environments. In D.L. Damos (Ed.), Multiple-task performance 534
(pp. 217-278). London: Taylor and Francis. 535
Ericsson, K.A. & Simon, H.A. (1980) Verbal reports as data. Psychological Review, 87(3), 215-536
251. 537
Fitts, P. (1954). The information capacity of the human motor system in controlling the 538
amplitude of movement. Journal of Experimental Psychology, 47(6), 381–391. 539
Friedman, H. & Amoo, T. (1999). Rating the rating scales. Journal of Marketing Management, 540
9(3), 114-123. 541
Gawron, V.J. (2008). Human performance, workload, and situational awareness measures 542
Handbook. Boca Raton, Florida: CRC. 543
Gendle, M., & Ransom, M. (2006). Use of the electronic game Simon as a measure of working 544
memory span in college age adults. Journal of Behavioral and Neuroscience Research, 545
4, 1-7. 546
Gopher, D., & Braune, R. (1984). On the psychophysics of workload: Why bother with 547
subjective measures? Human Factors, 26(5), 519-532. 548
Halamish, V., McGillivray, S., & Castel, A. (2011). Monitoring one’s own forgetting in younger 549
and older adults. Psychology and Aging, 26(3), 631-635. 550
Hancock, P.A., & Caird, J.K. (1993). Experimental evaluation of a model of mental workload. 551
Human Factors, 35(3), 413–429. 552
Hart, S. G. (2006). NASA-Task Load Index (NASA-TLX); 20 Years Later. Proceedings of the 553
Human Factors and Ergonomics Society, 26, 904-908. 554
Hart, S.G. & Staveland, L.E. (1988). Development of NASA-TLX (Task Load Index): Results of 555
empirical and theoretical research. In P.A. Hancock & N. Meshkati (Eds.), Human mental 556

The Workload Curve 29
workload (pp. 139-183). Amsterdam: North-Holland. 557
Hassin, R. R., Bargh, J. A., Engell, A., & McCulluch, K. C. (2009). Implicit Working Memory. 558
Consciousness and Cognition, 189(3), 665-678. 559
Hauser, J. R., Childress, M. E., & Hart, S. G. (1982). Rating consistency and component salience 560
in subjective workload estimation. In F. George (Ed.), Proceedings of the 18th Annual 561
Conference on Manual Control, 127-149. 562
Hess, R. (1971). The use of a nonadjectival, nonordinal, linear rating scale in a single axis 563
compensatory tracking task. (Unpublished master's thesis). Naval Postgraduate School, 564
Monterey, California. 565
Hick, W. E. (1952). On the rate of gain of information. Quarterly Journal of Experimental 566
Psychology, 4(1), 11–26. 567
Humes, L. & Flyod, S. (2005). Measures of working memory, sequence learning, and speech 568
recognition in the elderly. Journal of Speech, Language, and Hearing Research, 48, 224-569
235. 570
Jevons, W. S. (1871). The power of numerical discrimination. Nature, 3, 281-282. 571
Just, M.A. & Carpenter, P.A. (1992). A capacity theory of comprehension: individual differences 572
in working memory. Psychological Review, 99(1), 122–149. 573
Kahana, M. (2012). Foundations of human memory. New York, NY: Oxford University Press. 574
Kornell, N., Rhodes, M., Castel, A., & Tauber, S. (2011). The ease-of-processing heuristic and 575
stability bias: dissociating memory, memory beliefs, and memory judgements. 576
Psychological Science, 22(6), 787-794. 577
Lotka, A. (1925). Elements of physical biology. Baltimore, MD: Williams & Wilkins Co. 578
Miller, G. (1956). The magical number seven, plus or minus two: Some limits on our capacity 579

The Workload Curve 30
for processing information. The Psychological Review, 63, 81-97. 580
Moray, N., Johansen, J., Pew, R.W., Rasmussen, J., Sanders, A.F., & Wickens, C.D. (1979). 581
Mental workload: Its theory and measurement. New York, New York: Plenum Press. 582
Morton, J., Crowder, R., & Prussin, H. (1971). Experiments with the stimulus suffix effect. 583
Journal of Experimental Psychology, 91(1), 169-190. 584
Mueller, S., & Krawitz, A. (2009). Reconsidering the two-second decay hypothesis in verbal 585
working memory. Journal of Mathematical Psychology, 53(1), 14-25. 586
Newell, A. (1973). You can’t play 20 questions with nature and win: Projective comments on the 587
papers of this symposium. In W. G. Chase (Ed.), Visual Information Processing (pp. 283-588
308). New York: Academic Press. 589
Newell, A., & Rosenbloom, P. S. (1981). Mechanisms of skill acquisition and the law of 590
practice. In J. R. Anderson (Ed.), Cognitive skills and their acquisition. Hillsdale, NJ: 591
Erlbaum. 592
Oberauer, K., Lewandowsky, S., Farrell, S., Jarrold, C., & Greaves, M. (2012). Modeling 593
working memory: An interference model of complex span. Psychonomic Bulletin & 594
Review, 19(5), 779-819. 595
Pape, N., & Urbas, L. (2008). A model of time-estimation considering working memory 596
demands. In Proceedings of the 30th Annual Cognitive Science Society (pp. 1543-1548). 597
Austin, TX: Cognitive Science Society. 598
Pliska, V. (1987). Dose-response models: similarity with population growth dynamics. Trends in 599
Pharmacological Sciences, 8(2), 50-52. 600
Reid, G.B., & Nygren, T.E. (1988). The subjective workload assessment technique: A scaling 601
procedure for measuring mental workload. In P.A. Hancock & N. Meshkati (Eds.), 602

The Workload Curve 31
Human Mental Workload. Amsterdam: Elsevier. 603
Rubio, S., Diaz, E., Martin, J., & Puente, J. M. (2004) Evaluation of subjective mental workload: 604
A comparison of SWAT, NASA-TLX, and Workload Profile. Applied Psychology: An 605
International Review, 53(1), 61-86. 606
Schunn, C. D. & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to 607
data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der 608
Emeritierung von Werner Tack (115-154). Saarbrueken, Germany: University of 609
Saarland Press. 610
Sheridan, T.B. & Simpson, R.W. (1979). Toward the definition and measurement of the mental 611
workload of transport pilots (Technical Report No. R 79-4). MIT Flight Transportation 612
Laboratory. 613
Simon, H.A. (1969). The sciences of the artificial. Cambridge, Mass.: MIT Press. 614
Sohn, M.H., Anderson, J. R., Reder, L. M., & Goode, A. (2004) Differential fan effect and 615
attentional focus. Psychonomic Bulletin and Review, 11(4), 729-734. 616
Stevens, S.S. (1957). "On the psychophysical law". Psychological Review, 64(3), 153–181. 617
Taatgen, N. (2000). Dispelling the magic: Towards memory without capacity. Behavioral and 618
Brain Sciences, 24(1), 87-185. 619
Tryon, W. (2001). Evaluating Statistical Difference, Equivalence, and Indeterminacy Using 620
Inferential Confidence Intervals: An integrated alternative method of conducting null 621
hypothesis statistical tests. Psychological Methods, 6(4), 371-386. 622
Tulga, M. K., &.Sheridan, T. B. (1980). Dynamics decision and workload in multitask 623
supervisory control. IEEE Transactions on Systems, Man, and Cybernetics, SMC-10, 624
217-231. 625

The Workload Curve 32
Vidulich, M.A., (1988). The cognitive psychology of subjective mental workload. In P.A. 626
Hancock & N. Meshkati (Eds.), Human Mental Workload (pp. 219–229). New York, 627
New York: Elsevier. 628
Vidulich, M.A. & Tsang, P. (1985). Assessing subjective workload assessment: A comparison of 629
SWAT and the NASA-Bipolar methods. Proceedings of the Human Factors Society, 29, 630
71-75. 631
Wickens, C. D. (2008). Multiple Resources and Mental Workload. Human Factors, (50)3, 449-632
455. 633
Wierwille, W. W., & Casali, J.G. (1983). A validated rating scale for global mental workload 634
measurement applications. Proceedings of the Human Factors Society, 27, 129-133. 635
Wierwille, W. W., & Williges, R.C. (1978). Survey and analysis of operator workload 636
assessment techniques (Technical Report No. S-78-101). Systemetrics, Inc. 637
Yeh, Y.Y. & Wickens, C.D. (1988). Dissociation of performance and subjective measures of 638
workload. Human Factors, 30(1), 111-120. 639
Yerkes R. & Dodson J. (1908). The relation of strength of stimulus to rapidity of habit-640
formation. Journal of Comparative Neurology and Psychology, 18(5), 459–482. 641
Young, M. S., Brookhuis, K. A., Wickens, C. D., & Hancock, P. A. (2015). State of science: 642
Mental workload in ergonomics. Ergonomics, 58, 1-17. 643
644
645
Biography 646
Steven L. Estes lives in Savannah, Georgia. He holds a BA in History (1996) from the 647
University of Georgia in Athens, Georgia and an MA in Human Factors and Applied Cognition 648

The Workload Curve 33
(2002) from George Mason University in Fairfax, Virginia. He is currently a Principal Human 649
Factors Engineer at the MITRE Corporation’s Center for Advanced Aviation System Design in 650
McLean, Virginia. Prior to working for MITRE, he was employed as a human factors engineer at 651
Gulfstream Aerospace. Publications include the book chapter “Macrocognition in systems 652
engineering: supporting changes in the air traffic control tower”, published in the book 653
Naturalistic Decision Making and Macrocognition (Burlington, VT: Ashgate Publishing 654
Company, 2008). He is the developer of Cogulator, an applied tool for workload assessment and 655
task time estimation (http://cogulator.io). Research interests include cognitive engineering and 656
human computer interface design. 657