Content uploaded by Max Mulder
Author content
All content in this area was uploaded by Max Mulder on Dec 15, 2022
Content may be subject to copyright.
Solution Space Decision Support for Reducing
Controller Workload in a Route Merging Task
G. A. Mercado Velasco*, C. Borst†, M. M. van Paassen‡, M. Mulder§
Faculty of Aerospace Engineering, Delft University of Technology,
2600 GB Delft, The Netherlands
Air traffic controller workload is considered to be a limiting factor for further air
traffic growth. To reduce workload, increased automation levels and novel decision-
support tools are being investigated. This paper describes the adaptation and eval-
uation of a previously-developed interface, called the “Solution Space Diagram”, in
a route merging task. It portrays both constrained and unconstrained speed and
heading combinations and enables the controller, by means of direct manipulation,
to safely vector aircraft. We hypothesized that this interface enables controllers to
use it in their own preferred way, supporting their skills and strategies, reducing
their workload. A preliminary experiment was conducted in which twelve partici-
pants, grouped according to expertise level, controlled a sector and were faced with
different levels of traffic in a route merging task. Results show that the interface
aids in finding merging solutions faster: a significant reduction in the number of
commands and in perceived workload was observed. Our participants changed
their strategy to perform less vectoring and issue route interceptions at an earlier
stage, without affecting aircraft separation. These changes were also observed with
the professional controllers, although they showed to be more conservative to the
use of the diagram. This study justifies experimentation with a larger number of
participants and in a setup of higher operational realism.
I. Introduction
Predicted air traffic growth and pressing economic and environmental concerns are forcing a
fundamental redesign of the air traffic management (ATM) system, to increase airspace capacity
*Research Associate, Control and Simulation; g.a.mercadovelasco@tudelft.nl
†Assistant Professor, Control and Simulation; c.borst@tudelft.nl
‡Associate Professor, Control and Simulation; m.m.vanpaassen@tudelft.nl
§Professor, Control and Simulation; m.mulder@tudelft.nl. Associate Fellow AIAA
1
American Institute of Aeronautics and Astronautics
PLEASE REFER TO THIS PAPER AS
G. A. Mercado-Velasco, C. Borst, M. M. Van Paassen, and M. Mulder, “Solution Space Decision Support for
Reducing Controller Workload in Route Merging Task,” Journal of Aircraft, vol. 58, no. 1, pp. 125–137, 2021.
with higher levels of flight efficiency and operational safety. Air traffic controller workload is
one of the main factors that limits the safe and expeditious growth of air transport [1, 2, 3]. Un-
derstanding how task complexities (e.g., airspace organization) and decision-support tools (e.g.,
automation, including the human-machine interface) can affect controller workload is one of the
main research questions addressed in current ATM modernization efforts [4, 5, 6].
Hilburn and Jorna [7] stated that controller workload depends on several system factors as well
as on operator factors, see Figure 1. Whereas many studies are addressing workload by manipulat-
ing the task demand load [8] by, for example, clever dynamic sectorization [9] and airspace design
(e.g., change existing routing structures and sector geometries) [10], other studies investigate ways
to exploit new technologies that better support controllers in their tasks [11, 12, 13, 14, 15, 15, 16,
17, 18]. Especially the latter is most challenging in the Air Traffic Control (ATC) domain, [19]
given the many failed attempts in the past (see Ref. [20] for an extensive overview).
According to Westin et al. [20], controller acceptance plays a critical role in embracing new
technologies, where acceptance is driven by how much the support tool conforms or “matches”
with the skills and strategies of humans. Research on cockpit automation, amongst others, high-
lights also several other factors, such as the reliability, ease of use, false alarm rates, intuitiveness
vs. opaqueness, that play a role in an operator’s trust and acceptance of automation [21].
A successful decision support tool that can keep workload at acceptable levels would be one
that not only allows controllers to cope with task complexities, but also takes into account their
skills, experience and strategies, see Figure 1. In contrast, a support tool that works against a
controller will likely lead to more cognitive load in trying to understand what it is doing and why,
increasing workload. Thus the challenge is how to design a decision support tool that leverages
human and automation abilities, whilst accounting for the controllers’ skills and strategies. To
tackle this challenge, two research paths can be distinguished.
On the one hand, decision support tools that provide conflict resolution advisories have hinted
to be effective in reducing workload [22, 23]. Some initiatives (for a survey of many studies, see
Ref. [20]) have incorporated human heuristics as an “externalized mental library” that can be used
to provide solution recommendations after an algorithm has identified a potential conflict. Since
the suggested solutions might follow the controller’s expectation, it allows her to infer why the so-
2
American Institute of Aeronautics and Astronautics
Taskload
Workload
System
dynamics
Interface
demands
Task
complexity
etc.
Skill
Strategy
Experience
etc.
System factors Operator factors
Figure 1. Factors affecting controller workload (adapted from Hilburn and Jorna, [7]).
lution was proposed, which ultimately increases the acceptance of the support tool. Nevertheless,
it is challenging to identify a strategy that is acceptable to all controllers in scenarios where non-
homogeneous problem-solving strategies are being used [24]. Tailoring the advisories to controller
preferences may lead to a conservative system, reinforcing sub-optimal strategies. And by provid-
ing a limited set of possible solutions, the system may hinder users in exploring other alternatives
and preclude the full exploitation of the human unique problem-solving capabilities. Under-using
these capabilities can ultimately lead to complacency, over-trust, and skill loss.
On the other hand, support tools that show the constraints imposed by the work domain, often
in the form of “no-go zones” [25], can have a positive effect on reducing workload, while avoiding
the adverse effects of complacency. Since operators are encouraged to think through multiple
possibilities and are not confined to focus on a limited subset of relevant information, they can
remain purposely engaged in problem-solving, and complacency-related issues are likely to be
avoided [26]. The problem, however, lies in the complexity that results from integrating several
variables to present an overview of all possible solutions to the user.
The Highly Interactive Problem Solver (HIPS), for example, developed by Eurocontrol in the
1990’s, made use of no-go zones in three decision support tools that displayed conflict situations
relative to one selected aircraft, to help the controller find solutions to these conflicts without
presenting explicit resolution advisories. [27, 28] These three tools were projections of conflict
situations on horizontal, altitude, and time/speed displays. The altitude display is now known as
the Level Assessment Display (the LAD, currently implemented in the London Area Control man-
agement system,[29, 30]) but the other two displays were not adopted as initially designed.
Possibly, the way in which information was provided in the three displays resulted in a concep-
3
American Institute of Aeronautics and Astronautics
tual and usability issue. In the face of a traffic conflict, controllers had to decide for an instruction
that might be a change in altitude, speed, heading, or a combination of these. If users must acquire,
process, and integrate all the information needed to provide a resolution from different information
displays, they might not be able to comprehend the system status directly. More information dis-
plays mean more possible data relationships that need to be adequately understood, or the operator
may not always be capable (e.g., in high workload situations) to perceive the direct connection
with the physical system (the airspace, aircraft, routes, etc.). Some evaluations of HIPS identified
that controllers needed more explanation to understand that they were working with constraints
and not trajectories. [27, 28, 31] Nonetheless, the successful adoption of one of the HIPS displays,
i.e., the LAD, is a merit of the design philosophy, and shows the importance of the integration of
information in a way that is transparent and easy to access.
This study evaluates the capabilities of a tool also based on the no-go zones principle, called
the Solution Space Diagram (SSD), to assess whether its potential as tactical support tool could
motivate further research efforts. The SSD provides a visualization of all conflict-free velocity
vectors within the performance envelope of a selected aircraft. [32]. It integrates several sources
of information to present no-go zones in a velocity display. By allowing direct manipulation, the
controller can use the SSD to formulate and implement speed and heading clearances that avoid or
resolve separation conflicts. The SSD does not present advisories to controllers (although it can do
so as well, see [33]), but rather shows controllers all possible solutions, supporting their individual
skills, strategies and preferences, which may positively impact their acceptance.
Previous studies in ATC [34, 35, 36] have evaluated the capabilities of the SSD concept for
estimating metrics for airspace complexity and controller workload. Several SSD-based interfaces,
in two and three dimensions, have emerged for ATC [37, 38] and some have been successfully
evaluated as a decision-support tool for pilots, in self-separation tasks [39, 40, 41]. Recently, the
basic, state-based, SSD has been evaluated in studies focusing on basic ATC conflict detection
and resolution (CD&R). [42, 43] Using a novel intent-based SSD as a decision-support tool in the
air traffic controller’s merging task has not been reported before, and its effects on workload and
performance are yet unknown. These effects may not all be positive; the SSD integrates several
sources of information to depict the full “solution space”, including intent, in support of CD&R,
4
American Institute of Aeronautics and Astronautics
but in the merging task its novel presentation could induce more workload than it alleviates.
The research presented in this paper is a first investigation of the capabilities of the SSD to
support controllers in a route merging task. The main purpose was to assess the diagram’s effects
on controller workload and merging performance. A secondary purpose was to evaluate whether
the SSD allows controllers, with different levels of expertise, to use the tool in their own preferred
way. An analysis of the impact of the SSD on the different controllers’ control strategies could
provide unique insights, possibly explaining any observed changes in workload.
The evaluation of the SSD as ATC interface is performed in a medium-fidelity simulator in
which controllers performed two-dimensional CD&R in the context of a spacing and route merging
task. Our study does not benchmark the tool’s performance with other support interfaces, and does
not evaluate the effects that could be measured after a prolongued use of the tool, such us overtrust,
complacency, or skill loss. Controller acceptance will also not be a part of the current evaluation,
but is likely to become a major dependent measure when the evaluation is successful, allowing for
a larger group of (experienced) controllers to be invited.
In this context, a series of human-in-the-loop simulations were performed, in which participants
with three different levels of expertise in ATC (ranging from university students to experienced air
traffic controllers) performed an aircraft merging task, with and without the help of the SSD. The
main hypothesis was that, because the SSD makes the limitations imposed by the work domain
visible to the operator, she can in turn directly perceive the “space of solutions”, leading to a lower
workload. Second, the evaluation investigated whether the SSD is able to support the control strate-
gies adopted by the three groups of controllers. Third, we investigated whether showing the SSD
led to changes in these control strategies which could account for the observed changes in work-
load. The analysis of the effects on workload is performed with instantaneous self-assessments,
measured throughout the simulation runs, and also using quantitative performance-related metrics.
The paper is structured as follows. Section II describes the rationale of the SSD and motivates
its use as a decision aid. The simulator used in the evaluation is discussed in Section III, followed
by a description of the experiment in Section IV. Results are described in Section V, followed by
an extensive discussion in Section VI; the paper ends with conclusions in Section VII.
5
American Institute of Aeronautics and Astronautics
II. The solution space diagram as a decision aid
A. Construction of the solution space diagram
Having its foundations in the Velocity Obstacle theory [44], the solution space diagram (SSD) is a
two-dimensional representation that covers all heading/speed combinations possible for a specific
aircraft, indicating which velocity vectors offer safe solutions and which velocity vectors lead to a
loss of separation with another aircraft [32]. A description of the classical VO theory in its simplest
form (two vehicles following two-dimensional rectilinear trajectories at constant speed) provides
sufficient background information for understanding the construction of the diagram.
VOA|B
PZB
pA
pB
vA
vB
−vB
vrel
a) Velocity Obstacle for aircraft A(the controlled
aircraft), imposed by aircraft B(VOA|B).
VOA|B
Vmin
Vmax
vA
vB
b) Solution space diagram for aircraft A.
Figure 2. Basic solution space construction.
A Velocity Obstacle (VO) is defined as the set of all velocity vectors of a moving vehicle that
will result in a collision (or a loss of separation) with a moving obstacle at some moment in time,
assuming that the moving obstacle maintains a constant velocity vector (adapted from Ref. [45]).
Consider a controlled vehicle A, at position pA, and an observed (moving) obstacle B, at posi-
tion pB, with circular protected zone PZB, see Figure 2a. If a ray starting at pA, coinciding with the
relative velocity vector of Awith respect to B(vrel =vA-vB) intersects PZB, then vAis in the Veloc-
ity Obstacle of vehicle A, imposed by obstacle B(adapted from Ref. [46]). This velocity obstacle,
denoted VOA|B, represents the set of velocities of vehicle A, that will result in a loss of separation
with obstacle B. Figure 2a shows graphically how VOA|Bcan be derived. A relative velocity cone
can be constructed from the traffic geometry, with its edges originating in pA, and tangent to PZB.
VOA|Bis then determined by adding the velocity of Bto the relative velocity cone. Such cone can
then be drawn in the velocity plane of a solution space diagram, as depicted in Figure 2b. Note
6
American Institute of Aeronautics and Astronautics
that the large and small circles represent the aircraft performance limits, i.e., the maximum and
minimum speed, respectively, of the aircraft under control.
The SSD for the selected aircraft Ashows which combinations of heading and speed will lead
to a loss of separation with B, the shaded cone. Vice versa, it also shows which heading and
speed combinations are ‘free’ to select, when there is a reason to change the vehicle’s heading or
speed, such as in vectoring the aircraft as done by ATC. In the situation illustrated in Figure 2b the
selected aircraft Ais flying close to its minimum velocity (the end-point of the vector vAlies close
to the minimum velocity circle), and the aircraft will ‘pass behind’ the faster aircraft B. The reader
is referred to Refs. [44, 47] for a more detailed discussion of the SSD elements and use, and to
Refs. [32, 40, 48] for the theoretical underpinnings of our displays.
When intended trajectories are not rectilinear paths, or contain speed changes, the calculation of
the VO is not as straightforward. Whereas D’Engelbronner et al. [35] proposed an approximation,
Mercado et al. [44] derived a closed mathematical form for calculating VOs that take intended
trajectory changes (in piecewise or continuous form) into account. Figure 3 illustrates the effects
that such intent information induces on the VO, and, therefore, on the SSD. This latter, intent-
based SSD is the subject of the current paper. Note that what would initially be regarded as a
conflict under the classical VO theory (Figure 3b), is no longer a conflict when intent information
is incorporated (Figure 3c).
B
A
a) Aircraft Bsharing intent infor-
mation with A.
vA
b) SSD for Awithout the intent in-
formation of B.
vA
c) SSD for Awith the intent infor-
mation of B.
Figure 3. Effects of including intent information on the SSD.
For traffic situations that include more than two aircraft, the SSD for a controlled aircraft must
contain the VOs induced by all other observed aircraft that are close. Figure 4 shows an example
traffic situation with the corresponding SSD, drawn for one particular aircraft that is under control
of the ATCo. In this example, the aircraft under control (A) is surrounded by two other aircraft (B
and C). Two conflict zones are shown, and since aircraft Band Cmerge on the same trajectory,
7
American Institute of Aeronautics and Astronautics
perpendicular to the controlled aircraft’s flight, the conflict zones point to the same origin. This is
because, after the merge, aircraft Band Chave the same velocity vector, i.e., the same direction
and speed. Finally, note that the SSD shows the situation for just one aircraft; when another aircraft
is selected, like Bor C, the picture will generally look very different.
B
C
A
a) Traffic situation with aircraft Band Csharing
their intent information with A.
Vmin
Vmax
vA
b) Solution space diagram for aircraft A.
Figure 4. The SSD for a multi-aircraft scenario.
B. Decision support
The SSD could be seen as a “what-if” aid for tactical CD&R. However, it differs from some other
tools of this sort, since it does not force the operator out of the loop for projecting and analyzing
user- (or computer-) initiated hypothetical situations until a satisfactory solution is found. Instead,
it makes use of no-go zones to communicate whether the selected aircraft’s current or tentative
velocity vector would conflict with the trajectories that the aircraft nearby intend to fly, leaving
the planning task (the basis of the strategy to undertake) completely up to the operator. With the
implementation presented in this study, the controller remains fully responsible for the decision-
making and action implementation tasks.
Leaving all decision-making in the hands of the operator would also mean that the operator
can develop her or his own preferred strategy to perform the task, such as the merging task in
this study, with or without the help of the SSD. For instance, operators with little experience
in doing the task could build their expertise – and rely more upon – the SSD, as it simplifies
the information collection and integration stages needed to understand the traffic situation under
8
American Institute of Aeronautics and Astronautics
control. In contrast, highly experienced controllers, who have learned to perform the merging task
without the help of any tool, can be assumed to already have the required expertise, and would
rely less on the SSD but could use it as a tool that could “confirm” their view on the situation and
the expected effects of their control actions. Tentatively, one can expect that the SSD will have
different effects on the different control strategies by experienced and less-experienced controllers,
but its success could lie in the fact that it supports them all in doing their job in each individual’s
preferred way.
Nevertheless, supporting information analysis comes with a risk. Providing the user with more
information requires additional cognitive effort for its processing, and increased visual display de-
mands are known to also possibly induce an increment in workload [49]. The authors expected
the SSD not to take the controller out of the loop in a way that would demand more cognitive re-
sources, however, the effects that the diagram can have on controller workload need to be carefully
studied. For this purpose, and to study other elements of SSD usage in the future (such as operator
acceptance), a medium-fidelity simulator has been developed, described in the following section.
III. Simulator setup
The main hypothesis explored in this paper is that presenting the SSD to support CD&R and
merging tasks in the horizontal plane reduces controller workload. A medium-fidelity simulator
was developed (in Matlab™) to test this central hypothesis. Figure 5 shows a screenshot of the
simulator layout. Note that the coloring and font sizes have been changed so that the image could
be printed out with more clarity.
The simulator included a Plan View Display (PVD) of the airspace on the left-hand side
(marked by 1) and the SSD at the top right (marked by 2). In some scenarios, the VOs of
the SSD were shown, in others they were hidden, but in both cases, the SSD interface served as
direct manipulation interface for heading and speed commands. At the lower right, the interface
provided a set of virtual buttons (marked by 3) that the participants could use to give instructions
to the aircraft within the sector (sector colored in white). These buttons served as “shortcuts” for
Direct-to and Intercept clearances. A mouse was used for all interactions.
9
American Institute of Aeronautics and Astronautics
Figure 5. Experimental simulator screenshot.
Participants were expected to interact with the simulator in the following way. They had to
select the aircraft they were interested in controlling by clicking on it, and the SSD would be
updated to contain all the VOs that all other aircraft were imposing on the selected aircraft at that
moment. The SSD was then updated every second until the aircraft was deselected, or another
aircraft was selected. While an aircraft remained selected, the participant could click on the SSD
to change the aircraft’s velocity vector, having a clear picture of which velocity vectors would or
would not create conflicts. Later, the participant could decide to merge the aircraft with the given
velocity vector by clicking on the Intercept button. Alternatively, the participant could observe if
changing the heading towards a particular waypoint was a conflict-free possibility or not, and then
possibly decide to click on one of the Direct-to buttons.
In scenarios in which no support was provided, the way of interacting with the simulator was
similar; the single difference was that VOs were not displayed at all. That is, when selecting an
aircraft, the participant could still click on the SSD interface to change the aircraft’s velocity vector,
or use the Direct-to buttons, but no VOs were shown and the operator had to judge herself whether
the control actions were valid.
The setup explained here contains certain elements that may drive the experiment away from
10
American Institute of Aeronautics and Astronautics
the reality of the ATC task it was meant to emulate. The simplification has the benefit, however,
of reducing training time and learning effects. Since the objective of this study is to evaluate a
tool meant to support merging and separation in the horizontal plane, the simulator did not allow
participants to issue altitude changes. Furthermore, they did not have to maintain flight strips, deal
with aircraft deviating from their prescribed trajectories, or having to communicate verbally.
A. Airspace and aircraft
The airspace presented contained the sector to be controlled, inside which aircraft could receive
commands. Although also the aircraft surrounding the sector were shown, they could only be con-
trolled when they entered the sector. The route to which aircraft were to be merged was indicated
on the PVD, together with the route points’ names (see Figure 5). The last point of the route
(HOOKS) served as the sector’s exit point.
Two types of aircraft were present in the simulation, “heavy” and “light”, with speed ranges
160-250 knots and 120-250 knots, respectively. Different sizes of aircraft icons were displayed on
the PVD to provide the notion of different aircraft types.
Different colors were used for the aircraft symbols. When a loss of separation would take place,
the aircraft involved turned red. After a Direct-to or Intercept Route command was issued, the color
of the data tag would turn from yellow to green. In other words, a yellow tag meant that the aircraft
has still not received a route merging instruction; a green tag implied that the aircraft did receive
a merging instruction. The selected aircraft would also show a circle around it, representing the 5
NM separation radius assumed for this research. The aircraft tags contained the aircraft callsign,
the current and target speed, and the current and target heading.
B. The solution space diagram implementation
Two important visual dynamic elements were provided in the SSD: target and current velocity
vectors (to give the notion of the transition between states), and color-coded conflict areas (VOs).
Different tones of blue were used for VOs that corresponded to aircraft that had already received
merging instructions, and different tones of gray for the VOs that corresponded to aircraft that still
had not received such directions.
11
American Institute of Aeronautics and Astronautics
This color-coding was adopted to allow the controller to be more selective on which VOs to
consider when issuing merging clearances. The SSD can become almost entirely covered in high
traffic scenarios and, without the color coding, the complex lumped area that results from all the
VOs may leave the controller with very few choices and possibly lead her or him to refrain from
using the aid. With the color coding, the controller can consider vector commands that would
result in conflicts with other aircraft that have not yet received merging instructions because these
other aircraft would eventually receive additional instructions as well. Note that several properties
of a VO (e.g., its orientation in the SSD, the location of its tip and the width of the VO) enable a
controller to link the different VOs to corresponding aircraft on the PVD.
The SSD also acted as direct manipulation interface for the controller. By a mouse click inside
the minimum and maximum speed circles, the corresponding vector command was issued to the
selected aircraft. In conditions when the VOs were not shown, the same command interface was
used. This interface setup was selected because other types of input would add an extra task to
the controller. It is of great importance to maintain interface and equipment demands low, since
the main task of the controller is to keep the “mental picture” of the traffic situation, and, already
by including the diagram, a momentary diversion of the controller’s attention from the PVD is
introduced. Separate buttons were used to enter route intercept and direct to waypoint commands.
C. Route intercept buttons
The interface included four additional buttons in the lower right corner, see Figure 5. One of these
was the Intercept Route button and the other three were Direct-to buttons for different waypoints.
These could be used to direct aircraft either to intercept the route (i.e., maintain current heading
and speed until the route is crossed, then continue along the route) or to fly to a particular route
point and from then on stay on the route. When one of these buttons was clicked, the selected
aircraft tag would turn from yellow to green, to indicate that that particular aircraft had received
merging instructions.
12
American Institute of Aeronautics and Astronautics
D. Instantaneous Self Assessment
Every decision a controller takes has its consequences in the future, which means that the task,
and in fact the experiment as a whole, has a dynamic nature. The workload experienced by the
controller will vary in time, and each controller will experience a different workload. Hence, when
workload is measured, it must be done at various points in time (see also Ref. [35]).
From the many different developed techniques for subjective workload determination, the In-
stantaneous Self-Assessment (ISA) method is one of the simplest tools with which an estimate
of perceived workload can be obtained during real-time simulations or actual tasks [50]. This
method requires the operator to give a discrete rating between 1 (very low) and 5 (very high) of the
workload she/he perceives, either verbally or by using a keyboard.
This method has shown to be highly correlated with the NASA Task Load Index and other
workload measures [51]. It is also easy to implement and has low intrusiveness. It was hence
considered to be well suited for the current experiment. Every 60 seconds, a red blinking message
displayed on the PVD (in an area not taken up by the visualization of traffic) requested the con-
troller to subjectively rate his/her workload by pressing the relevant number key on the keyboard.
IV. Experiment
A. Experiment goal
The capabilities of the SSD as decision support tool in the tactical CD&R and merging task were
explored through a series of human-in-the-loop simulations, including participants with three levels
of ATC expertise. The main goal of the experiment was to determine whether presenting the SSD
VOs to controllers can alleviate their workload when faced with the task of merging aircraft into a
single route, by giving heading and velocity commands. Additionally, to facilitate an analysis of
changes in the adopted control strategies of the three participant groups, the effects of the SSD on
performance indicators such as the number (and type) of issued commands, aircraft in the sector,
and separation violations were investigated.
13
American Institute of Aeronautics and Astronautics
B. Experiment Design
Two independent variables were present in the experiment, each with two levels: the solution space
display (On/Off) and traffic(Low/High). When the SSD was “On”, all the VOs imposed by aircraft
close to the controlled aircraft were shown, as illustrated in Figure 5. Participants could control
the aircraft using the SSD interface, as discussed in the previous section. When the SSD display
was “Off”, the interaction functionality remained the same, but the VOs were not shown. The
differences in traffic levels are explained later on.
A mixed-design was used. Participants belonged to a specific population group (between-
subjects variable) and performed once in every experimental condition (within-subjects variable).
The sequence of the experimental runs was established by a Latin Square design for a four treat-
ment experiment to counterbalance carryover effects such as practice and fatigue. Table 1 shows
these sequences.
Table 1. Orthogonal Latin square design for four treatments.
Sequence Treatment
1234
1 A B C D
2 B A D C
3 C D A B
4 D C B A
5 A D B C
6 B C A D
7 C B D A
8 D A C B
9 A C D B
10 B D C A
11 C A B D
12 D B A C
It is evident from Table 1 that twelve (or a multiple of twelve) participants are needed to have
a balanced design for a four treatment experiment. Each of the twelve participants was randomly
assigned to a sequence from Table 1.
C. Scenarios
1. Simulation speed and time
In an attempt to gather more information on the dynamics of events, avoid under-achievement from
the controllers, and following the procedure of previous experiments [34, 35], a faster-than-real-
time simulation was run, i.e., four times as fast as real-time. With this simulation speed, every
14
American Institute of Aeronautics and Astronautics
scenario ran for 20 minutes of real time (80 minutes of simulated time), with some break between
scenarios.
2. Traffic conditions
Incoming aircraft streams were located at three fixed locations outside the sector. As depicted in
Figure 5, aircraft came in from either the northwest, northeast, or southwest. The sector geometry
and traffic routing structure are representative for a realistic en-route sector, except that all aircraft
fly at the same altitude. A new aircraft was created at one of these fixed locations (randomly
selected) every 200 seconds for the low traffic condition, and every 150 seconds for the high traffic
condition, expressed in simulated time. These aircraft influx rates also represent realistic values
for en-route sectors.
The initial traffic condition was identical in scenarios of same traffic level, showing 11 aircraft
in the low-intensity condition and 13 in the high-intensity condition. Preliminary tests showed
that these number of aircraft were appropriate to rapidly achieve a steady number of aircraft in the
sector, thus reducing the duration of transition effects.
3. Solution space prediction time and geometry of the sector
Sector and route geometry were kept constant throughout all scenarios. With this geometry setup,
and depending on the traffic condition, the time an aircraft would stay inside the sector would on
average be around six or eight minutes, real time.
Previous research on SSD properties concluded that in simulations running four times as fast as
real time, operators would try to plan ahead the development of events approximately ten minutes
in real time [35].
With these reasons in mind, the displayed SSD was calculated considering a trajectory predic-
tion with a horizon of ten minutes (real time). This explains the rounded shape in the origin of the
VO cone in Figure 5, as the true origin of the VO angle point lies at infinite time. [44]
15
American Institute of Aeronautics and Astronautics
D. Participants and instructions
Twelve participants performed in the experiment. Four of them were professional air traffic con-
trollers, four others had recently received an extensive, multiple-day, hands-on ATC instruction
course (facilitated by the Netherlands Aerospace Center (NLR)), and the final four were aerospace
engineering graduate students from Delft University of Technology. These three populations are
from now on referred to as ATCo’s, Experts and Students, respectively.
The ATCo’s ages ranged from 35 to 61 years (µ=49.25, σ=11.96), the Experts’ ages ranged
between 27 and 47 (µ=36.75, σ=9.32), and the Students’ ages ranged between 24 and 26
(µ=25.00, σ=1.16). The limited availability of subjects, especially in the ATCO group, did not
allow us to balance-out the experiment for age.
Participants were briefed on the experiment goal and on how they should interact with the
interface. They were instructed to merge all aircraft onto the single route, without being able to
make altitude changes, trying to avoid separation violations at all costs, and have all aircraft exit
the sector at 180 kts (to better emulate reality and prevent the excessive use of maximum speed
commands). There were no minimum performance thresholds set for the participants, as this turned
out to be unnecessary in pre-experimental checks.
Participants performed two training scenarios, during which they were able to get familiarized
with the interface. The purpose was to avoid learning effects during the experiment as much as
possible. Every training scenario lasted for ten minutes. The first scenario had low-intensity traffic
and did not present the SSD so that participants could get familiarized with the simulator and focus
on learning how to control aircraft. The second scenario offered a higher level of traffic intensity,
with the SSD available, and aimed to have participants learn how to interact with the SSD and how
to interpret it.
Given that the simulation was limited and all subjects had basic to expert knowledge in air
traffic control, training time was considered sufficient. After participants had confirmed their un-
derstanding of the procedure and the use of the ISA workload rating, four full scenarios were
performed.
16
American Institute of Aeronautics and Astronautics
E. Dependent measures
After each experimental run, participants had the opportunity to provide open comments.
During the experimental runs, besides the ISA ratings of workload performed every minute,
other variables were monitored in order to get more insight in the participants’ performance and
control strategy:
1. Number of Commands. Every click in either the solution space interface, or one of the
command buttons, was counted as one command. Despite the fact that a click on the solution
space interface can have the effect of giving two commands (heading and speed changes),
we assume that one click does not represent a cognitive effort worth of two commands.
2. Aircraft Count. The median number of aircraft inside the sector (i.e., not in the entire visu-
alized airspace), measured every second (simulated time).
3. Loss of Separation Count, defined as the number of times the distance between two aircraft
was smaller than 5 NM.
4. Extra Distance Ratio: For each aircraft, the most efficient trajectory would be a straight line
from the point at which the aircraft enters the sector to HOOKS (the minimum distance). Any
other trajectory adds to this distance. This dependent measure was calculated as the ratio
between the additional flown distance and the minimum distance.
5. Smallest Aircraft Separation. The median value of the smallest separation between all flights
during their passing through the sector.
6. Sector Time. The median value of the time aircraft needed to transit the sector.
7. Handling Time. The median value of time it took the controller to issue the final merging
command to every aircraft after their entry into the sector.
All dependent measures, except for Number of Commands and Loss of Separation Count, were
analyzed with a central tendency measure to provide a single measure per scenario and participant.
Especially with this experimental setup, in which the presence of learning effects is a possibility,
the occurrence of situations in which a flight requires significantly more vectoring than normal is
17
American Institute of Aeronautics and Astronautics
likely. These situations could result in outlier values, and to rule out these effects, the median was
preferred over the mean as central tendency measure.
F. Hypotheses
The main hypothesis was that showing the SSD’s VO’s to our participants would contribute to a
reduction of their experienced workload, irrespective of traffic levels and operational expertise.
In case the VO’s were presented on the SSD, these were expected to aid participants in finding
“empty spaces” in the aircraft sequence. It would allow them to merge traffic faster and easier,
compared to the situation where the participants had to predict the future traffic flow by themselves.
Showing the VO’s was expected to result in participants to direct aircraft more towards the sector
exit (leading to a reduction in the Sector Time, Handling Time, and Extra Distance Ratio measures)
and to issue fewer clearances (reduction in the Number of Commands measure). As a consequence,
aircraft are expected to be flying closer together (reduction in the Smallest Separation measure)
when the VOs are presented.
V. Results
Three types of analyses were conducted on the data collected in the experiment. The perfor-
mance metrics analysis studied the dependent measures discussed in Subsection IV.E. This was
followed by a workload analysis based on ISA ratings and, finally, a control strategy analysis that
sought to identify the controller adaptions to the independent variables.
The population group variable showed varying levels of statistical significance in all three anal-
yses. In the performance and workload analyses, a non-parametric test (Kruskal-Wallis) was first
used to test for differences between groups. Since all scenarios showed no significant differences,
all participants were evaluated as a single population group. The control strategy analysis, on
the other side, was based on a series of multinomial logistic regressions that showed significant
differences across population groups. A more elaborate discussion is provided next.
18
American Institute of Aeronautics and Astronautics
A. Performance metrics analysis
Every population group studied in this experiment contained four participants. Being a small
sample size, a non parametric test (Kruskal-Wallis) was first used to test for differences across
population groups. For every dependent measure analyzed, results showed that, in all scenarios,
non-significant differences were present across population groups. Although differences in con-
trol strategy between the population groups were to be expected, we can conclude that the sample
size was not big enough for detecting significant differences between these groups. Therefore, the
population group variable was disregarded in the performance metrics analysis, making the popu-
lation size of the single analysis group (twelve participants) large enough for parametric statistical
methods.
The population size used in this experiment provided enough power for detecting large effect
sizes (r=0.5) of the dependent measures. To detect medium (r=0.3) or small (r=0.1) effect
sizes, a population size of 32 or 274, respectively, would have been required.a
Dependent measures that satisfied the normality and homogeneity of variance assumptions
(Number of Commands, Aircraft Count, Extra Distance Ratio, Smallest Separation, Sector Time,
and Handling Time) were studied with a parametric hypothesis test (repeated measures ANOVA).
The rest of the dependent measures (Loss of Separation Count) were studied with a non-parametric
test (Friedman’s ANOVA and pairwise comparisons with Bonferroni correction). Boxplots of all
dependent measures are shown in Figure 6.
Table 2 provides the significance levels obtained with the factorial analyses. Significant effects
of Traffic on all dependent measures are evident. Interestingly, Table 2 shows significant effects of
the SSD only on the ISA Ratings of Workload (discussed later on in Subsection V.B), the Number
of Commands, and Handling Time variables.
1. Number of Commands
There were significant main effects of Traffic and the SSD on the Number of Commands (Ncom)
that were issued. Irrespective of other predictors, the increase of traffic led to an averaged increase
aCalculated with the G*Power software, version 3.1.9.2. http://www.gpower.hhu.de/en.html.Accessed on March
1st, 2015. Inputs: α=0.05, β=0.2.
19
American Institute of Aeronautics and Astronautics
Low Tr. High Tr.
80
120
160
200
[-]
SSD Off
SSD On
a) Number of commands.
Low Tr. High Tr.
6
8
10
12
14
[-]
SSD Off
SSD On
b) Aircraft count.
Low Tr. High Tr.
0
2
4
6
8
10
[-]
SSD Off
SSD On
c) Loss of separation
count.
Low Tr. High Tr.
0
0.2
0.4
0.6
0.8
[-]
SSD Off
SSD On
d) Extra distance ratio.
Low Tr. High Tr.
6
7
8
9
[NM]
SSD Off
SSD On
e) Smallest separation.
Low Tr. High Tr.
10
15
20
25
30
35
40
[min]
SSD Off
SSD On
f) Sector time.
Low Tr. High Tr.
0
2
4
6
8
10
12
14
16
18
[min]
SSD Off
SSD On
g) Handling time.
Low Tr. High Tr.
250
500
750
1,000
1,250
Sum of Ranks
SSD Off
SSD On
h) ISA ratings of work-
load.
Figure 6. Boxplots for all dependent measures.
Table 2. Dependent measures’ ANOVA (parametric and non-parametric) results.
Metrics Factors
SSD T SSD*T
ISA Ratings of Workload *** *** -
Number of Commands *** *** -
Loss of Separation Count - ** N/A
Aircraft Count - *** -
Extra Distance Ratio - *** -
Smallest Separation - *** -
Sector Time - *** -
Handling Time * ** -
Note: SSD =Solution space diagram, T =Traffic,
-=non significant, * =p<0.05, ** =p<0.01,
*** =p<0.001.
of 50 commands (F(1,11) =87.65, p<0.001, r=0.89) and the SSD, irrespective of other
predictors, led to an averaged decrease of 17 commands (F(1,11) =23.76, p<0.001, r=0.68).
Note that both Traffic and the SSD had a large size effect (r>0.5).
An increase of Ncom at the high traffic condition would be naturally expected. The decrease of
Ncom when the SSD aid was displayed was hypothesized prior to the experiment. Several studies
indicate that Aircraft Count and Number of Commands have the highest influence on subjective
ratings of workload [2]. Therefore, a reduction in Ncomwould indicate that the SSD has a potential
20
American Institute of Aeronautics and Astronautics
to reduce workload.
2. Handling Time
There were significant main effects of Traffic and the SSD on the Handling Time variable. Ir-
respective of other predictors, the increase of traffic led to an averaged increase of 2.66 minutes
(simulated time) in handling time (F(1,11) =12.92, p<0.01, r=0.54) and the SSD, irrespective
of other predictors, led to an averaged decrease of 1.57 minutes (simulated time) in handling time
(F(1,11) =5.37, p<0.05, r=0.33).
Note that Traffic had a large size effect (r=0.54), and the SSD showed a medium size effect
(r=0.33). These results suggest that the participants were able to merge traffic onto the route
sooner with the help of the SSD. This might also explain the reduction in the number of commands
observed when the SSD was displayed.
3. Other dependent measures
All other dependent measures introduced in Subsection IV.E were not affected by the SSD sig-
nificantly, see Table 2. Only the traffic level had a significant effect on these measures, with a
higher aircraft count, longer travelled distances, more losses of separation, longer sector times,
and reduced separation between aircraft for the high traffic level, as it would be expected.
The SSD was expected to help participants to “find holes” for merging traffic in the aircraft
trains more easily (resulting in aircraft flying closer together), and to aid in directing the traffic
more towards the sector exit. It was therefore hypothesized that the SSD would induce a reduction
in the Smallest Separation, Sector Time, and Extra Distance Ratio dependent measures. Results
showed, however, no significant influences of the SSD on these metrics.
B. Workload analysis
With 20 ratings per simulated scenario, 12 participants and 4 scenarios per participant, 960 sub-
jective ratings of workload were measured during the entire experiment. These subjective ratings
were all measured in an ordinal scale, i.e., a ranking in terms of degree that has no established
numerical difference between rankings. For data analysis, an ordinal scale requires a permissible
21
American Institute of Aeronautics and Astronautics
transformation that preserves its ordinality [52], and one of them is a rank-based transformation.
For this reason, all 80 subjective ISA ratings provided by every participant (4 scenarios, each hav-
ing 20 ratings) were ranked, and then the sum of ranks for every scenario was calculated, summed
across participants.
Due to the small population size in every participating group, a non parametric test (Kruskal-
Wallis) was first used to test for differences across population groups. Since all scenarios showed
no significant differences between groups, all participants were considered to belong to a single
group.
After testing for the relevant parametric assumptions (normality and homogeneity of variance),
a repeated measures ANOVA was performed on this transformed metric. Figure 6h shows a boxplot
for this dependent measure.
Results showed significantly higher ISA ratings with an increase of traffic (F(1,11) =118.83,
p<0.001, r=0.92), and significantly lower ratings when the SSD interface was available
(F(1,11) =5.02, p=0.047, r=0.31). Subsection V.A showed similar results for the Num-
ber of Commands metric. Note that effect of size shows that traffic had a large effect on the total
variance, while the SSD aid had a medium effect.
C. Strategy analysis
In subsection V.A we showed that Traffic and SSD had a significant effect on the number of com-
mands. This section analyzes the type of commands issued to obtain more insight into the strategy
the participants adopted.
Participants had three options for executing their task: clicking on the SSD interface, directing
traffic to a specific waypoint, or issuing a route intercept command. It was expected that partici-
pants would only make use of the SSD interface for vectoring aircraft, but their behavior showed
differently.
Every command the participant issued was classified as either being a “vectoring”, “speed
adjustment”, or “merging” command. Merging commands were identified as the last Direct-to
or Intercept Route command issued to every aircraft. Vectoring commands were identified as all
Direct-to commands issued before the final merge command, and any click on the SSD that would
22
American Institute of Aeronautics and Astronautics
produce a change in heading. Finally, any click on the SSD that would produce only a change in
speed but not in heading was identified as a speed adjustment command.
Within the merging commands, the participant had to choose between either a Direct-to or
an Intercept Route command. Further, the choice of a Direct-to command needed to specify one
of the three available waypoints. This setup, in which participants must make choices within a
finite number of possibilities, may be analyzed with a classification method such as the logistic
regression [53, pp. 264–265]. Since more than two possible discrete outcomes were available
to participants, the multinomial logistic regression was the method of choice for studying their
strategy adaptations.
1. Vector, speed adjustment, and merge commands
The coefficients of the multinomial logistic regression model are shown in Table 3. As a goodness
of fit indicator, the deviance statistic was calculated as χ2(14) =14.621, p=0.405. Hence, the
null hypothesis of the goodness of fit test (the data follow the specified distribution) can not be
rejected.
Table 3. Multinomial logistic regression coefficients for model of issued commands.
95% CI for Odds Ratio
Variable B(SE) Lower Odds Ratio Upper
Vectoring vs. Merging
Intercept 0.11 (0.08) 0.96 1.12 1.31
SSD -0.21 (0.07)** 0.71 0.81 0.93
Traffic 0.34 (0.07)*** 1.23 1.41 1.62
Group =Experts 0.21 (0.09)* 1.04 1.23 1.46
Group =ATCo’s 0.35 (0.08)*** 1.20 1.42 1.67
Speed adjustments vs. Merging
Intercept 0.68 (0.07)*** 1.71 1.97 2.28
SSD -0.19 (0.07)** 0.73 0.83 0.94
Traffic 0.14 (0.07)* 1.01 1.15 1.31
Group =Experts -0.07 (0.08) 0.80 0.93 1.09
Group =ATCo’s -0.09 (0.08) 0.78 0.92 1.08
Note: * =p<0.05, ** =p<0.01, *** =p<0.001.
Note that the analysis performs a series of comparisons between two categories. Having three
outcome categories (i.e., “Vectoring”, “Merging”, and “Speed adjustment”), the analysis consists
of two comparisons [53, p. 300], in which “Merging” was selected as the baseline category.
The increase of traffic had a significant influence on whether a vector (p<0.001) or a speed
adjustment (p<0.05) command was preferred over a merge command. The odds ratio shows that
23
American Institute of Aeronautics and Astronautics
the change in odds of vectoring or adjusting speed rather than merging traffic was 1.41 and 1.15,
respectively. Participants were therefore more likely to vector or adjust speed than merge flights
when confronted with the high traffic level.
Displaying the SSD had a significant influence (p<0.01) on the type of command issued as
well. The odds ratio shows that the change in odds of vectoring or adjusting speed rather than
merging traffic was 0.81 and 0.83, respectively. Participants were therefore less likely to vector or
adjust speed than merge flights when the SSD aid was displayed.
Based on the model, the probabilities of issuing all different types of commands are shown
in Figure 7. Note that, in this figure, the dashed and full lines represent the low and high traffic
conditions, respectively.
SSD OffSSD On
0.2
0.3
0.4
0.5
0.6
Vectoring
Speed adjustment
Merging
a) Student group.
SSD OffSSD On
0.2
0.3
0.4
0.5
0.6
Vectoring
Speed adjustment
Merging
b) Expert group.
SSD OffSSD On
0.2
0.3
0.4
0.5
0.6
Vectoring
Speed adjustment
Merging
c) ATCo group.
Figure 7. Modeled probabilities of either vectoring, adjusting speed, or merging traffic.
2. Merge commands
Traffic was to be merged with either a Direct-to or Intercept Route command. The coefficients of
the multinomial logistic regression model are shown in Table 4. The regression model showed to
have an acceptable goodness of fit indicator (deviance): χ2(5) =6.087, p=0.298.
The increase of traffic had a significant influence on the issued merge command (p<0.001).
The odds ratio show that the change in odds of having an aircraft intercept the route rather than
directing it to a specific waypoint was 1.63. Participants were therefore more likely to issue a
Route Intercept command when confronted with high traffic.
Displaying the SSD had a significant influence (p<0.05) as well. The odds ratio show that
the change in odds of having an aircraft intercept the route rather than directing it to a specific
24
American Institute of Aeronautics and Astronautics
Table 4. Multinomial logistic regression coefficients for the model of issued merge commands.
95% CI for Odds Ratio
Variable B(SE) Lower Odds Ratio Upper
Intercept Route vs. Direct-to
Intercept 0.01 (0.15) 0.76 1.01 1.35
SSD -0.45 (0.19)* 0.44 0.64 0.92
Traffic 0.49 (0.11)*** 1.30 1.63 2.03
Group =Experts -0.55 (0.19)** 0.40 0.57 0.83
Group =ATCo’s -1.58 (0.21)*** 0.14 0.21 0.31
Group =Experts * SSD -0.13 (0.27) 0.52 0.88 1.49
Group =ATCo’s * SSD 1.18 (0.28)*** 1.87 3.24 5.61
Note: * =p<0.05, ** =p<0.01, *** =p<0.001.
waypoint was 0.64. Participants were therefore less likely to issue a Route Intercept command
when the SSD aid was displayed. This effect, though, was surpassed in size by the interaction with
the ATCo group ( p<0.001). For the ATCo’s, the change in odds of having an aircraft intercept
the route was 3.24. They were therefore much more likely to intercept traffic with the SSD aid.
Based on the model, the probabilities of issuing an Intercept Route rather than a Direct-to
command are shown in Figure 8.
SSD OffSSD On
0.2
0.3
0.4
0.5
0.6
0.7High traffic
Low traffic
a) Students group.
SSD OffSSD On
0.2
0.3
0.4
0.5
0.6
0.7High traffic
Low traffic
b) Experts group.
SSD OffSSD On
0.2
0.3
0.4
0.5
0.6
0.7High traffic
Low traffic
c) ATCo’s group.
Figure 8. Modeled probabilities of merging traffic with an Intercept Route rather than a Direct-to command.
SSD OffSSD On
0.2
0.3
0.4
0.5
0.6
0.7High traffic
Low traffic
a) Students group.
SSD OffSSD On
0.2
0.3
0.4
0.5
0.6
0.7High traffic
Low traffic
b) Experts group.
SSD OffSSD On
0.2
0.3
0.4
0.5
0.6
0.7
High traffic
Low traffic
c) ATCo’s group.
Figure 9. Modeled probabilities of directing traffic to the sector’s exit waypoint.
25
American Institute of Aeronautics and Astronautics
3. Directing traffic to the sector’s exit
Participants had three waypoints available for directing traffic. Choosing the third waypoint was
equivalent to sending aircraft straight to the sector’s exit point, HOOKS, see Figure 5. The regres-
sion model was built to compare the possibilities of directing traffic to the first or second waypoints
vs. directing traffic to the sector’s exit point; regression coefficients are shown in Table 5. The
model showed to have an acceptable goodness of fit (deviance): χ2(4) =1.999, p=0.736.
Table 5. Multinomial logistic regression coefficients for the directing strategy model.
95% CI for Odds Ratio
Variable B(SE) Lower Odds Ratio Upper
Directing traffic to 1st or 2nd waypoints vs Directing traffic to the exit
Intercept -0.90 (0.18)*** 0.29 0.41 0.57
Traffic 1.34 (0.23)*** 2.42 3.82 6.04
Group =Students 1.63 (0.31)*** 2.75 5.09 9.44
Group =Experts 0.89 (0.27)** 1.42 2.43 4.15
Group =Students * Traffic -0.85 (0.38)* 0.20 0.43 0.90
Group =Experts * Traffic -1.08 (0.33)** 0.18 0.34 0.65
Group =Students * SSD -0.03 (0.30) 0.54 0.97 1.73
Group =Experts * SSD 0.53 (0.24)* 1.06 1.70 2.72
Note: * =p<0.05, ** =p<0.01, *** =p<0.001.
The increase of traffic had a significant influence on selection of waypoint for directing traffic
(p<0.001). With such increase, all population groups were less likely to direct traffic to the
sector’s exit. The SSD aid showed to have a significant effect only in the Experts group (p<0.05).
This group was less likely to direct traffic to the sector’s exit when the aid was displayed. Based
on the model, the probabilities of directing traffic to the sector’s exit are shown in Figure 9.
VI. Discussion
This study intends to shed light on the effects the solution space diagram (SSD) has on work-
load, performance, and control strategy when performing tactical merging and separation tasks. Due
to two major caveats, its results should be interpreted and used with caution.
The medium-fidelity simulation studied in this paper attempted to emulate a limited subset of
common ATC tasks. Participants were not required to keep track of altitude changes because the
goal was to evaluate the potential of the SSD as a tactical CD&R and merging tool in the horizontal
plane. A tool that supports controllers in the vertical plane by showing no-go zones (a design phi-
losophy similar to the SSD) already exists [25] and is being used in operational environments. As
26
American Institute of Aeronautics and Astronautics
additional limitations, participants did not have to maintain flight strips, deal with aircraft inad-
vertently deviating from their prescribed trajectories, or having to make verbal communicates. All
these additional tasks would increase the perceived workload. Their exclusion had the purpose of
simplifying the experimental setup and reducing training time and learning effects. Despite this last
consideration, some of the experienced controllers that participated in the experiment mentioned
that they felt they still had not passed over the full learning curve when it came to making use of
the SSD. Future experiments should consider this.
Another point of consideration is the statistical power that was available for the experiment. A
sample size of 12 participants was good enough for detecting large effects of the independent
variables. To detect medium size effects, the current experiment would have needed a total of 32
participants, ideally all professional air traffic controllers. This number of highly-skilled profes-
sionals represents a large-scale endeavor, which was not feasible in the current study. Furthermore,
participants represented different population groups. The population group variable had no sig-
nificant effect on measures of performance or workload, but did significantly influence control
strategy. With a larger population sample, the impact that the control strategy has on performance
and workload should have noticeable effects.
A. Effects on workload
Several studies state that aircraft count and number of commands correlate with the most important
metrics for workload [2]. It was hypothesized that the SSD would induce the participants to send
aircraft more often towards the sector exit, which in turn would have reduced the average number
of aircraft inside the sector. This study showed that the SSD significantly reduced the number of
commands (large size effect) issued by the participants and the handling time per aircraft (medium
size effect). The perceived workload levels were also influenced by the SSD, showing a medium
sized reduction effect.
After the experimental trials, however, some participants reported that the aid had detrimen-
tal effects on their workload (especially during high traffic conditions) because of the momentary
attention diversion required for accessing the information in the diagram (refer to Figure 5). This
must have influenced the Instantaneous Self-Assessments of workload performed during the sim-
27
American Institute of Aeronautics and Astronautics
ulation runs, and that would explain the medium size effect on the reduction of workload when
a large size effect on the reduction of the number of commands was observed. Possibly, some
changes in the interface might solve this issue, like overlaying the SSD on the aircraft icons of the
plan view display.
Our results show the potential of the SSD as an information analysis aid.bClamann et al. [55]
and Kaber et al. [49] have shown, through empirical studies, that when automation is applied
to support cognitive functions such as information analysis or decision-making, higher workload
levels are experienced. In contrast, automated support for lower-level sensory and psychomo-
tor functions, such as information acquisition and action implementation, results in better overall
human-machine system performance. Whether these findings hold on a general level, however,
is debated in the literature (see e.g., [56, 57]). Design rules for what levels of automation, in all
its possible dimensions, from information-gathering to decision-making, seem impossible, as all
depends on context, dynamics, operator strategies and tasks. Nevertheless, in our study the SSD
provided a low level of automation in decision-making, and a high level of automation in informa-
tion analysis, resulting in an overall reduction in the workload experienced by the participants.
In this study, participants were responsible for the acquisition of information; they had to mon-
itor all aircraft actively, and whenever they needed additional information for a specific aircraft,
by clicking on it they could have access to the SSD. They were also responsible for the imple-
mentation of actions; once they had decided on a plan of action, they had to issue the instructions
themselves. The SSD implementation seems to have induced an increase of performance (reduction
in the number of issued commands and the time needed for handling every aircraft) even though
information acquisition and action implementation were not automated.
The authors would further argue that the “what-if” analysis which the SSD supports resulted
in the active involvement of the user, and that the way in which the SSD presents information not
only allowed for a reduction in workload, but also helped participants to maintain their situation-
awareness by supporting the comprehension and projection of system status. Further studies that
evaluate the SSD’s influence on situation awareness are needed, however, to investigate this claim.
bAccording to Parasuraman et al. [54], a tool that projects aircraft positions to evaluate for conflicts is an informa-
tion analysis tool.
28
American Institute of Aeronautics and Astronautics
B. Effects on control strategy
The SSD had a significant influence on the strategy adopted by the controller. Irrespective of the
traffic level, the reduction in the number of issued commands came together with a reduction in
the amount of vectoring and speed adjustments. This gave room to an increase in the number of
merging commands. Such a trend is known to have a positive effect on controller workload [58].
Moreover, merging was performed in different ways for the different population groups. While
the Students and Experts groups used the aid to reduce (by about 10%) the number of Intercept
Route commands and increase (by the same amount) the Direct-to clearances, the ATCo’s group
did the exact opposite (in the same proportion). However, the ATCo’s group always maintained
their overall preference for issuing Direct-to clearances (always higher than 50%) over Intercept
Route commands. The slight change in preference could be attributed to a trust and/or safety is-
sue. The diagram allowed ATCo’s to provide a merge clearance at an earlier stage, but they might
have been somewhat conservative when issuing a Direct-to clearance due to concerns about the
diagram’s accuracy. Furthermore, with the new information provided by the diagram, they might
have seen waypoints as possible bottlenecks and tried to avoid an excessive use of Direct-to clear-
ances.
In terms of a preference for issuing a Direct-to clearance to the exit waypoint over the other
sector waypoints, no clear trend was observed.
As it would be expected, traffic had the effect of changing the controller’s strategy to perform
more vectoring and to issue route interceptions in a later stage in order to ensure separation. The
SSD, however, showed to have induced an opposite effect (less vectoring and earlier route inter-
cepts) irrespective of the traffic level, and without significantly influencing separation. Since the
aiding tool did not suggest any conflict resolution commands, we conclude that the change in strat-
egy is most likely because the interface allowed the controller to acquire conflict information that
was otherwise not available.
The SSD had a significant influence in the strategy adopted by the controller. Even though
some participants indicated that it was slightly distracting and that they felt they were still in their
learning curve, important beneficial effects were observed, like less vectoring, earlier route inter-
cept, a reduction in workload and in the number of issued commands. Therefore, further research
29
American Institute of Aeronautics and Astronautics
seems justified of the effects the SSD has on controllers once they are more familiar with it.
C. Future research
This paper investigated whether the workload experienced by an ATCo performing a two-dimen-
sional aircraft merging task could be reduced by presenting the SSD. The results observed justify
experimentation with a larger group of experienced controllers in a higher fidelity simulation setup.
Introducing a new tool in Air Traffic Control is a tedious effort, however. Case studies of the adop-
tion and adaptation of the User Request Evaluation Tool (URET), for instance, showed differences
in adopting the tool from center to center, from controller teams to teams, and from controller to
controller. [59]. Other likely directions of future work are as follows.
It was observed during the experimental trials that there might be a traffic threshold level above
which the diagram is effective in reducing workload. The existence of a threshold up to which
the diagram remains effective is also a possibility. This indicates that the SSD might show to be
a valuable tool in the area of adaptive automation, in which the allocation of aiding tools, based
on states of the collective human-machine system, has the purpose of reducing the complexity of
the control problem at hand [60]. Preliminary tests in using the SSD as a trigger mechanism [61]
or as a way to predict individual controller actions [62] show promising results, but more work is
needed to substantiate the use of an SSD-based automation trigger mechanism.
Some participants reported that the aiding diagram had a distracting effect in some high traf-
fic situations. Further reducing the effort required for accessing the diagram by, e.g., displaying
the diagram around the aircraft icon on the plan-view display whenever the controller selects the
aircraft, might further enhance the controller’s performance and reduce subjective workload.
The fact that the diagram affects the adopted control strategy is apparent. Evidence suggests
that the learning curve will require a significant degree of controller experience with the SSD
to predict its steady-state impact. Thus, another research direction is possibly the analysis of the
strategies undertaken by participants with more experience using the diagram. Training experi-
enced controllers in the use of the diagram may not be a feasible option, as the mental strategies
they have developed over the years may dominate whatever new strategy that the SSD may make
possible. Training other population groups, or even educating student air traffic controllers, and
30
American Institute of Aeronautics and Astronautics
analyzing their strategies and performance improvements, is an avenue of further research [43].
Finally, the inclusion of the third dimension in the diagram is an important extension. Since
aircraft separation can also be achieved by changing altitude levels, the reduction of the cognitive
load for two-dimensional conflict resolution may lead to an overall reduction in the level of per-
ceived workload. Yet, it could also be the case that the interpretation of a three-dimensional SSD
demands additional cognitive resources from the controller, resulting in higher workload levels. In
this respect, two design concepts of a two-dimensional diagram that allow discrete altitude (level)
changes have been created [37, 38], but not yet extensively evaluated. Thus, it remains to be seen
whether the additional complexity resulting from including the altitude dimension renders the SSD
useless, or not.
VII. Conclusions
The solution space diagram (SSD) visualizes the functional constraints of an aircraft caused by
the trajectories of other aircraft nearby. A pilot study is presented in which novice and experienced
air traffic controllers performed a two-dimensional aircraft merging task, at low and high traffic
conditions, with and without the SSD. We investigated the diagram’s potential to reduce controller
workload and its effects on the adopted control strategy.
Results showed that, at both traffic levels and irrespective of expertise group, the diagram
effectively reduced the number of commands the participants issued to fulfill the ATC task. Most
likely this reduction was caused by decrements in the time needed for handling every aircraft. The
reduction in the number of commands also reflected a significant trend in the reduction of the
workload levels perceived by the participants.
Observations suggest that participants were able to find a merging solution faster with than
without the SSD. With its use, participants changed their strategy to perform less vectoring and
to issue route intercepts at an earlier stage, without significantly influencing aircraft separation.
These changes were also observed with the professional controllers, even though they showed to
have been more conservative in using the diagram. The results justify experimentation with a larger
group of (experienced) controllers in a higher-fidelity simulation environment.
31
American Institute of Aeronautics and Astronautics
Acknowledgements
The authors would like to thank the twelve participants of the experiment, and also the review-
ers for their useful feedback on the two drafts of this manuscript.
References
[1] Majumdar, A. and Polak, J., “Estimating capacity of Europe’s airspace using a simulation model of air
traffic controller workload,” Transportation Research Record, Vol. 1744, 2001, pp. 30–43,
doi:10.3141/1744-05.
[2] Hilburn, B. G., “Cognitive Complexity in Air Traffic Control a Literature Review,” Tech. Rep. EEC
Note No. 04/04, EUROCONTROL, 2004.
[3] Erzberger, H., Lauderdale, T. A., and Chu, Y. C., “Automated conflict resolution, arrival management,
and weather avoidance for air traffic management,” in “Proceedings of the Institution of Mechanical
Engineers, Part G: Journal of Aerospace Engineering,” Inst. of Mechanical Engineers, 2012, pp. 930–
949,
doi:10.1177/0954410011417347.
[4] “SESAR Concept of Operations,” Tech. rep., Brussels, Belgium, 2007,
doi:DLM-0612-001-02-00.
[5] “Concept of Operations for the Next Generation Air Transportation System, Version 3.2,” Report, Joint
Planning and Development Office, 2011.
[6] “Air Traffic Control Decision Support Tool Design and Implementation Handbook,” Report
DOT/FAA/TC-19/37, US Department of Transportation, Federal Aviation Administration, 2019.
[7] Hilburn, B. G. and Jorna, P. G. A. M., “Workload and Air Traffic Control,” in Hancock, P. A. and
Desmond, P. A., eds., “Stress, Workload and Fatigue: Theory, Research and Practice,” Lawrence
Erlbaum Associates, pp. 384–394, 2001,
doi:10.1002/hfm.1009.
32
American Institute of Aeronautics and Astronautics
[8] Stassen, H. G., Johannsen, G., and Moray, N., “Internal Representation, Internal Model, Human Per-
formance and Mental Workload,” Automatica, Vol. 26, No. 4, 1990, pp. 811–820,
doi:10.1016/S1474-6670(17)53877-X.
[9] Gra˜
na, M., “Dynamic Airspace Configuration: A Short Review of Computational Approaches,” in
Nguyen, N. T., Chbeir, R., Exposito, E., Aniort´
e, P., and Trawi´
nski, B., eds., “Computational Collective
Intelligence,” Springer International Publishing, 2019, pp. 486–497,
doi:10.1007/978-3-030-28377-3 40.
[10] “A Proposal for the Future Architecture of the European Airspace,” Report, SESAR Joint Undertaking,
2019,
doi:10.2829/309090.
[11] Prevot, T., Lee, P. U., Smith, N., and Palmer, E. A., “ATC Technologies for Controller-Managed and
Autonomous Flight Operations,” AIAA Guidance, Navigation and Control Conference, 15-18 August,
San Francisco, California, USA,
doi:10.2514/6.2005-6043.
[12] Prevot, T., “NextGen technologies for Mid-Term and Far-Term Air Traffic Control Operations,” IEEE-
AIAA Digital Avionics Systems Conference,
doi:10.1109/DASC.2009.5347556.
[13] Wong, B. L., Rozzi, S., Boccalatte, A., Gaukrodger, S., Amaldi, P., Fields, B., Loomes, M. J., and Mar-
tin, P., “3D-in-2D Displays for ATC,” Sixth Eurocontrol innovative research workshop &exhibition,
Bretigny, France, pp. 54–69.
[14] Wong, B. L., Rozzi, S., Gaukrodger, S., Boccalatte, A., Amaldi, P., Fields, B., Loomes, M. J., and
Martin, P., “Human-Centred Innovation: Developing 3D-in-2D Displays for ATC,” Third International
conference on Research in Air Transportation, Fairfax (VA), USA, pp. 295–302.
[15] Prevot, T., Lee, P. U., Callantine, T., Mercer, J., Homola, J., Smith, N., and Palmer, E. A., “Human-In-
the-Loop Evaluation of NextGen Concepts in the Airspace Operations Laboratory,” AIAA Modelling
and Simulation Technologies Conference, 2-5 August, Toronto, Canada,
doi:10.2514/6.2010-7609.
33
American Institute of Aeronautics and Astronautics
[16] Prevot, T., Homola, and Mercer, J., “An Integrated Tool Suite for En Route Radar Controllers in
NextGen,” 27th International Congress of the Aeronautical Sciences (ICAS).
[17] Prevot, T., Homola, J. R., Martin, L. H., Mercer, J. S., and Cabrall, C. D., “Toward Automated Air
Traffic Control—Investigating a Fundamental Paradigm Shift in Human/Systems Interaction,” Inter-
national Journal of Human-Computer Interaction, Vol. 2, No. 28, 2012, pp. 77–98,
doi:10.1080/10447318.2012.634756.
[18] Lundberg, J., Johansson, J., Forsell, C., and Josefsson, B., “The Use of Conflict Detection Tools in Air
Traffic Management – an Unobtrusive Eye Tracking Field Experiment During Controller Competence
Assurance,” in “Proceedings of the HCI-Aero Conference, July 30-August 1, Santa Clara (CA), USA,”
, 2014,
doi:10.1145/2669592.2669655.
[19] Bekier, M., Molesworth, B. R., and Williamson, A., “Tipping point: The narrow path between au-
tomation acceptance and rejection in air traffic management,” Safety Science, Vol. 50, No. 2, 2012, pp.
259–265,
doi:10.1016/j.ssci.2011.08.059.
[20] Westin, C., Borst, C., and Hilburn, B. G., “Strategic Conformance: Overcoming Acceptance Issues of
Decision Aiding Automation?” IEEE Transactions on Human-Machine Systems, Vol. 46, No. 1, 2016,
pp. 41–52,
doi:10.1109/THMS.2015.2482480.
[21] Sarter, N. B. and Woods, D. D., “Cognitive Engineering in Aerospace Application: Pilot Interaction
with Cockpit Automation,” Tech. Rep. CR-177617, 1993.
[22] Hilburn, B. G., “Evaluating Human Interaction with Advanced Air Traffic Management Automation,”
in “RTO Meeting Proceedings,” Research and Technology Organisation, 2002, pp. 1–12.
[23] Brooker, P., “Air Traffic Control automation: for humans or people?” Human Factors and Aerospace
Safety, Vol. 5, No. 1, 2005, pp. 23–42.
[24] Kirwan, B., Scaife, R., and Kennedy, R. L., “Investigating complexity factors in UK Air Traffic Man-
agement,” Human factors and aerospace safety, Vol. 1, No. 2, 2001, pp. 125–144.
34
American Institute of Aeronautics and Astronautics
[25] Irfan, M., Bull, M. J., Clinch, A. T., and Pember, S. J., “Air Traffic Control,” U.S. Patent Application
Publication US 2012/0303253 A1, 2012.
[26] Endsley, M. R., “From Here to Autonomy: Lessons Learned from Human-Automation Research,”
Human Factors, Vol. 59, No. 1, 2017, pp. 5–27,
doi:10.1177/0018720816681350.
[27] Jorna, P. G. A. M., Pavet, D., van Blanken, M., and Pichancourt, I., “PHARE Ground Human Machine
Interface (GHMI) project: Summary Report,” Tech. Rep. DOC 99-70-02, EUROCONTROL, 1999.
[28] Whiteley, M. J. and Wilson, I., “PHARE Advanced Tools Problem Solver (Final Report),” Tech. Rep.
DOC 98-70-18, EUROCONTROL, 1999.
[29] NATS, “Validation Report (VALR),” Tech. Rep. Project ID 04.07.08 (D10: Validation Report (VALR)),
SESAR, 2011.
[30] Cort´
es-Obrero, G., “Work as done by Controllers: A Practical Approach in the OPS Room,” Hindsight,
Vol. 25, 2017, pp. 26–29.
[31] Van Gool, M. and Schroter, H., “PHARE final report,” Tech. Rep. DOC 99-70-09, EUROCONTROL,
1999.
[32] Van Dam, S. B. J., Mulder, M., and Van Paassen, M. M., “Ecological Interface Design of a Tactical
Airborne Separation Assistance Tool,” IEEE Trans. on Systems, Man &Cybernetics, Part A, Vol. 38,
No. 6, 2008, pp. 1221–1233,
doi:10.1109/TSMCA.2008.2001069.
[33] Borst, C., Bijsterbosch, V. A., van Paassen, M. M., and Mulder, M., “Ecological interface design:
supporting fault diagnosis of automated advice in a supervisory air traffic control task,” Cognition,
Technology &Work, Vol. 19, No. 4, 2017, pp. 545–560,
doi:10.1007/s10111-017-0438-y.
[34] Hermes, P., Mulder, M., van Paassen, M. M., and Boering, J. H. L., “Solution-Space-Based Analysis
of the Difficulty of Aircraft Merging Tasks,” Journal of Aircraft, Vol. 46, No. 6, 2009, pp. 1995–2015,
doi:10.2514/1.42886.
35
American Institute of Aeronautics and Astronautics
[35] D’Engelbronner, J. G., Borst, C., Ellerbroek, J., van Paassen, M. M., and Mulder, M., “Solution Space-
Based Analysis of Dynamic Air Traffic Controller Workload,” Journal of Aircraft, Vol. 52, No. 4, 2015,
pp. 1146–1160,
doi:10.2514/1.C032847.
[36] Abdul Rahman, S. M., Borst, C., Mulder, M., and Van Paassen, M. M., “Sector Complexity Measures:
A Comparison,” Jurnal Teknologi, Vol. 76, No. 11, 2015, pp. 131–139,
doi:10.11113/jt.v76.5923.
[37] Lodder, J., Comans, J., Van Paassen, M. M., and Mulder, M., “Altitude-Extended Solution Space
Diagram for Air Traffic Controllers,” in “Proc. of the 16th International Symposium on Aviation Psy-
chology (ISAP), Dayton (OH), May 2-5,” Wright State University, 2011, pp. 345–350.
[38] Beernink, B., Borst, C., Ellerbroek, J., Van Paassen, M. M., and Mulder, M., “Toward an Integrated
Ecological Plan View Display for Air Traffic Controllers,” in “Proc. of the 18th International Sym-
posium on Aviation Psychology (ISAP), Dayton (OH), May 4-7,” Wright State University, 2015, pp.
55–60.
[39] Ellerbroek, J., Visser, S. B. J., M. van Dam, Mulder, M., and Van Paassen, M. M., “Design of an
Airborne Three-Dimensional Separation Assistance Display,” IEEE Transactions on Systems, Man &
Cybernetics, Part A, Vol. 41, No. 5, 2011, pp. 863–875,
doi:10.1109/TSMCA.2010.2093890.
[40] Ellerbroek, J., Brantegem, K. C. R., Van Paassen, M. M., and Mulder, M., “Design of a Coplanar
Airborne Separation Display,” IEEE Transactions on Human-Machine Systems, Vol. 43, No. 3, 2013,
pp. 277–289,
doi:10.1109/TSMC.2013.2242888.
[41] Ellerbroek, J., Brantegem, K. C. R., Van Paassen, M. M., de Gelder, N., and Mulder, M., “Experimen-
tal Evaluation of a Coplanar Airborne Separation Display,” IEEE Transactions on Human-Machine
Systems, Vol. 43, No. 3, 2013, pp. 290–301,
doi:10.1109/TSMC.2013.2238925.
[42] Borst, C., Bijsterbosch, V. A., Van Paassen, M. M., and Mulder, M., “Ecological Interface Design:
Supporting Fault Diagnosis of Automated Advice in a Supervisory Air Traffic Control Task,” Cogni-
36
American Institute of Aeronautics and Astronautics
tion, Technology, and Work, Vol. 19, 2017, pp. 545–560,
doi:10.1007/s10111-017-0438-y.
[43] Borst, C., Visser, R. M., Van Paassen, M. M., and Mulder, M., “Exploring Short-Term Training Effects
of Ecological Interfaces: A Case Study in Air Traffic Control,” IEEE Transactions on Human-Machine
Systems, Vol. 49, No. 6, 2019, pp. 623–632,
doi:10.1109/THMS.2019.2919742.
[44] Mercado-Velasco, G. A., Borst, C., Ellerbroek, J., Mulder, M., and van Paassen, M. M., “The Use of
Intent Information in Conflict Detection and Resolution Models Based on Dynamic Velocity Obsta-
cles,” IEEE Transactions on Intelligent Transportation Systems, Vol. 16, No. 4, 2015, pp. 2297 – 2302,
doi:10.1109/TITS.2014.2376031.
[45] Fiorini, P. and Shiller, Z., “Motion Planning in Dynamic Environments Using Velocity Obstacles,” The
International Journal of Robotics Research, Vol. 17, No. 7, 1998, pp. 760–772,
doi:10.1177/027836499801700706.
[46] Snape, J., Berg, J. v. d., Guy, S. J., and Manocha, D., “The Hybrid Reciprocal Velocity Obstacle,”
IEEE Transactions on Robotics, Vol. 27, No. 4, 2011, pp. 696–706,
doi:10.1109/TRO.2011.2120810.
[47] Mulder, M., Borst, C., and Van Paassen, M. M., “Improving Operator Situation Awareness through
Ecological Interfaces: Lessons from Aviation,” in Holzinger, A., Pl´
acido da Silva, H., and Helfert,
M., eds., “Computer-Human Interaction Research and Applications, CHIRA 2017, Revised Selected
Papers,” Springer International Publishing, pp. 20–44, 2019,
doi:10.1007/978-3-030-32965-5 2. Series: Communications in Computer and Information Science,
volume 654.
[48] van Paassen, M. M., Borst, C., Ellerbroek, J., Mulder, M., and Flach, J. M., “Ecological Interface
Design for Vehicle Locomotion Control,” IEEE Trans. on Human-Machine Systems, Vol. 48, No. 5,
2018, pp. 541–555.
[49] Kaber, D. B., Perry, C. M., Segall, N., McClernon, C. K., and Prinzel, L. J., “Situation awareness
implications of adaptive automation for information processing in an air traffic control-related task,”
37
American Institute of Aeronautics and Astronautics
International Journal of Industrial Ergonomics, Vol. 36, No. 5, 2006, pp. 447–462,
doi:10.1016/j.ergon.2006.01.008.
[50] Tattersall, A. and Foord, P., “An experimental evaluation of instantaneous self-assessment as a measure
of workload,” Ergonomics, Vol. 39, No. 5, 1996, pp. 740–748,
doi:10.1080/00140139608964495.
[51] Farmer, E. and Brownson, A., “Review of Workload Measurement, Analysis and Interpretation Meth-
ods,” Tech. Rep. CARE-Integra-TRS-130-02-WP2, Eurocontrol, 2003.
[52] Stevens, S., “On the theory of scales of measurement,” Science, Vol. 103, No. 2684, 1946, pp. 677–680.
[53] Field, A., Discovering Statistics Using SPSS, SAGE Publications, Thousand Oaks, CA, 3rd ed., 2009.
[54] Parasuraman, R., Sheridan, T. B., and Wickens, C. D., “A model for types and levels of human inter-
action with automation.” IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and
Humans, Vol. 30, No. 3, 2000, pp. 286–297,
doi:10.1109/3468.844354.
[55] Clamann, M. P., Wright, M., and Kaber, D. B., “Comparison of Performance Effects of Adaptive Au-
tomation Applied to Various Stages of Human-Machine System Information Processing,” Proceedings
of the Human Factors and Ergonomics Society Annual Meeting, Vol. 46, No. 3, 2002, pp. 342–346,
doi:10.1177/154193120204600327.
[56] Roth, E. M. and Pritchett, A. R., “Preface to the Special Issue on Advancing Models of Hu-
man–Automation Interaction,” Journal of Cognitive Engineering and Decision Making, Vol. 12, No. 1,
2018, pp. 3–6,
doi:10.1177/1555343417749192.
[57] Sheridan, T. B., “Comments on “Issues in Human–Automation Interaction Modeling: Presumptive
Aspects of Frameworks of Types and Levels of Automation” by David B. Kaber,” Journal of Cognitive
Engineering and Decision Making, Vol. 12, No. 1, 2018, pp. 25–28,
doi:10.1177/1555343417724964.
38
American Institute of Aeronautics and Astronautics
[58] Boursier, L., Favennec, B., Hoffman, E., Trzmiel, A., Vergne, F., and Zeghal, K., “Merging Arrival
Flows Without Heading Instructions,” in “USA/Europe Air Traffic Management Research and Devel-
opment Seminar,” , 2007, pp. 1–8.
[59] Bolic, T. and Hansen, M., “User Request Evaluation Tool (URET) Adoption and Adaptation, Three
Center Case Study,” in “Proc. of the 6th USA/Europe ATM Seminar, Baltimore (MD), June 27-30,” ,
2005, pp. 1–7.
[60] Kaber, D. B., Riley, J. M., Tan, K.-W., and Endsley, M. R., “On the Design of Adaptive Automation for
Complex Systems,” International Journal of Cognitive Ergonomics, Vol. 5, No. 1, 2001, pp. 37–57,
doi:10.1207/S15327566IJCE0501 3.
[61] IJtsma, M., Borst, C., Mercado-Velasco, G. A., Mulder, M., and van Paassen, M. M., “Adaptive Au-
tomation Based on Air Traffic Controller Decision-Making,” in “Proc. of the 19th International Sym-
posium on Aviation Psychology (ISAP), Dayton (OH), May 8-11,” Wright State University, 2017.
[62] van Rooijen, S. J., Ellerbroek, J., Borst, C., and van Kampen, E., “Toward Individual-Sensitive Au-
tomation for Air Traffic Control Using Convolutional Neural Networks,” Journal of Air Transportation,
Vol. 0, No. 0, 2020, pp. 1–9,
doi:10.2514/1.D0180. Available online: https://doi.org/10.2514/1.D0180.
39
American Institute of Aeronautics and Astronautics
