Content uploaded by Ming Hou
Author content
All content in this area was uploaded by Ming Hou on Feb 22, 2021
Content may be subject to copyright.
REVIEW
A review of human factors research performed
from 2014 to 2017 in support of the Royal
Canadian Air Force remotely piloted aircraft
system project
G. Robert Arrabito, Ming Hou, Simon Banbury, Blake C.W. Martin, Fahad Ahmad,
and Scott Fang
Abstract: Remotely piloted aircraft systems (RPASs) are tools for military organizations to
help remove humans from dangerous situations and permit operations in severe and inhos-
pitable environments. To support the procurement of an RPAS fleet under Canada’s Strong,
Secure, Engaged 2017 defence policy, the Royal Canadian Air Force (RCAF) under the RCAF
Joint Unmanned Surveillance and Target Acquisition System project (subsequently replaced
by the RCAF RPAS project) funded Defence Research and Development Canada –Toronto
Research Centre to conduct a preliminary investigation of human factors (HF) issues relat-
ing to the performance of the crew in the ground control station (GCS) to control a RPAS.
This paper presents a review of the RCAF research program conducted between 2014 and
2017 that discusses HF issues in RPAS operations and how training is associated with the
HF attributes of decision-making, skills and knowledge, and mission preparation. Also, this
paper presents a training needs analysis methodology and analysis that identified essential
RPAS crew competencies (expressed as the knowledge, skills, and abilities required by each
crew member to perform their respective tasks). Finally, this paper discusses work that
investigated experimentation and evaluation capabilities to support RPAS operator training
and GCS airworthiness certification.
Key words: RCAF, RPAS, ground control station, human factors, airworthiness certification
Résumé : Les systèmes d’aéronefs télépilotés (SATP) sont des outils destinés aux organisa-
tions militaires pour aider à déplacer les humains hors de situations dangereuses et per-
mettre des opérations dans des environnements difficiles et inhospitaliers. Dans le but de
soutenir l’acquisition d’uneflottedeSATPdanslecadredelapolitiquededéfense
Protection, Sécurité, Engagement 2017 du Canada, l’Aviation royale canadienne (ARC) dans
le cadre du projet du Système interarmées de surveillance et d’acquisition d’objectifs au
moyen de véhicules aériens sans pilote (JUSTAS) de l’ARC (remplacé par la suite par le projet
de SATP de l’ARC) a accordé le financement à Recherche et développement pour la défense
Canada –Centre de recherches de Toronto pour mener une recherche préliminaire sur les
facteurs humains (FH) liés aux capacités de l’équipage au poste de contrôle au sol (PCS) à
contrôler un SATP. Le présent document présente un examen du programme de recherche
de l’ARC mené entre 2014 et 2017 qui porte sur les enjeux des FH dans les opérations des
SATP et sur la façon dont la formation est associée aux attributs de FH que sont la prise de
décisions, les compétences/connaissances et la préparation de la mission. En outre, le
présent document présente une méthodologie pour l’analyse des besoins en formation qui
permet de cerner les compétences essentielles des membres d’équipage du SATP
Received 23 February 2020. Accepted 16 November 2020.
G.R.Arrabito,M.Hou,S.Banbury,B.C.W.Martin,F.Ahmad,andS.Fang.Defence Research and Development
Canada, Toronto Research Centre, 1133 Sheppard Avenue West, Toronto, ON M3K 2C9, Canada.
Corresponding author: G. Robert Arrabito (e-mail: Robert.Arrabito@drdc-rddc.gc.ca).
© Her Majesty the Queen in right of Canada 2021.
1
J. Unmanned Veh. Syst. 00: 1–20 (0000) dx.doi.org/10.1139/juvs-2020-0012 Published at www.nrcresearchpress.com/juvs on 25 November 2020.
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
(exprimées sous forme de connaissances, de compétences et d’aptitudes requises par cha-
que membre d’équipage pour exécuter ses tâches respectives). Enfin, le présent document
traite des travaux qui ont porté sur les capacités d’expérimentation et d’évaluation à l’appui
de la formation des opérateurs de SATP et de la certification de navigabilité du PCS. [Traduit
par la Rédaction]
Mots-clés : Aviation royale canadienne (ARC), système d'aéronef télépiloté (SATP), poste de contrôle
au sol (PCS), facteurs humains, certification de navigabilité
Introduction
Canada’s Strong, Secure, Engaged (SSE) 2017 defence policy anticipates procurement and
deployment of medium altitude, long endurance (MALE) remotely piloted aircraft systems
(RPASs), also referred to as unmanned aircraft systems (UASs), that are capable of
conducting surveillance and precision strikes (National Defence 2017). A MALE RPAS is
comprised of a remotely piloted aircraft (RPA), its operators, support equipment, and inter-
faces that allow the operators to operate the RPA from the ground control station (GCS)
through radio frequency and satellite communications (see Fahlstrom and Gleason 2009
for an overview of RPAS).
The Royal Canadian Air Force (RCAF), under the RCAF RPAS project, is defining the
requirementstoprocureaMALERPASfleetinresponsetotheSSE2017defencepolicy
(Garrett-Rempel 2015). To help reduce RPAS life-cycle cost for the anticipated RPAS fleet,
the RCAF is adopting the practice of other defence procurement agencies for the use of
commercial off-the-shelf (COTS)/military off-the-shelf (MOTS) solutions (e.g., Barber and
Parsons 2009). However, the procurement agency does not develop the system design of
the acquisition. As COTS/MOTS acquisition project processes do not permit the same level
of influence over system design, necessary consideration needs to be given on how the
chosen solution will affect the deployment of the system, its operational concepts, impact
on human performance, safety, training requirements, organizational structure, and career
progression (Greenley et al. 2008).
These unidentified limitations present challenges to RPAS project acquisition, imple-
mentations, operations, and performances. To address these challenges, the RCAF tasked
Defence Research and Development Canada (DRDC) –Toronto Research Centre to perform
human factors (HF) research in support of the RCAF Joint Unmanned Surveillance and
Target Acquisition System (JUSTAS) project that was subsequently replaced by the RCAF
RPAS project (Hou 2015). The goal of HF is to examine human interaction with systems, to
facilitate enhanced performance, and to increase safety and user satisfaction (Wickens et al.
2004).
This paper reviews the RCAF HF research performed at DRDC –Toronto Research Centre
from 2014 to 2017 with the aim to: (i) develop training approaches for the RPAS,
(ii) determine HF specifications for possible inclusion in the definition of system require-
ments for the acquisition of the envisioned JUSTAS RPAS fleet, and (iii) develop RPAS GCS
certification recommendations for the RCAF’s operation of military RPASs in nonsegre-
gated airspace. The reviewed DRDC documents are: Banbury et al. (2014a,2014b,2014c,
2015a,2015b,2017),andStewart (2014). Each of these seven documents contains a single
component of the research conducted for the RCAF; collectively, they form the basis of
the reported work in this paper.
The outline of this paper is as follows. We first discuss HF issues in RPAS operations.
Next, we discuss how training is associated with the HF attributes of decision-making, skills
and knowledge, and mission preparation. The development of a training needs methodol-
ogy employed in a training needs analysis for selection and training of knowledge, skills,
2 J. Unmanned Veh. Syst. Vol. 00, 0000
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
and abilities in RPAS crew is then discussed. Finally, we discuss work that investigated
experimentation and evaluation capabilities to support RPAS operator training and GCS
airworthiness certification. The research presented in this paper is a preliminary investiga-
tion of HF issues relating to the crew performance in the RPAS GCS and was conducted to
support Canada’s first procurement of a MALE RPAS fleet.
Human factors issues for RPAS operations
There is considerable demand for RPASs (US GAO 2012). However, the rapid rise in RPAS
employment has been accompanied by high mishap rates. The RPAS mishap rate in 2001
was significantly higher than that of manned aircraft (Williams 2004). More recently, the
mishap rate for RPAS is lower than that of manned aircraft due to adjustments to crew
training curriculum for improving safety (Stanley 2014). Although RPAS mishap rates are
lower than general aviation, they still occur and are attributable to human error (Slenski
2016). United States Col. James A. Marshall, former director of safety for Headquarters Air
Combat Command, said that while his directorate is “never satisfied until the [RPAS]
mishap rate is down to zero,”he is pleased with progress made in the RPA program
(Stanley 2014).
The GCS is the primary interface link between the RPA and its human operators. RPASs
are prone to human error because their operators are deprived of sensory cues (i.e., visual,
auditory, and motion) available to the pilot of a manned aircraft (McCarley and Wickens
2005; Carretta et al. 2017). HF issues in the GCS pertain to situation awareness (SA) and
mental workload. The most widely cited definition of SA is: “::: the perception of the
elements in the environment within a volume of time and space, the comprehension of
their meaning, and the projection of their status in the near future”(Endsley 1995).
Workload is the term used to describe the relationship between task demands and
available operator mental capacity (Parasuraman et al. 2008).
High levels of workload and loss of SA have been attributed to RPAS mishaps
(e.g., Tvaryanas et al. 2006; Arrabito et al. 2010). A relatively high frequency of RPAS mishaps
that occurred over 10 years in the United States Army, Air Force, and Navy or Marines
included HF issues of high workload, attention, and crew coordination and communication
(Tvaryanas et al. 2006). DRDC –Toronto Research Centre conducted a literature review and
consulted with experienced RPAS operators within the United States Air Force to identify
HF issues relating to controlling MALE RPASs (Arrabito et al. 2010). The HF issues were
related to maintaining vigilance, loss of SA, complacency (i.e., over-trusting the automa-
tion), skill degradation, and increased levels of mental workload.
A human systems integration (HSI) approach must be adopted to reduce RPAS HF
mishaps. HSI is a framework in which human capabilities and limitations across various
domains are considered in the context of a dynamic system of people, technology, environ-
ment, tasks, and other systems with the ultimate goal of achieving system resilience and
adaptation, approaching joint optimizations (Cooke and Gawron 2017). There are five HSI
domains that are described by Tvaryanas et al. (2012); these HSI domains are human factors
engineering (HFE), manpower or personnel training, organisational and social, safety and
health, and training. Within each HSI domain, there is a set of HF attributes that determine
the efficiency of that domain in the system; the system here is the RPAS. Table 1 presents
the HSI domains, their descriptions, and their objectives.
One HF attribute in the HSI domain of training relevant to RPAS operation is skills and
knowledge. Inadequate classroom and RPAS simulator training, and insufficient outside
aircraft flying experience can lead to knowledge or skill gaps resulting in RPAS mishaps
(Herz 2008). The emergence of skill, knowledge, and teamwork in RPAS mishaps can be
improved with training (Salas et al. 2006). The goal of training in the military is to achieve
Arrabito et al. 3
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
Table 1. Human systems integration domains, description, and objectives for the remotely piloted aircraft system (RPAS).
Human systems integration domain Description Objective
Human factors engineering •User’s cognitive, physical, sensory, and dynamic abilities
required to perform RPAS-specific operational,
maintenance, and support job tasks
•Integration of human characteristics into RPAS design,
including all aspects of workstation, workspace design,
and system safety
•Maximize user efficiency in the workplace and minimize
risk of injury to personnel and others
Manpower and personnel •Number of required men, women, civilians, and
contractors to operate and maintain the RPAS
•Considers worker’s aptitudes, experience, and other
human characteristics
•Required for RPAS selection processes
Organisational and social •Organisational complexity as a feature of networked
enabled abilities typified by modern RPAS
•Networked enabled capabilities comprising building trust
and confidence between people in separate organisations
for required collaborations
•Enables people to adapt to a more open culture that
requires greater sharing and trust between colleagues and
coalition partners
Safety and health •RPAS design features that minimise risk of injury, acute or
chronic illness, and (or) discomfort of personnel who
operate, maintain, or support the system
•RPAS design features that mitigate the risk of errors or
accidents that result from degraded job performance
•Applying expertise to minimise safety risk to RPAS users
and bystanders occurring as a result of the system
functioning either in a normal or abnormal manner
Training •Specification and evaluation of optimal combination of
instructional systems, education, and on-the-job training
•Required to develop the knowledge, skills, and abilities of
RPAS operators
•Required by personnel to operate and maintain the RPAS
to a specified level of effectiveness under the full range of
operating considerations
Note: Adapted from Tvaryanas et al. (2012, pp. 12–18).
4 J. Unmanned Veh. Syst. Vol. 00, 0000
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
and maintain a high level of mission readiness and operational effectiveness (Carretta
et al. 2017).
One aspect of RPAS GCS airworthiness certification is to train the RPAS crew to operate
in nonsegregated airspace (Neville and Williams 2017). To study RPAS crew training, DRDC
and the United States Air Force Research Laboratory (AFRL; Dayton, OH) jointly designed
and developed a GCS simulator to operate a MALE RPA. The GCS simulator is in support of
the concept proposed by the RCAF to establish RPAS cells at the squadron level as part of
the JUSTAS procurement project. The crew in the GCS at each RPAS cell consists of an air
vehicle operator (AVO), a payload operator (PO), one to two image analysts (IMAs), and
one to two electronic warfare analysts (EWAs). The GCS simulator is called Testbed for
Integrated Ground control station Experimentation and Rehearsal (TIGER; Covas-Smith
et al. 2015;Hou 2015); TIGER has six crew workstations: AVO, PO, image analyst (IMA-A),
image reporter (IMA-R), electronic warfare analyst (EW-A), and electronic warfare reporter
(EW-R; see Table 2 foradescriptionofeachoperatorrole).TheTIGERsimulatorwas
established to make recommendations to the RCAF for assessing operator training needs
and to enable the provision of analysis and recommendations to the RCAF regarding RPAS
GCS airworthiness certification (Hou 2015).
Human factors considerations for GCS airworthiness certification
This section describes work on critical HF attributes related to HSI domains, RPAS HF
issues, and potential solutions for RPAS GCS airworthiness certification.
Human factors attributes
A baseline set of HF airworthiness requirements for the RPAS GCS was established to
provide recommendations regarding the GCS airworthiness certification in support of the
RCAF’s envisioned RPAS platform. These RPAS-specific requirements were investigated
and refined with respect to their HF attributes in related HSI domains with subject matter
experts (SMEs) from RPAS operations, military systems acquisition, airworthiness require-
ments, and HSI process. The potential solutions were then further explored through SME
focus group discussions and a literature review (Stewart 2014).
Mapping of human factors attributes to human systems integration domains
Based on the SME focus group discussions and the literature review (Stewart 2014), eight
critical HF attributes were identified that may cause serious RPAS operational issues:
interface, decision-making, skills and knowledge, situation awareness, teamwork,
Table 2. Operator roles in a remotely piloted aircraft (RPA) system ground control station proposed by the Royal
Canadian Air Force.
Role Function
Air vehicle operator (AVO) Acts as the crew commander, flies RPA, and controls the release of any
weapons
Payload operator Controls RPA cameras, ensures sensor settings are optimal for the current
mission, and controls the RPA targeting laser
Image analyst (IMA-A) Views the video feed from the RPA sensor and identifies any potentially
important events, and communicates these to the IMA-R
Image reporter (IMA-R) Reviews events from IMA-A and produces reports for the RPA tasking
authority and AVO
Electronic warfare analyst (EW-A) Focuses on the RPA electronic support measures and identifies any
potentially important events and communicates these to the EW-R
Electronic warfare reporter (EW-R) Reviews events from EW-A and produces reports for the RPA tasking
authority and AVO
Arrabito et al. 5
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
documentation, mission preparation, and organization. The eight HF attributes were then
mapped to HSI domains. Table 3 providesasampleofRPASHFissuesidentifiedbySMEs
and their recommended solutions aligned to HF attributes and associated HSI domains.
Research directions
The recommendations presented in Table 3 as potential solutions for minimizing RPAS
mishaps, if adopted, will help establish HF requirements for RCAF RPAS GCS air worthiness
certification. As a result, two main research directions were undertaken to address HF
requirements to procure a MALE RPAS fleet for the RCAF, which are discussed in the
following two sections. First, proper selection and training of RPAS crews minimizes the
occurrences of mishaps due to inadequate training of RPAS crews. As such, training of
relevant competencies in RPAS crews is essential for effective functioning of an RPAS.
A detailed description of proposed selection and training of RPAS crews is provided.
Second, experimentation and evaluation of training in a simulated GCS is necessary for
adequate training of RPAS crew. The TIGER simulator (Covas-Smith et al. 2015;Hou 2015)
was used to investigate HF issues in the GCS and develop RPAS GCS airworthiness
certification.
Identification of RPAS GCS operator competencies and training strategies
At the time of this study, the JUSTAS project had no training content for RPAS operations
that described how the RCAF intended to use the envisaged RPAS fleet. The selection and
training of RCAF RPAS crews was determined by a training needs analysis (TNA; Banbury
et al. 2014b,2015b). As part of the TNA, competencies were expressed as the knowledge,
skills, and abilities (KSAs) that are required to perform each of the unique RPAS crew roles
throughout the conduct of the mission. The identified KSAs were then grouped for the
development of training strategies. The TNA methodology is described in this section.
Training needs analysis methodology
The TNA was developed in accordance with the TNA methodology that should be fol-
lowed for JUSTAS in reference to the user populations that will undertake the roles of
AVO, PO, IMA, and EWA in future RCAF RPAS operations (Banbury et al. 2014b). The TNA
included the following steps: (i) identification of RPAS crew competencies, (ii) allocation of
RPAS crew competencies, (iii) refinement of RPAS crew competencies, and (iv) identification
of training strategies for RPAS crew competencies (Banbury et al. 2014b,2015b).
Identification of RPAS crew competencies
Based on the JUSTAS mission functions likely to be undertaken by the future RCAF RPAS
(see Kobierski 2013), a cognitive task analysis (CTA) was conducted to further refine the cog-
nitive requirements associated with the RPAS operations for JUSTAS (Banbury et al. 2014a).
The CTA identified decisions that need to be made by the RPAS operators (either alone or in
collaboration with other crew members) to support higher-level goals. The CTA served as a
first step in the identification of the competencies required for the crew to operate the
RPAS GCS (including psychomotor and cognitive skills) at both an individual and team
level.
KSAs were compiled from a review of the scientific literature and the searches of occupa-
tional databases for civilians that are relevant to the AVO, PO, IMA, and EWA roles.
Additional KSAs were compiled from seven structured interviews with SMEs having opera-
tional experience as an RPAS crew member (AVO, PO, IMA, and EWA). The objectives of the
interviews were to identify the KSAs required to perform each of the four operator roles
and to determine any current training gaps based on SME experiences leading up to and
during previous deployments as RPAS crew members. These KSAs support the context of
6 J. Unmanned Veh. Syst. Vol. 00, 0000
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
Table 3. A sample of RPAS HF issues identified by subject matter experts and their recommended solutions aligned to HF attributes and associated HSI domains.
HF attribute HSI domain RPAS operational issues Potential RPAS solutions
Interface HFE Poor functional design of GCS interface (e.g., limited
field of view)
Consider and address design guidelines to achieve
effective human machine interface (Hou et al.
2014).
Decision-making Training Poor training leads to inadequate risk assessment or
poor judgement in crew and results in:
•checklist errors
•task misprioritizations
•delayed action
RPAS crew member should be tasked to prioritize
missions and follow through with checklists
(Stewart 2014).
Skills and knowledge Training Inadequate training results in:
•poor systems knowledge
•inadequate technical procedural knowledge
Adequate simulations of missions should be
undertaken and crew training is recommended to
be conducted by experienced RPAS crews (Banbury
et al. 2014b).
Situation awareness HFE Poor GCS interface leads to:
•monitoring issues
•channelized attention
Provide auditory and tactile cues (Arrabito et al. 2010).
Teamwork Manpower and personnel Missions can involve multiple crews and as a result
mismanagement of teamwork leads to:
•miscommunication
•errors during handoff procedures
Develop formal procedures for hand-over of RPAS
controls between teams of operators (Stewart 2014).
Efficient communication between teams of operators
(Stewart 2014).
Documentation Safety and health Poor documentation can lead to:
•poor written procedures
•damage of equipment due to unclear and (or)
unrefined instructions
Provide training for maintaining proper records and
documents (Stewart 2014).
Creating highly adaptive, effective, and safe training
procedures through previous lessons, exploratory
studies and mission analysis (Stewart 2014).
Mission preparation Training Poor training in mission preparation leads to:
•inadequate flight planning
•nawareness of resources
Providing adequate flight briefing (Banbury et al.
2015b).
Guide and train crew on how to take into account the
limited (but growing) resources (Stewart 2014).
Organization Training Inadequate training in RPAS operations results in
operational mishaps
Include all procedures for operating the RPAS
(Stewart 2014).
Note: RPAS operational issues extracted from Herz (2008). RPAS, remotely piloted aircraft system; HF, human factor; HSI, human systems integration; HFE, human factors engineering; GCS,
ground control station.
Arrabito et al. 7
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
mission training that will provide the Canadian Armed Forces (CAF) with an intelligence,
surveillance, target acquisition, and reconnaissance (ISTAR) and strike capability. These
sources identified an initial list of 35 KSAs; this list was used as a starting point for the
TNA (Banbury et al. 2014b).
Following the identification of the 35 KSAs, an SME focus group study was conducted to
determine levels of importance placed on each competency by RPAS crew role. There were
different levels of importance placed on some competencies by RPAS role. For example,
critical thinking was considered by SMEs as a competency highly relied upon by AVO,
IMA, and EWA but moderately relied upon by the PO (Banbury et al. 2015b).
The research literature supports the finding that there is an overlap and a variability of
reliance on some competencies by RPAS roles (e.g., see Howse 2011 for a review). For exam-
ple, Mangos et al. (2014) collected ratings of importance for several KSAs from AVO, PO, and
mission commander (MC; the MC is responsible for developing the mission plan based on
the tasking authority requirements). Some competencies were rated of high importance
by AVO and PO but lower importance by MC (e.g., spatial and navigation skills).
Allocation of competencies to RPAS crew members
Following the identification of 35 RPAS crew competencies, the KSAs were mapped to
operator decisions identified in the CTA (Banbury et al. 2014a) and allocated to one or more
of the RPAS crew roles. For example, for the goal of “Establish Contact Track”, there are six
decisions to be made by the operator. One decision is: “Are the RPAS flight parameters con-
figured correctly for contact search?”This decision is assigned to the AVO. In contrast, the
decision “Are the electro-optical and infrared sensors configured correctly for contact
search?”is assigned to both the PO and IMA-A (see Table 4 for KSAs associated with both
these decisions). As a result, certain competencies are relevant to decisions made by one
or more RPAS operator roles.
Refinement of RPAS crew competencies
Following the allocation of competencies to RPAS crew members, three structured inter-
views with one SME were conducted to refine and revise the 35 KSAs. The SME had RCAF
RPAS operational experience, including the training of crews. The extent to which each of
the four crew roles (i.e., AVO, PO, IMA, and EWA) rely upon each competency was verbally
rated as high, medium, or low by the SME. To provide context and a rationale for these
ratings, additional contextual information was provided by the SME for each competency
based on his RPAS operational experience.
Out of the original 35 KSAs, 19 competencies were selected that were consistent with the
training priorities identified by a literature review and by the interviews with the SME in
support of the RPAS crew competency refinement. Sixteen competencies were removed
from the original 35 KSAs because they were labelled by the SME as “basic”competencies
throughout the TNA, they were not prevalent across the CTA (i.e., the competency was
not informative), or RCAF had existing training and framework for those competencies
(Banbury et al. 2015b). These 19 competencies are summarized in Table 5.
Identification of training strategies for RPAS crew competencies
For each of the 19 chosen competencies from the initial list of 35 competencies, the SME
provided suggestions for training strategies. In addition, the SME indicated whether the
competency was to be trained as part of a new program or to be added as a training compo-
nent in an existing RCAF training program.
Relevant training strategies were identified from a review of the open-source literature
relating to each competency deemed to support the high-priority training needs identified
by the SMEs as part of the structured interviews conducted over the course of the
8 J. Unmanned Veh. Syst. Vol. 00, 0000
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
Table 4. Relationship of RPAS operator role and competencies associated with each decision for the goal “Ensure remotely piloted aircraft system (RPAS) is optimised for
searching for contacts”.
RPAS operator analysis Unit of analysis Role Knowledge Skills Abilities
Are the RPAS flight
parameters configured
correctly for contact
search?
Decision AVO •Education and training
•Understanding of
information requirements
•Coordination and prioritization
•Critical thinking
•Judgment and decision making
•Numeracy reasoning
•Oral communication
•Trust in automation
•Working with others
•Collaboration
•Mentoring
•Pattern perception
•Problem sensitivity
•Shared situation awareness
Are the electro-optical/
infrared sensors
configured correctly
for contact search?
Decision PO, IMA-A •Computers and electronics
•Education and training
•Language skills
•Understanding of
information requirements
•Coordination and prioritization
•Judgment and decision making
•Oral communication
•Reasoning
•Working with others
•Attentional control
•Problem sensitivity
•Shared situation awareness
•Stress management
Note: Data adapted from Banbury et al. (2014a, Appendix B, items 5.1.1, 5.1.1.1, and 5.1.1.3) and Banbury et al. (2015b, Appendix A, items 5.1.1, 5.1.1.1, and 5.1.1.3). Presented data are abridged
version of the results for purposes of clarity and brevity. AVO, air vehicle operator; PO, payload operator; IMA-A, image analyst.
Arrabito et al. 9
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
identification of the initial list of 35 RPAS crew competencies (Banburyetal.2014b). For
example, monitoring was indicated by the SME as being a skill for which a suitable training
program was required. Monitoring was defined as active monitoring of current mission s
tatus and technology (e.g., instrumentation, gauges, indicators, etc.) accessible through
the use of hardware and software in the GCS (Banbury et al. 2015b).
Training strategies were reviewed from the literature to determine a suitable training
program for monitoring skills (Banbury et al. 2015b). One training strategy identified in
the literature was from a study in which training of monitoring skills was examined in
helicopter pilots (Potter et al. 2014). A group of helicopter pilots completed a computer
course on monitoring and cross-checking skill training, whereas those in the control condi-
tion had no such pretraining. Subsequently, all participants flew two scenarios in a flight
simulator. Participants who completed the training module showed more monitoring types
of behaviours in simulated flights than the control group.
Following identification of training strategies for the list of competencies, a two-step
approach was taken to associate training strategies with the selected 19 competencies.
These steps are presented in the following subsections.
Classification of competencies by training delivery and type of training content
The first step to associate the 19 competencies with suitable training strategies involved
classifying each competency by training delivery and by type of training content. The
delivery of training can be characterised in terms of individual or group training.
Individual training focuses on the training content and strategy on individual learning
goals. In this context, individual training can be independent and self-paced (e.g., individual
study or computer-based training) or delivered in a group setting (e.g., classroom). Group
(mission crew) training requires all crew personnel be brought together as co-located or
distributed teams to ensure that they work and train optimally on shared tasks and
operational objectives. For example, group training of RPAS crews might consist of flight
training (to include simulations) leading to a crew qualification. A competency could also
be trained by a combination of both types of training (i.e., progressing from individual to
group training).
Training content is characterised in terms of general or RPAS-specific training. General
training relates to training within the CAF but not related directly to the operation of the
RPAS. RPAS-specific training is normally provided by the Squadron Standards and
Training Flights, although from time-to-time, depending upon the RPAS platform and
payload package, additional contracted training may be required. It is possible that a
Table 5. The 19 competencies (expressed as the knowledge (K), skills (S), and abilities (A)) identified by the
literature review and three structured interviews with one subject matter expert having operational
experience of remotely piloted aircraft systems, including the training of crews.
Knowledge Skills Abilities
•K3 Computers and electronics
•K4 Team command structure
•K5 Knowledge, and adherence to
the law
•K6 Culture
•K7 Psychology
•S4 Monitoring
•S5 Coordination and
prioritisation
•S6 Critical thinking
•S10 Trust in automation
•S15 Working with others
•S16 Writing
•A1 Multi-limb coordination
•A3 Stress management
•A4 Collaboration
•A6 Shared situation awareness
•A7 Mentoring
•A8 Working in multi-dimensional
spaces
•A10 Attentional control
•A11 Eye-hand coordination
10 J. Unmanned Veh. Syst. Vol. 00, 0000
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
competency can be trained both generally (as basic training) and expanded upon during
RPAS-specific training.
The second step involved grouping the 19 competencies under training content and
training delivery (see Table 6). The same formatting was used for competencies that shared
similar properties or training strategies (as identified in the literature review).
The results of this step clearly show that the RPAS crew’s reliance on the 19 competen-
cies, as rated by the SME, are mostly related to RPAS-specific topics and should be delivered
using a combination of individual and group-based training. For three competencies (team
command structure, culture, and psychology), a combination of individual and group-based
training was identified, suggesting that these competencies need to be trained both first
individually to cover the basic knowledge and then as a group to apply that knowledge in
crew settings. Similarly, two competencies (multi-limb coordination and shared situation
awareness) were identified to require the combination of general and RPAS-specific train-
ing. Once again, these competencies in general training (e.g., piloting skills) need to be
further refined for RPAS-specific operations and crew complements.
Classification of competencies into training clusters
From the relationship among the SME-prioritised competencies identified in Table 6,
three training strategy–competency relationship clusters were identified. Each training
strategy–competency relationship cluster was composed of competencies that were related
to each other in terms of the type of training strategies. For example, training strategies to
target the training strategy–competency relationship cluster named teamwork would tar-
get team command structure, psychology, monitoring, working with others, collaboration,
shared situation awareness and mentoring. The three training–strategy competency rela-
tionship clusters were named: (i)teamwork,(ii) AVO relationship with technology, and
(iii) crew collaboration to determine pattern of life (POL) and conduct reporting. Table 7
presents the grouping of competencies under three training strategy–competency
relationship clusters.
The cluster relating to “teamwork”encompasses collaboration and information sharing
among crew members and is necessary for making critical decisions. Although the AVO role
Table 6. Competencies grouped under training content (either general or RPAS-specific) and training delivery
(individual, group, or both individual and group).
Training
content
Training delivery
Individual Group
Both individual
and group
General •S16: Writing
•A1: Multi-limb coordination
•A3: Stress management
•A11: Eye-hand coordination
•A6: Shared situation
awareness
•A7: Mentoring
•S6: Critical thinking
RPAS-specific •K3: Computer and electronics
•K5: Knowledge of the law
•K7: Psychology
•S5: Coordination and prioritization
•S10: Trust in automation
•A1: Multi-limb coordination
•A8: Working in multi-dimensional
spaces
•A10: Attentional control
•S4: Monitoring
•S15: Working with others
•A4: Collaboration
•A6: Shared situation
awareness
•K4: Team command
structure
•K6: Culture
•K7: Psychology
Note: K, knowledge (bold); S, skills (italic); and A, abilities (underline). Competencies that share similar training strategies are in
the same text formatting. Competencies are presented sequentially to improve readability. Data adapted from Banbury et al. (2015b).
Arrabito et al. 11
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
assumes overall responsibility for the mission, all individual crew members must fully
understand and be equipped through appropriate training to collaborate with other
members to support the decisions that must be made.
In contrast, within the context of future RPAS operations, the cluster relating to “AVO
relationship with technology”will be largely focused on the interaction with the auto-
mated systems within the GCS. If the relationship between operator and automated system
is not optimal, issues relating to lack of trust, “out-of-the-loop”problems, skill fade, and (or)
increased workload are likely to occur. In the future, the AVOs are more likely to be trained
to plan and execute missions using automated systems.
Finally, the cluster relating to “crew collaboration to determine POL and conduct
reporting”is relevant to the development of cultural understanding and is essential to
the formation of the POL that forms the basis of the tactical decisions and reporting
performed by the crew. This understanding is deeper for the EWA and IMA crew roles as
they work directly on producing the intelligence products for the RPAS tasking authority.
Limitations of training needs analysis
The TNA had some limitations. First, there was an identification of only general training
strategies from the refinement of the competencies required by each RPAS crew member
(Banbury et al. 2015b). There was no specific identification and development of training
content because, at the time of this study, the capabilities and functional requirements of
the RPAS and GCS had not yet been determined by the JUSTAS project. Second, only one
SME was interviewed three times to refine the initial list of 35 competencies (Table 5).
There was a lack of available RCAF RPAS operators during this study; interviews with more
Table 7. Competencies grouped under training strategy–competency relationship clusters.
Number Clusters Knowledge, skills, abilities
1 Teamwork •K4: Team command structure
•K7: Psychology
•S4: Monitoring
•S15: Working with others
•A4: Collaboration
•A6: Shared situation awareness
•A7: Mentoring
2 AVO relationship with technology •K3: Computer and electronics
•K7: Psychology
•S5: Coordination and prioritization
•S10: Trust in automation
•A1: Multi-limb coordination
•A3: Stress management
•A8: Working in multi-dimensional spaces
•A10: Attentional control
•A11: Eye-hand coordination
3 Crew collaboration to determine of
pattern life and conduct reporting
•K3: Computer and electronics
•K4: Team command structure
•K5: Knowledge of the law
•K6: Culture
•K7: Psychology
•S6: Critical thinking
•S16: Writing
•A1: Multi-limb coordination
•A8: Working in multi-dimensional spaces
Note: K, knowledge (bold); S, skills (italic); and A, abilities (underline). Competencies are presented
sequentially to improve readability. Data adapted from Banbury et al. (2015b).
12 J. Unmanned Veh. Syst. Vol. 00, 0000
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
SMEs will be conducted in the future. Third, the study of the relationships among compe-
tencies to identify high-level clusters of related competencies was entirely subjective and
did not use a systematic methodology. A more rigorous technique should be used to gain
a robust insight into how the identified 35 competencies (Banbury et al. 2014b)mightbe
clustered. Despite these limitations, the identification of conceptual relationships between
RPAS crew competencies helps to identify high-level strategies for training-related
competencies. Indeed, the body of knowledge developed at DRDC –Toronto Research
Centre can be used to help develop RPAS GCS crew training plans from a bottom-up
perspective and support the development of the Occupational Specification and
Qualification Standard for RPAS crew members. This will help serve as a valuable resource
for the RCAF and the operators of its future RPAS fleet.
Experimentation and evaluation to support training for GCS airworthiness certification
This section describes experimentation and evaluation to support training for GCS
airworthiness certification. Training and evaluation were assessed using the DRDC RPAS
GCS simulator, called TIGER (Covas-Smith et al. 2015;Hou 2015). A brief description of
TIGER is first provided followed by three empirical trials of TIGER.
TIGER description
The TIGER platform comprises custom and commercial off-the-shelf technologies
(Covas-Smith et al. 2015;Hou 2015). Piloting and sensor operation can be performed by
human operators or according to programmed scripts. The TIGER platform permits
considerable flexibility with respect to location of components (workspace arrangement),
training or testing focus (team, part team, or individual), and environmental perspective
(aerial overflight, aerial surveillance, or ground-level viewing), as well as a command and
control cell for information processing, exploitation, and dissemination. TIGER can
simulateastandaloneGCSand(or)supportnetwork-centric warfare for distributed
exercises. Each workstation consists of associated monitors and the software enabling
realistic mission emulation (e.g., threat generation software, flight simulation software,
simulated sensor controls, imagery analysis software, intelligence gathering, communica-
tions, chat, and mapping).
The six TIGER crew workstations (Table 2) are reconfigurable to different workspace lay-
outs. The “classroom”GCS workspace layout has three pairs of operators (i.e., AVO–PO,
IMA-A–IMA-R, and EW-A–EW-R) seated in three rows facing the same direction (see Fig. 1);
this workspace layout was used for the first two TIGER empirical trials (Banbury et al.
2014c,2015a).
The TIGER workspace layout can also simulate distributed team or co-located situations.
The TIGER distributed workspace layout (Fig. 2) has three pairs of operators physically
separated (e.g., separated with acoustic panels in the same room) that required communica-
tion through radio and chat. This layout simulates a concept of operations in which the
AVO and PO might be in theatre, and the IMA and EW personnel are in their respective
headquarters. This crew arrangement could have benefits such as reduced need for forward
deployment of crew members, keeping analysts near their technical and administrative
support, and the ability for intelligence analysts to form crews with system operators
located in many different locations.
The TIGER co-located crew members face toward a common centre that is referred to as a
“Boardroom”GCS workspace layout (Fig. 3), which is an optimized configuration based on
crew visual and verbal interaction requirements identified using the results of a CTA
(Banbury et al. 2014a). In the boardroom configuration, personnel are gathered around a
central table, or a group of tables, to encourage interactions and collaboration.
Arrabito et al. 13
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
TIGER empirical trials
Three empirical trials were conducted to demonstrate and review TIGER capability. Each
trial comprised a structured, human-in-the-loop (HITL) simulation-based evaluation for a
crew of a MALE RPA. Several measures were used to ascertain the HMI’s impact on operator
task completion, SA, mental workload, and trust in the system (see Table 8).
CAF personnel (experienced in RPAS crew roles of AVO, PO, IMA, and EWA) served as
participants. For each trial, the participants completed a series of critical tasks within a
simulated mission that is representative of CAF RPAS mission functions related to an
RPAS ISTAR mission environment whereby multiple ground contacts need to be identified,
targeted, and engaged (see Kobierski 2013). There was a different mission scenario for each
trial. Trials 1 and 2 were approved by the DRDC Human Research Ethics Committee (HREC),
and AFRL Institutional Review Board; Trial 3 was approved by the DRDC HREC. All partici-
pants signed an informed consent form.
Trial 1
The objectives of Trial 1 were to demonstrate and review TIGER capabilities and establish
baseline crew performance with a four-person RPAS crew consisting of an AVO, PO, IMA-A,
and EW-A (Banbury et al. 2014c). Prior to the evaluation, the RPAS crew were familiarised
Fig. 1. Schematic diagram of the TIGER crew stations configured as classroom layout. AVO, air vehicle operator;
CGF, computer generated forces; IMA-A, image analyst; IMA-R, image reporter; EW-A, electronic warfare analyst;
EW-R, electronic warfare reporter; EXP, experimenter; IOS, instructor operator station; PO, payload operator; RP,
white-force role player.
RP
POAVO
EW-A
IMA-A IMA-R
EW-R
EXP-1
EXP-2
IOS CGF
Crew Stations
Exercise
Management
& Data
Collection
14 J. Unmanned Veh. Syst. Vol. 00, 0000
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
with the major components of TIGER, providing them with overview information about
system functionality and the HMI.
Training was sufficient to use the system. There was an indication that the team
functioned well over the entire mission based on the behavioural markers. Adequate
performance was shown by the AVO for handling of RPAS. However, there was large
divergence in individual crew member’s SA over many ISTAR-related mission vignettes.
Finally, high level of workload and low trust was shown by all crew members
(see Banbury et al. 2014cfor full results of Trial 1).
Trial 2
The objectives of Trial 2 were to demonstrate and review TIGER capabilities and estab-
lish baseline crew performance with two six-person RPAS crews consisting of an AVO, PO,
IMA-A, IMA-R, EW-A, and EW-R (Banbury et al. 2015a). There were 10 participants who were
divided into two six-person RPAS crews, crew A and B. One participant familiar with image
analysis and reporting filled the role of IMA-A on crew A and IMA-R on crew B. One of the
personnel familiar with electronic warfare analysis and reporting filled the role of EW-A
and EW-R for both crews A and B.
The two crew member teams reported that training was insufficient. Crew A and crew B
performed reasonably well across all aspects of the mission. RPAS handling quality was
Fig. 2. Schematic diagram of the TIGER crew stations configured as distributed layout. AVO, air vehicle operator;
CGF, computer generated forces; IMA-A, image analyst; IMA-R, image reporter; EW-A, electronic warfare analyst;
EW-R, electronic warfare reporter; EXP, experimenter; IOS, instructor operator station; PO, payload operator; RP,
white-force role player.
RP
EW-A
IMA-A IMA-R
EW-R
EXP-1
EXP-2
IOS CGF
Crew Stations
Exercise
Management
&Data
Collection
POAVO
Room
Divider
Arrabito et al. 15
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
adequate. SA scores suggested that the AVOs and POs were able to supply sufficient mental
resources to meet the demands of the mission and achieve relatively high levels of SA,
which coincided with good levels of performance as rated by the SME. Crew B showed
higher levels of SA than crew A. Both crews showed moderate levels of workload. The
majority of participants reported HMI usability issues relating to placement of menu items
and icons, the general arrangement of the interface, system prompts, and error messages
(see Banbury et al. 2015afor full results of Trial 2).
Trial 3
The objective of Trial 3 was to evaluate three RPAS GCS workspace layouts (see Figs. 1–3)
with a six-person RPAS crew consisting of an AVO, PO, IMA-A, IMA-R, EW-A, and EW-R
(Banbury et al. 2017). The same RPAS six-person crew performed a different mission for each
workspace layout. The following order of missions was randomly determined: distributed,
classroom, and boardroom.
All crew roles had sufficient levels of training. Behavioural markers were rated more pos-
itively in the boardroom condition than the other two layout conditions despite relatively
high ratings in the latter. Data from the chat logs showed that crew members adopted their
Fig. 3. Schematic diagram of the TIGER Crew stations configured as boardroom layout. AVO, air vehicle operator;
CGF, computer generated forces; IMA-A, image analyst; IMA-R, image reporter; EW-A, electronic warfare analyst;
EW-R, electronic warfare reporter; EXP, experimenter; IOS, instructor operator station; PO, payload operator; RP,
white-force role player.
RP
POAVO
IMA-A IMA-R EW-A EW-R
EXP-1 EXP-2
IOS CGF
Crew Stations
Exercise
Management
& Data
Collection
16 J. Unmanned Veh. Syst. Vol. 00, 0000
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
role seriously. Handling qualities of the simulation were good. Initially, Banbury et al. (2017)
reported that the boardroom configuration resulted in higher SA; however, a closer
re-examination of the same data shows higher SA scores in the distributed GCS workspace
layout, followed by the boardroom GCS workspace layout, then the classroom GCS work-
space layout. Consistent with both Banbury et al. (2017) and the present findings, the
observed SA pattern at the crew level was consistent across the majority of the crew, except
for the AVO and IMA-R. The crew was always near optimal workload levels. Interestingly,
the crew’s perception of their own team effectiveness was relatively high in all layout
conditions. All dimensions of team effectiveness were rated higher on average in the board-
room condition than the other two workstation layouts (see Banbury et al. 2017 for full
results of Trial 3).
Limitations of TIGER empirical trials
The three TIGER empirical trials had some limitations. First, data were based on a small
sample size that did not permit inferential statistics. Second, the training on TIGER was
developed during the trials and the software for the intelligence workstations of TIGER
was updated between trials, which were not consistent across trials. Third, although the
missions performed by the crews included the same types of tasks, the narratives and tim-
ings were different prohibiting a quantitative comparison of mission completion.
Table 8. Operator measures of performance for three Testbed for Integrated Ground control station
Experimentation and Rehearsal (TIGER) empirical trials.
Operator measures of performance Description
Speed of task completion (Trial 1) Objective measure for operator performance for predefined
sequences of events (e.g., time taken to detect target).
Accuracy of task completion (Trial 1) Objective measure of operator performance of predefined sequence
of events (e.g., correct target identification).
Behavioural markers (Trials 1–3) Behavioural markers (Flin and Martin 2001) designed to rate
operators’and team-based behaviours in response to
predetermined scenario events (e.g., Operator: mission analysis;
e.g., Team: communication).
RCAF training evaluator (with operational experience as AVO in
RPAS missions) rated participants against five-point scale of
competence for each behavioural marker.
Chat log (Trial 3) Allowed for tracking crew’s actual knowledge about the situation.
RPAS handling qualities (Trials 1–3) Cooper–Harper scale (Cooper and Harper 1969) described handling
qualities on 1–10 scale.
Situation Awareness Rating Technique
(Trials 1–3)
Subjective rating tool for measurement of situation awareness (Taylor
1990).
Situation Awareness and Workload
In-Flight Measure (Trials 1–3)
Participants provided subjective ratings of situation awareness and
workload via adapted three-point rating scale (Banbury et al. 2004).
NASA Task Load Index (Trials 1–3) Subjective workload assessments on operator(s) working with
human–machine systems (Hart and Staveland 1988).
Human–computer trust (Trial 1) Subjective assessment was used to measure the degree of operator
trust in the TIGER’s automated systems (Madsen and Gregor 2000).
Utility and usability (Trial 1) Perceived usefulness: degree to which person believes that using a
particular system would enhance his/her job performance
(i.e., utility).
Perceived ease of use: degree to which person believes that using
particular system would be free from effort (i.e., usability).
Workstation usability (Trials 2 and 3) Participants rated their workstation against human factor
engineering criteria.
Team effectiveness (Trial 3) Perception by crew on degree to which they feel they are effective as a
crew in performing tasks associated with mission to completion.
Arrabito et al. 17
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
Despite these limitations, results of the three trials provide initial empirical evidence
that can be used to guide the design of RPAS crew configuration, crew training approaches,
GCS workspace layout, and HMI requirements. Indeed, TIGER was successfully employed
during Exercise Virtual 2016 (Exercise Virtual is an annual distributed simulation event
using diverse technologies to create a common synthetic environment for highly efficient
and effective training for the RCAF) to provide simulation of RPA overwatch (Martin et al.
2018). Both AVO and PO crew members operated the RPA in Toronto, Ontario, whereas
CH-146, CH-147, CC-130J, CF-18, Joint Terminal Attack Controller, Tomahawk missile systems,
and enemy forces were simulated in Gagetown, New Brunswick; Valcartier, Quebec;
Edmonton, Alberta; Petawawa, Ontario; Halifax, Nova Scotia; and Trenton, Ontario, with
the control center for the exercise located at the Canadian Forces Aerospace Warfare
Centre in Trenton and additional support in Ottawa, Ontario. This usage reflects the distrib-
uted workstation arrangement of Trial 3, in that the AVO and PO were not co-located with
other players. The crew was nevertheless successful in their tasks; however, there was no
measurement of efficacy nor comparison with other arrangements. This instance demon-
strates TIGER’s potential to play in distributed simulation exercises and to provide versatil-
ity within an exercise.
Conclusion
This paper reviewed preliminary HF research conducted at DRDC –Toronto Research
Centre from 2014 to 2017 in support of the RCAF to define the requirements for the procure-
ment of a MALE RPAS fleet (National Defence 2017). HF work plays a crucial role within the
acquisition and operation of RPAS for the RCAF RPAS project (Hou 2015). Nontechnical skills
for RPAS crews are critical; selecting personnel with the right KSAs is equally important.
Adequate simulations of mission scenarios should be undertaken, and crew training is rec-
ommended to be conducted by experienced RPAS crews (Banbury et al. 2014b). Indeed, the
RCAF is leveraging the DRDC RPAS GCS simulator, called TIGER (Covas-Smith et al. 2015;
Hou 2015) for Joint RPAS Operator Training. Continued experimentation and evaluation
will have a vital impact on the effectiveness of RPAS crews for the safe operation of an
RPAS fleet capable of supporting a broad spectrum of activities for domestic and overseas
missions.
Finally, this study contains some limitations that relate to the provided level of detail in
the data and rigour by the approach taken for the TNA and TIGER empirical trials. We had
access to a small sample of available RCAF RPAS operators, the study was conducted in a
shortened timeline, and the generalizability of the findings was reduced by the narrow
scope of the study. Despite these limitations, the research described in this paper provides
a foundation for the development of training concepts to support the RCAF’s future RPAS
fleet. In particular, our findings highlight the importance that COTS and MOTS solutions
require consideration on how the chosen solution will affect the deployment of the system,
its operational concepts, impact on human performance, safety, training requirements,
organizational structure, and career progression (Greenley et al. 2008). We hope this work
stimulates additional research that identifies essential RPAS crew competencies and RPAS
GCS airworthiness certification recommendations. The goal is for military organizations
to adopt an HSI approach to help reduce RPAS mishaps attributable to human error.
References
Arrabito, G.R., Ho, G., Lambert, A., Rutley, M., Keillor, J., Chiu, A., et al. 2010. Human factors issues for controlling
uninhabited aerial vehicles: Preliminary findings in support of the Canadian forces joint unmanned aerial
vehicle surveillance target acquisition system. Technical Report No. TR-2009-043. Defence Research and
Development Canada, Toronto, Ont., Canada.
18 J. Unmanned Veh. Syst. Vol. 00, 0000
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
Banbury, S., Dudfield, H., and Hoermann, H.J. 2004. Development of novel measures to assess the effectiveness of
commercial airline pilot situation awareness training. In Proceedings of the Human Factors and Ergonomics
Society 48th Annual Meeting, New Orleans, La., USA, 20–24 September. pp. 80–84. doi: 10.1177/
154193120404800118.
Banbury, S., Baker, K., Tremblay, S., and Proulx, R. 2014a. TA1: Unmanned air system operator information flow and
cognitive task analyses. Contract Report No. DRDC-RDDC-2014-C307. Defence Research and Development Canada,
Toronto, Ont., Canada.
Banbury, S., Forbes, K., Pronovost, S., Tremblay, S., and Proulx, R. 2014b. TA3: Ground control station training needs
analysis and performance modeling. Contract Report No. DRDC-RDDC-2014-C211. Defence Research and
Development Canada, Toronto, Ont., Canada.
Banbury, S., Tremblay, S., and Gagnon, J.F. 2014c. TA4: Human factors support to unmanned aircraft system ground
control station experimental trial at Air Force Research Laboratory. Contract Report No. DRDC-RDDC-2015-C213.
Defence Research and Development Canada, Toronto, Ont., Canada.
Banbury, S., Pelletier, S., and Baker, K. 2015a. TA8: Unmanned aircraft system ground control station human
machine interface and workspace refinement, and human factors engineering trial support. Contract Report
No. DRDC-RDDC-2015-C144. Defence Research and Development Canada, Toronto, Ont., Canada.
Banbury, S., Pelletier, S., Baker, K., Tremblay, S., and Proulx, R. 2015b. TA6: Development of an anticipatory compe-
tency analysis for JUSTAS. Contract Report No. DRDC-RDDC-2015-C142. Defence Research and Development
Canada, Toronto, Ont., Canada.
Banbury, S., Pronovost, S., Pelletier, S., and Gagnon, J F. 2017. TA11: Human factors support to evaluation of ground
control station workspace options (Part II). Contract Report No. DRDC-RDDC-2017-C176. Defence Research and
Development Canada, Toronto, Ont., Canada.
Barber, E., and Parsons, N. 2009. A performance model to optimise the capability choices made in the procurement
phase within the Australian Defence Force. Inter. J. Def. Acquis. Manage. 2:32–48.
Carretta, T.R., Rose, M.R., and Bruskiewicz, K.T. 2017. Selection methods for operators of remotely piloted aircraft
systems. In Remotely piloted aircraft systems: A human systems integration perspective.Edited by N.J. Cooke,
L.J. Rowe, W. Bennett Jr., and D.Q. Joralmon. John Wiley & Sons Ltd., West Sussex, UK. pp. 137–163. doi: 10.1002/
9781118965900.ch6.
Cooke, N.J., and Gawron, V. 2017. Human systems integration for remotely piloted aircraft systems. In Remotely
piloted aircraft systems: A human systems integration perspective. Edited by N.J. Cooke, L.J. Rowe, W. Bennett
Jr., and D.Q. Joralmon. John Wiley & Sons Ltd., West Sussex, UK, pp. 1–14. doi: 10.1002/9781118965900.ch1.
Cooper, G.E, and Harper, R.P., Jr.1969.The use of pilot rating in the evaluation of aircraft handling qualities. Report
No. NASATN-D-5153. NASA Ames Research Center, Moffatt Field, Calif., USA.
Covas-Smith, C.M., Grant, S.C., Hou, M., Joralmon, D.Q., and Banbury, S. 2015. Development of a Testbed for
Integrated Ground Control Station Experimentation and Rehearsal (TIGER): Training remotely piloted aircraft
operations and data exploitation. In Unmanned systems. Association for Unmanned Vehicle Systems
International (AUVSI), Arlington, Va., USA. pp. 1–11.
Endsley, M.R. 1995. Toward a theory of situation awareness in dynamic systems. Hum. Factors, 37(1): 32–64. doi:
10.1518/001872095779049543.
Fahlstrom, P.G., and Gleason, T.J. 2009. Introduction to UAV systems. 3rd ed. UAV Systems, Inc., Columbia, Md.,
USA.
Flin, R., and Martin, L. 2001. Behavioral markers for crew resource management: A review of current practice. Inter.
J. Aviat. Psychol. 11(1): 95–118. doi: 10.1207/S15327108IJAP1101_6.
Garrett-Rempel, D. 2015. Will JUSTAS prevail? Procuring a UAS capability for Canada. Royal Can. Air Force J. 4(1):
19–31.
Greenley, M., Scipione, A., Brooks, J., and Salwaycott, A. 2008. The development and validation of a human systems
integration (HSI) program for the Canadian Department of National Defence (DND). Contract Report No. CR-2008-
005. Defence Research and Development Canada, Toronto, Ont., Canada.
Hart, S.G., and Staveland, L.E. 1988. Development of the NASA TLX (Task Load Index): Results of empirical and theo-
retical research. In Human mental workload. Edited by P.A. Hancock and N. Meshkati. Elsevier Science,
Amsterdam, Netherlands. pp. 139–183. doi: 10.1016/S0166-4115(08)62386-9.
Herz, R.P. 2008. Assessing the influence of human factors and experience on predator mishaps. Unpublished
doctoral dissertation, Northcentral University, Scottsdale, Ariz., USA.
Hou, M. 2015. Report on the development of Testbed for Integrated GCS Experimentation and Rehearsal (TIGER).
Scientific Letter No. DRDC-RDDC-2015-L282. Defence Research and Development Canada, Toronto, Ont., USA.
Hou, M., Banbury, S., and Burns, C. 2014. Intelligent adaptive systems: An interaction-centered design perspective.
CRC Press, Boca Raton, Fla., USA. doi: 10.1201/b17742.
Howse, W.R. 2011. Knowledge, skills, abilities, and other characteristics for remotely piloted aircraft pilots and
operators. Report No. DAS 2011-04, Air Force Personnel Center, Randolph Air Force Base, Randolph, Tex., USA.
Kobierski, R.D. 2013. UAV operations mission analysis and composite scenario. Contract Report No. CR-2013-116.
Defence Research and Development Canada, Toronto, Ont., USA.
Madsen, M., and Gregor, S. 2000. Measuring human-computer trust. In Proceedings of Eleventh Australasian
Conference on Information Systems, pp. 6–8.
Mangos, P., Vincenzi, D., Shrader, D., Williams, H., and Arnold, R. 2014. UAS cross platform JTA. NAMRU-D Report
No. 14-44. Naval Medical Research Unit Dayton, Wright-Patterson AFB, Ohio, USA.
Arrabito et al. 19
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
Martin, B.C.W., Grant, S.C., and McColl, D.J. 2018. Evaluation of Exercise Virtual 16. Scientific Letter No. DRDC-
RDDC-2018-L054. Defence Research and Development Canada, Toronto, Ont., USA.
McCarley, J.S., and Wickens, C.D. 2005. Human factors implications of UAVs in the national airspace. Technical
Report No. AHFD-05-05/FAA-05-01. Aviation Human Factors Division, University of Illinois at Urbana-
Champaign, Savoy, Ill., USA.
National Defence. 2017. Strong, secure, engaged: Canada’s defence policy. Available from http://publications.gc.ca/
site/eng/9.835971/publication.html [accessed 12 March 2020].
Neville, K., and Williams, K. 2017. Integrating remotely piloted aircraft systems into the national airspace system. In
Remotely piloted aircraft systems: A human systems integration perspective. Edited by N.J. Cooke, L.J. Rowe, W.
Bennett Jr., and D.Q. Joralmon. John Wiley & Sons Ltd., West Sussex, UK. pp. 197–222. doi: 10.1002/
9781118965900.ch9.
Parasuraman, R., Sheridan, T.B., and Wickens, C.D. 2008. Situation awareness, mental workload, and trust in auto-
mation: Viable, empirically supported cognitive engineering constructs. J. Cognit. Eng. Decis. Mak. 2(2): 140–160.
doi: 10.1518/155534308X284417.
Potter, B.A., Blickensderfer, E.L., and Boquet, A.J. 2014. Training monitoring skills in helicopter pilots. Aviat. Space
Environ. Medic. 85(5): 543–549. doi: 10.3357/ASEM.3771.2014. PMID: 24834569.
Salas, E., Wilson, K.A., Burke, C.S., and Wightman, D.C. 2006. Does crew resource management training work? An
update, an extension, and some critical needs. Hum. Factors, 48(2): 392–412. doi: 10.1518/001872006777724444.
PMID: 16884057.
Slenski, G. 2016. Analysis of UAV military aircraft mishaps. Available from https://www.researchgate.net/
publication/327135551_Analysis_of_UAV_Military_Aircraft_Mishaps [accessed 28 May 2020].
Stanley, S. 2014. RPAs meet mission goals safe and on time. Available from https://www.af.mil/News/Article-Display/
Article/485651/rpas-meet-mission-goals-safe-and-on-time/ [accessed 9 July 2020].
Stewart, A. 2014. UAS GCS human systems integration (HSI) guidance and human factors (HF) airworthiness consid-
erations. Contract Report No. DRDC-RDDC-2014-C212. Defence Research and Development Canada, Toronto, Ont.,
Canada.
Taylor, R.M. 1990. Situation Awareness Rating Technique (SART): The development of a tool for aircrew systems
design. In Situation awareness in aerospace operations. AGARD-CP478, NATO-AGARD, Neuilly Sur Seine, France.
pp. 3/1–3/17.
Tvaryanas, A., Singer, R., and Palmer, R. 2012. The NATO unmanned aircraft systems human systems integration
guidebook. Technical paper No. AFRL-RH-WP-TP-2012-0047. Wright-Patterson Air Force Base, Dayton, Ohio, USA.
Tvaryanas, A.P., Thompson, W.T., and Constable, S.H. 2006. Human factors in remotely piloted aircraft operations:
HFACS analysis of 221 mishaps over 10 years. Aviat. Space Environ. Med. 77(7): 724–732. PMID: 16856358.
US GAO 2012. Nonproliferation: Agencies could improve information sharing and end-use monitoring on
unmanned aerial vehicle export. Report No. GAO-12-536. United States Accountability Office, Washington, D.C.,
USA.
Wickens, C.D., Lee, J.D., Liu, Y., and Becker, S.G. 2004. An introduction to human factors engineering. 2nd ed.
Pearson Prentice Hall, Upper Saddle River, N.J., USA.
Williams, K.W. 2004. A summary of unmanned aircraft accident/incident data: Human factors implications. Report
No. DOT/FAA/AM-04/24. FAA Civil Aerospace Medical Institute, Oklahoma City, Okla., USA.
20 J. Unmanned Veh. Syst. Vol. 00, 0000
Published by NRC Research Press
J. Unmanned Veh. Sys. Downloaded from cdnsciencepub.com by Robert Arrabito on 02/08/21
For personal use only.
