Content uploaded by Sherry Draisey
Author content
All content in this area was uploaded by Sherry Draisey on Apr 03, 2015
Content may be subject to copyright.
Human Audio/Vestibular System: Data Input Channels for Robotic Force
and Moment Sensor Measurements
Sherry Draisey
Good Vibrations Engineering Ltd
sherry@gve.on.ca
Mayes Mullins
Good Vibrations Engineering Ltd.
crs0546@ca.inter.net
Abstract
The design goal was development of an intuitive
human machine interface for force and moment data
from space robotic operations. This paper defines
overall requirements and goals. It describes
experimental approaches used to evaluate our ‘nature’
inspired solution. The final portion of the paper
discusses the design and prototyping of the segment
of the problem which has lead to our first product.
One of nature’s ways of presenting multiple degree of
freedom (dof), vector data is through our audio and
vestibular systems. This directional capability is
being applied as a human machine interface (HMI)
for robotic force sensing.
Human audio direction ability is accurate except for
sounds generated above and behind our heads. This
inaccuracy has lead us to the development of the
vestibulator.
The vestibulator is a wireless device which applies
low levels of current, to the human subject mastoid
bones through surface mounted electrodes. These
induce perceptions of tilt. The polarity of the signals
provide directional stimulus.
1. Introduction
From a design perspective, the work described in this
paper follows the milestones of:
• goal
• design concept/inspiration
• requirements
• design approach
• demonstration/experimental evaluation
•design
• prototype build
It started with a goal to add more human capabilities,
i.e. force sensing, to telerobotic operations. It has
split into two streams. Figure 1, shows the design
path which has lead from our original goal to the
vestibulator prototypes.
Figure 1
Engineering Problem Decomposition
Teleoperational robotic manoeuvres are not as
common in the robotic world as ‘pick and place’
robotics. This may account for the lack of force
sensors implemented in robotic applications. The
Canadarms (Shuttle Remote Manipulator Systems,
SRMS) have no force sensor, though there was an
experimental one flown in 1994 (STS-62). The
Canadarm2 and SPDM (Special Purpose Dextrous
Manipulator) have both been built with force sensors,
but they have yet to be operationally utilized. There
are certainly tasks one can imagine that require force
sensing at the tip - such as washing a window, but
these tasks are not yet required. The human-robotic
force feedback problem remains significant.
While we continue in our efforts to improve force
moment sensor capabilities, this paper deals with the
HMI (human machine interface) problem. A simple
software demonstration of the audio aspect of the
HMI approach is presented. The actual design and
prototype build work described is for the vestibulator
hardware/software portion of such an HMI system.
The vestibulator is a wireless electronic device for
use on human subjects.
2. Teleoperational Scenario
The requirements which we have based our design
on have been derived from the Canadian space
robots.
The original force moment sensor design
considerations began in the early 80's and were
based on the idea of upgrading the shuttle Canadarm
to include force sensing. The HMI problem which is
addressed in this paper has been targeted towards the
Canadarm2 and began its development in 1996 [1].
Two systems of presenting directional data were the
basis of ‘inspiration’ for the design to meet the HMI
requirement - audio and vestibular.
The need for robotic force sensing seems intuitively
obvious, when we consider the ultimate robot - the
human. Humans have many types of force and
pressure sensors (tactile, proprioceptive) distributed
throughout our ‘arm’ system. The difference between
human force sensing and teleoperational force and
moment sensing for a space environment is twofold.
A space sensor requires a wide dynamic range and
must be very stiff, due to it position near the tip of a
long arm (15.3 metre shuttle arm, and 17 metre
station arm). In addition to these two requirements,
the space environment adds serious complications
due to the thermal gradients which are experienced.
Space robots have some requirements that terrestrial
robots do not yet have, though we humans do. The
capture of what is known as a ‘free-flyer’ in space,
i.e. a spinning satellite is slightly comparable to a
human capturing a football or baseball. The space
robotic ‘free-flyer’ problem is probably a bit tougher,
because it requires reaction of very large torques
(~600 Nm). We do not yet expect terrestrial robots to
capture ‘free flyers’.
The HMI problem is in providing the force
information to the operator in useful real-time format.
The robotic space arm tasks are done fairly slowly,
but with a force data rate of 10 Hz, 6 degrees of
freedom force data is not easy to utilize. Our human
system of force sensing is very effective even and
complements our visual system. The Human
teleoperators (e.g. astronauts) might best be served
by force HMI that complements rather than distracts
from their visual processing of camera views.
There have been other non-visual HMI alternatives,
similar to our human force system, for example,
force feedback hand controllers. There are some
problems with these systems if one is using the robot
for long term tasks involving large force levels - the
human teleoperator tires out long before the robot.
The need for force sensing in teleoperation is less
driven by capturing free-flyer’s than it is by moving
large payloads around in somewhat constrained
quarters. The Canadarm is 50 feet long, with joint
locations pretty similar to our own arm (shoulder,
elbow and wrist). It is always possible to see where
the arm tip (end effector) is, because there is a camera
on the end effector (there is also an elbow camera).
But the payload itself may obscure obstacles it may
contact (e.g. shuttle cargo bay). The force sensor
would provide a means of identifying if an
interference has occurred - and a means of backing
away from the obstacle without damage or jam.
The Canadarm has functioned well, even without a
force sensor, since 1981. There was one experiment,
STS-62 [2], where a FMS was added to the
Canadarm. The Canadarm2, the ‘space station’
crane has been built with force sensing capability but
it has not been operationally utilized yet. The SPDM,
another robot soon to be delivered to International
Space Station also has force sensing capability. The
force sensing HMI’s for these two devices are
graphically presented to operators.
3. Audio Data Presentation
Human recognition of sound can be broken into three
groups: pitch (frequency), intensity (amplitude or
loudness) and timbre (complexities of component
frequencies). In addition to being able to differentiate
sounds on these bases, humans are remarkably good
at localizing sound in 3-D, with the exception of an
area above and behind our head, the cone of
confusion, Figure 2.
It is this ability to localize sound that sparked our
Figure 2
Human audio system - cone of confusion
inspiration into its usefulness as a multi-degree of
freedom HMI. Sound has been used as HMI by
others in a variety of applications. Flower and
associates have employed it as a means of
representing time series amplitude information in
place of visual, on the basis of musical presentation
[4],[5]. Sheridan et al [3] applies a combination of
auditory and tactile stimulation to the robotic force
feedback problem.
The robotic force feedback problem presents a
problem of wide dynamic range requirement (1.5
Nm- 3000 Nm). The pitch (frequency) element of
music presents a solution to the dynamic range
requirement and the timbre (such as distinguishable
between musical instruments) allows us to extend the
3-D nature of sound, to cover the 6-D nature required
for force. The low level amplitude forces are to be
represented as low frequency sound - increasing
frequency represents increasing force levels. The
distinction between the 3 linear force degrees of
freedom and the 3 moment degrees of freedom is met
with the use of timbre - i.e. two different musical
instruments.
Prior to writing the software to demonstrate the
potential of the method, some limited auditory
experimental evaluations were performed [1] and [8],
to assure ourselves of the suitability of frequency
sweep in combination with directionality as a means
of intuitive presentation.
The experimental evaluations performed were for a
set of 3 humans. We were attempting to establish the
‘intuitiveness’ of their ability to utilize sounds on the
basis of frequency, direction and amplitude. The
subjects received no training This was intended to
provide the most intuitive environment.
The subjects responses were tested for:
1) frequency sweep over 500-3000 Hz (ability to
detect the magnitude and sense of the sweep range)
2) azimuth/amplitude sensitivity (ability to sense
loudness and direction). Amplitude range from 0 to
30 dB.
3) azimuth/frequency sensitivity (ability to sense
frequency magnitude and direction). Frequency
range from 250-1750 Hz.
Table 1 presents the summarized measured error
values for each test.
Table 1
Human Errors for Sound Magnitude and
Localization
Frequency
Sweep
Azimuth &
Amplitude
Azimuth &
Frequency
15% 24% 17%
There is no clear indicator that our concept was either
viable or optimal. These results were only intended
as initial indicators.
The force to audio demonstration software which was
developed converts force to sound. The numerical
determination of which octave a force is represented
by is taken from the formula:
where n is an integer
nforce
=+
[log( )
log ]
21
representing octave number. A doubling of force is 1
Octave.
A 2 dimensional software demonstration for the use
of audio presentation to a simulated robotic peg-in-
hole task is shown in Figure 3.
The demo software assumed a linear force range of 0-
450 N and a moment range of 350 N-m.
The first view of Figure 3 is the start position of the
peg with respect to block with tight fitting hole. The
2nd view shows one linear, one rotation arrow
(downward to the left) which represent the magnitude
of the contact forces (linear and moment forces). The
3rd view is the peg sliding into the hole - the upward
arrow represents the commanded velocity of the peg.
There are remaining friction and normal forces as it
slides in the hole.
Figure 3
Peg-in-hole task ‘screen views’
After successful task completion, a window displays
the time taken to complete the task, and the maximum
forces generated during the task. The audio signals
which are generated are sufficient for an operator to
complete the task without visual input - but it takes a
bit of practice. The audio data is presented as two
musical instruments - one instrument for the linear
force direction , the other for the moment degrees of
freedom (demonstration software, stearsim, available
at www.gve.on.ca). The force magnitude data is
presented as high frequency for large forces and low
frequency for small forces.
4. Audio/Vestibular Presentation
The usefulness of audio presentation of 6 dof force
data has some limits when the force directions to be
detected are equivalent to somewhere above and
behind human head - i.e. cone of confusion, as shown
in Figure 2 . The cone represents geometries of
sound source with theoretically produce identical IID
(interaural intensity difference) and ITD (interaural
time difference).
The human vestibular system is another system
which is very directionally sensitive (it is the sensor
system that lets us know if we’re leaning or about to
tip over). It is possible to artificially stimulate the
human vestibular system with surface applied current
- i.e. galvanic stimulation, applied behind subjects
ears. The mapping of this type of stimulation to
humans is an active area of research for psychology
researchers (particularly in the visual/vestibular fields
such as virtual reality), Inglis et al, [7].
For our robotic HMI application, it offers the ability
to improve the directional integrity of audio data
presentation.
The audio systems needed for such an HMI
application are readily available, but programmable
vestibular stimulation is not. To meet this need, we
began development of a ‘vestibulator’. Its
development has been based on a wireless
enhancement, to give it a widespread research
application.
Prior to our wireless vestibulator development, we
participated in some simple vestibulator experiments.
A commercial current generator was used to apply
current levels between 0.1-1.0 mA to 6 human
subjects for stimulus durations between 0.5-3.0
seconds. Current levels above 0.3 mA produced clear
and consistent results in 5 of the 6 subjects. The
subjects swayed in the direction of the anode. These
results were duplications of work which has been
done by Inglis and many others.
5. Vestibulator Development
We have now developed 4 vestibulator prototypes.
They are all based on the same PIC (PIC16C63
controller) software, activated and controlled from a
PC. The PC host software is available in either
Windows or JAVA form.
The first development was a version which could
either receive PC generated data via a hardwire link
(RS-232 serial link), or by storing script files,
downloaded via the hardwire link, for subsequent
activation by human subject.
This ‘root’ system has 5 operating modes, to prevent
inappropriate commands from being executed at the
wrong time. Table 2 shows the functionality of each
mode.
Table 2
Operating Modes
Mode Electrode
Actions
Script
Upload
Script
Running
Tilt
Sensor
Idle No No No Yes
Direct Yes No No Yes
Program No Yes No Yes
Run No1No Yes No1
Fault No No No No
1) Control through the Host software is not allowed
but access is available using script commands
The second version is identical to the first, except that
a truly real time wireless connection has been
implemented using 2 r/f wireless serial connection
devices (Proxim Inc).
The third prototype was developed for potential use
on International Space Station (ISS). The target
market being researchers interested in isolating the 1-
G parameter for mapping studies between electrical
stimulation and human response. The ISS version,
drawings shown in Figure 4, was modified
mechanically by the development of a housing which
is stronger and somewhat astronaut friendly (e.g. no
sharp corners).
Figure 4
ISS Vestibulator
It is capable of surviving space shuttle launch
vibrations, 6.6 grms (Figure 5, vibrations spectra). It
was modified electrically by developing an I/R
wireless link (to reduce potential electrio magnetic
interference problems on ISS).
The infrared link was developed with an I/R LED as
part of a Fire-Stick II (Rentron) transmitter and a
standard infrared detector module as receiver.
Figure 5 - Vibration Test Spectrum
For the fourth version (based on terrestrial
developments) head mounted accelerometers have
been added, to allow researchers to monitor head
motion, based on applied stimulation. Our York
University researchers have chosen to utilize this
initially as a means of calibrating the signal being
applied to human subjects
A MATLAB interface capability was also added to
both generate subject stimulation data files and to
store input and output data. Table 3 presents the
specifications for the tilt-sensor version vestibulator.
Table 3
Vestibulator Specifications
Weight 600 grams
Power:
Consumption
Source
40-55 mA @9.5 V
8 AA batteries
Electrodes:
No. of Sources
Nominal Current
Measured Current
4
-2.56 to +2.54 mA
-2.57 to +2.59 mA
Serial Comm.
Data Rate 9600 baud
Scripts
EEPROM Memory
Frame Rate
2 Kilobytes
40 Hz.
Tilt Sensor
Gravity Range
Analog Noise
Quantization Nz.
Resolution
Analog BV
Acquisition Time
-2g to +2g
0.456 degrees pp1
0.471 degrees pp
0.655 degrees2
10 Hz.
2-4 ms (2 axes)3
1) probability 95.4%
2) root-sum-square of analog and quantization noise
3) Not including data transfer over serial link
The r/f wireless data link module is in addition to that
of Table 3. It has a weight of 340 grams (10 cm
x16.5 cm x 2 cm).
Figure 6 is a picture of the tilt sensor vestibulator,
headpiece with tilt sensor on left, main vestibulator
box on right.
Figure 6
Tilt Sensor Vestibulator
The electrodes and their attachment are something
left to the psychology researchers. For testing, we
have been using large diameter (1-2") stimulator pads
obtained from medical supply stores.
Figure 7 shows the PC host User Interface.
The MATLAB interface is available to generate
current vs time text files in a 4 column wide format.
It can then be used to generate and upload a script file
(time step increments of 25 ms).
The tilt sensor must be calibrated between uses. This
is done by levelling the system and sending calibrate
commands to it, while it is held motionless.
We have been working with Human Performance
researchers at York University during the multi-year
duration of this project. Their experimental work has
been supported by the Ontario Center of Excellence,
CRESTECH. The York Researchers have provided
us with input to version updates. More importantly
though, they perform the human testing which
demonstrates the influence of artificial galvanic
stimulation on human subjects in their research area
(visual/vestibular). Figure 8 is a graph of early
results on the type of influence they have measured.
Their measurement parameter is based on human
subjects ability to distinguish concave and convex
shapes in pictures.
Figure 7
Tilt Sensor User Interface
Figure 8
Effect of Vestibulator on Human Perception
6. Summary
The design path for this development has not
followed standard engineering practice, largely
because the problem we were solving was evolving
along with the engineering. A new product has been
developed and it remains to be seen if it will become
widely useful.
In all of our design, we considered the human model
and compared it to our requirements. Because we
were solving a problem which nature has already
solved, that was the obvious path to take. There are
two reasons for our diversion from what nature has
done. In one case, the robotic capabilities have not
progressed to where humans are, in the other case, the
robotic requirements are different than we humans
have evolved to meet. The lack of gravity is
something humans have had no reason to evolve to,
as yet, and have not come across any species that has.
Unaided humans do not mechanically move large
payloads, over long periods of time - a robot is
clearly better. But in general, the design assumption
was that if nature had already done the optimization
then engineering might save a lot of time if it gave
natures’s designs careful consideration.
7. References
[1] Draisey, Sherry. Feasibility Study of MDOFF.
Good Vibrations Engineering Report for Stear 10
Phase 1 Study. March
11, 1997.
[2] CDR Data Package, Remote Manipulator System
- Force Torque Sensor. NASA JPL, Pasadena CA,
1985.
[3] Massimino and Sheridan (1992). “ Using auditory
and tactile displays for force feedback.” SPIE Vol.
1833 Telemanipulator Technology.
[4] Turnage, Bonebright, Buhman and Flowers
(1996). “The effects of Task demands on the
equivalence of visual and auditory representations of
periodic numerical data”. Behavior Research
Methods, Instruments & Computers.
[5] Flower and Hauer (1995). “Musical vs Visual
Graphs: Cross-Modal Equivalence in Perception of
Time Series Data”. Human Factors 37(3), 553-569.
[6] Draisey. “Feasibility Study of MDOFF, Volume
1". Good Vibrations Engineering Report for Stear 10
Project. 1997.
[7] Inglis, Shupert, Hvlavacka & Horak. (1995).
“Effect of Galvanic Vestibular Stimulation on Human
Postural Responses during Support Surface
Translations”. Journal of Neurophysiology, Vol 73,
No. 2.
[8] Draisey. “Audio Test Report, MDOFF, Phase 1".
Good Vibrations Engineering Report for Stear 10
Project. 1997.
