ArticlePDF Available

Abstract and Figures

Stroke commonly leads to partial or complete paralysis of one side of the body and there is limited availability of therapists to provide rehabilitation. It is a priority therefore to identify the most effective rehabilitation strategies and/or pharmacotherapies. Motor learning, the essential process underpinning rehabilitation, can be assessed more quickly and robustly than outcomes from rehabilitation. In this paper we describe a proof of concept system utilising a commodity input device to play a bespoke video game to measure the critical components of motor learning. We demonstrate that we can detect how simple changes in therapist instruction significantly change motor performance and learning. Although video games have been shown to aid in rehabilitation, this is the first time video games have been used to derive early response markers, based on the measurement of performance and motor learning, for use in the evaluation of the efficacy of a rehabilitation strategy.
Content may be subject to copyright.
Early Response Markers from Video Games
for Rehabilitation Strategies
Graziadio S1, Davison R2, Shalabi K1, Sahota K. M. A1, Ushaw G2, Morgan G2, Eyre J. A.1
1Institute of Neuroscience, 2School of Computing Science,
Newcastle University, UK
{sara.graziadio, richard.davison4, k.m.a.shalabi, gary.ushaw, graham.morgan,
janet.eyre}@newcastle.ac.uk
3rd E-mail
ABSTRACT
Stroke commonly leads to partial or complete paralysis of one
side of the body and there is limited availability of therapists to
provide rehabilitation. It is a priority therefore to identify the most
effective rehabilitation strategies and/or pharmacotherapies.
Motor learning, the essential process underpinning rehabilitation,
can be assessed more quickly and robustly than outcomes from
rehabilitation. In this paper we describe a proof of concept system
utilising two commodity input devices to play a bespoke video
game to measure the critical components of motor learning. We
demonstrate that we can detect how simple changes in therapist
instruction significantly change motor performance and learning.
Although video games have been shown to aid in rehabilitation,
this is the first time video games have been used to derive early
response markers, based on the measurement of performance and
motor learning, for use in the evaluation of the efficacy of a
rehabilitation strategy.
Categories and Subject Descriptors
J.3 [LIFE AND MEDICAL SCIENCES]: Health
General Terms
Measurement, Design, Experimentation, Human Factors
Keywords
Rehabilitation, video games, motor learning
1. INTRODUCTION
Stroke is a major global problem; the current prevalence of 60
million stroke survivors is predicted to rise to 77 million by 2030.
Hemiparesis, a detrimental consequence that many stroke
survivors face is the partial or complete paralysis of one side of
the body that occurs due to brain injury. It is remarkably prevalent
occurring acutely in 80% [1, 2]. Unfortunately, upper limb
recovery is unacceptably poor with persisting impairments in 50-
70% of stroke survivors [2, 3]. Although it has been established
that intense rehabilitation therapy increases upper limb recovery,
resource limitation is the main barrier to implementation of this
evidence-base. Given the increasing prevalence of stroke
survivors, it is a priority to identify the most effective
rehabilitation strategies and/or pharmacotherapies that augment
neuroplasticity in order to maximise a patient’s response to the
available therapy time.
Motor learning is the essential process underpinning recovery
after hemiplegia, either through relearning to use the paretic arm
and hand and/or learning to compensate with the lesser-affected
side. Furthermore, motor learning in normal subjects and
functional neuroplasticity leading to post-stroke motor recovery
have been shown to share the same underlying molecular and
genetic substrates and brain networks [4]. Since motor learning
can be assessed more quickly and more robustly than behavioural
outcomes from rehabilitation of stroke survivors, assessment of
performance and motor learning can provide an ideal marker of
the biological system underpinning rehabilitation.
In this paper we describe a proof of concept system that utilises a
bespoke video game to generate high spatial-temporal resolution
data from players. Such data is key to measuring the critical
components of motor learning that in turn provide early response
markers. We demonstrate how our system is sufficiently sensitive
to detect how simple changes in therapist instruction significantly
influence the motor performance exhibited within in-game player
performance.
The literature presents numerous works indicating how serious
games may aid in rehabilitation. However, the main focus of this
paper is not to provide a rehabilitative game (where encouraging
while recognising broad movement is sufficient), but to show, for
the first time, an ability to measure early response markers to a
clinical standard that are key to informing rehabilitative strategies.
Such markers would otherwise be derived through time
consuming, and costly, therapist observation.
The capacity to automate evaluation brought about by the
utilisation of the approach described in this paper brings forward
additional opportunities in allowing targeted screening of
candidate drugs for repurposing into rehabilitation, prior to
initiating phase 2 or phase 3 trials, or patient stratification for
clinical trials, based on level of performance and indices of motor
learning.
2. SYSTEM OVERVIEW
Nearly all manipulation activities of daily living require the ability
to flexibly perform multiple steps to achieve a unitary task. Thus,
regaining the ability to perform efficiently linked action phases in
manipulation tasks is highly important for rehabilitation. In this
paper we illustrate the use of the proposed system to assess
whether the instructions given by a therapist to subjects learning a
task requiring two action phases, either describing the task as
having a single objective or by breaking the task down into its two
sequential action phases, significantly affects performance and
learning.
2.1 Bespoke video games
We have developed a series of bespoke video games that require
learning of sequentially linked action phases and capture key
features of natural manipulation tasks, namely spatial and
temporal control and the requirement that each phase is completed
before the next phase can be executed. Furthermore, each action
phase utilises characteristics of movements performed in real life
(controlled application and adaptation of forces, visuomotor
integration and adaptation, feed forward planning and feedback
correction).
The game (sample screenshots shown in figure 1) used in this
evaluation involves moving a spacecraft (avatar) around a screen
to avoid and/or destroy meteorites (targets). A patient controls the
movement of the avatar via isometric forces applied to a joystick.
In essence, the patient must move their avatar to the same location
as a target when it appears. The avoid/destroy are gameplay
elements acted out onscreen to increase interest for the patient.
Two sequentially linked action phases (transfer phase and lock
and track phase) are embedded in the game. Players initiate a trial
by relaxing their hand to bring the avatar to a central home zone.
Targets are then randomly presented at one of three locations,
namely to the right, to the left, or centrally above the home
position. The first phase comprises moving the avatar towards the
target (transfer phase), once having achieved the target the player
must then hold the avatar within the trajectory of the target for 1s
(lock and track phase). Feedback of performance is provided as a
score that builds up on the screen during each trial of 12 target
presentations. The final score is then presented at the end of each
trial.
2.2 Monitoring Play
In our experiments we wanted to create a cost efficient solution
using only commodity hardware. Our initial choice (a gaming
joystick for around $500) had to be adapted (placed in a box) as
patients could not hold the joystick in a traditional manner.
Patients could only lie their hands face down and apply pressure.
Although the joystick in a box solution was adequate, it did result
in a bulky (50cm x 60cm) device not easily moved. In addition,
the joystick was heavy (too heavy for a patient to lift) weighing
in at around 2kg. This may be an issue in a patients’ home if we
were considering future scenarios where our approach could be
tailored for home use as well as in clinical settings. Therefore, a
more portable alternative was considered.
Figure 2Joystick representation of force and direction
Figure 1 Sample screenshots from game
An alternative to the bulky joystick was to use four flat pressure
pads (no larger than a small coin each) that could easily be carried
around and weighed very little. Although approximately three
times the price of the joystick, they afforded a degree
of portability and adaptability the joystick lacked.
We do present a comprehensive description of both
devices’ capabilities. However, the purpose of the paper is not to
compare their merits in accuracy and performance, but to solely
determine if our portable and not so portable solutions can provide
the early response markers we require to inform a rehabilitation
strategy.
2.3 The joystick
The joystick chosen was a Saitek Pro Flight X-65F Combat
Control System (PC) Joystick. This joystick is considered a “high-
end” gaming device costing approximately $500. However, unlike
the typical joystick most gamers are familiar with, the Saitek Pro
Flight does not measure the degree of movement of the joystick
itself (typically found in controllers for consoles such as PS3, and
Xbox). Instead, the pressure and the direction of such pressure are
measured (the joystick is actually unmoveable). For the purposes
of our game we are concerned with the joystick measurements
associated to kilogram-force (kgf), and the degree to which this
force is measured in the x axis (left or right on joystick) and y axis
(up and down on joystick).
Understanding how the joystick works is important for realising
the actions patients are required to undertake to successfully
complete the game. The joystick informs the game of both force
and position via a kgf value in both the x and y coordinate. This is
then mapped into 2D space {x, y} floating-point coordinates
between 1 and -1 to represent points that exhibit both direction
and magnitude (which is the force). Assume figure 2.a indicates
the joystick at rest (no force applied) and the outer circle
represents the most force the joystick could recognise in all x and
y directions. Figure 2.b indicates force applied in the negative x
(left) and positive y (up) directions. Figure 2.c indicates a more
severe force than that applied in 2.b (due to its proximate to the
outer circle) in the positive x (right) and negative y (down)
directions. Therefore, a steady force maintained in a single,
unwavering, direction will provide an unmoving cursor at some
point, say {Px, Py}, with the distance from the origin {0, 0}
indicating force (relative to the maximum force achievable).
To ultimately accommodate patients with movement difficulties
the joystick has been adapted so forces orientated to the right, to
the left, up or down can be generated by movements of the
supported, out-stretched hand and/or by the arm and shoulder. The
patient does not grip the joystick, but rests their hand on a custom
made support on top of the joystick. This allows patients with
severe disability to apply pressure and participate in the game.
The pressure generates an {x, y} coordinate indicating force and
direction (as mentioned earlier). A translation is applied to these
coordinates to map them to “screen space”.
Screen space is not circular (reachability of {x, y} by joystick),
and represents the resolution of the game graphics (1280 pixels
wide by 800 pixels in height). Therefore, a basic scalar translation
is applied to enlarge the joystick {x, y} coordinate beyond that of
the screen area. As the derived coordinates of x and y generated
by the joystick is a floating-point number between -1 to 1 in both
x and y, this can be achieved by multiplying x and y by 1280 and
casting to integer values (for pixel alignment). If full force could
be achieved this could result in y coordinates occurring “off-
screen” (i.e., beyond 800 and -800). However, this does not occur
as input from the joystick relating to magnitude/force (distance
from origin) is capped to retain the avatar in-screen.
2.4 Joystick accuracy
We are primarily concerned with accuracy achieved by the
joystick in the presence of wavering, usually, uneven weighting.
This is because such weighting one would be expected from
human provided force. Therefore, we take the technical
specification as a guide and created our own experiments with
weights to judge accuracy of the joystick. The weights were
applied and measurements taken.
The Saitek X65 control stick is unique among flight sticks in that
it has zero stick movement; it is entirely solid. Movement is
instead determined by the amount of force imparted upon it by the
user.
The maximum amount of force detected on the x and y axis are
selectable, in a range from 0.2 kilograms of force, to 10 kilograms
of force. Once setup, the device outputs a 12-bit integer value,
with 0 representing 'full negative' on the axis, and 4095
representing 'full positive' on the axis. The value can be easily be
shifted via subtraction of 2047 to allow a value to 0 to represent
no movement, with extents of -2047 to 2047. As this output value
is of a fixed precision, it therefore stands to reason that the
precision of the output value in kgf is proportional to the selected
maximum at 0.2 kgf, there is a minimum possible detection of
change in force has a precision of 0.00009765 kgf (0.2 / 2048),
while at 10.0kgf, the minimum becomes 0.00488 kgf (10.0 /
2048). Therefore, it is important to calibrate the joystick sensing
range to approximately that of the user, so as to maximise the
detection precision.
While stable under no movement, the values reported from the
device were found to have a small
The device was found to have a shifting offset, which necessitated
tracking idle movement and correction.
A 30 second sample of the device recorded at rest, with axis set to
a force rating of 10kgf revealed a mean x axis position of -22.54
(stdev 1.24), and y axis position of -39.48 (stdev 1.45). It is
Weight Mean Stdev
Adjusted
Mean
Diff (Grams)
Inaccuracy %
0.2659937022
0.0024409681
-
-
-
1.4307216637
0.0122985759
1.1647279616
85.272038432
6.8217630746
2.6685066999
0.0093048713
2.4025129978
97.49
3.8994800899
4.921655171
0.0823442533
4.6556614688
344.34
6.8867706239
7.0868227861
0.0726138541
6.820829084
679.17
9.055612214
Table 1 - Accuracy of joystick using standard weighting
interesting to note that this offset was due to the mechanism itself,
and was proportional to the force setting selected therefore the -
22.54 offset on the axis equated to the mechanism being off by
approximately 0.1 kgf; by leaving the joystick alone and changing
its force setting to have a maximum of 0.2kgf movement, this
offset raised to approximately -1000, resulting in a potential 50%
inaccuracy at the lowest setting.
The accuracy of the KGF values reported by the device were
tested, by mounting the device such that it faced up, and by
attaching weights to it, around the trigger.
The device was recorded at idle in this state to create an offset to
account for the weight of the attaching cable, and the effect of
gravity on the stick itself. The results of this experiment are
described in table 1.
The difference between the hung weight and reported weight
grows non-linearly, suggesting a limitation in the mechanism used
for detecting force imparted upon the device.
As the device is handled and re-centered, the offset described
earlier shifts slightly. An example of this is shown in the graph in
figure 3.
The graph represents a recording where the device was pushed
hard in one direction for 2 seconds, left idle for 2 seconds, and
then pushed hard once more for 2 seconds in the alternate
direction. For clarity, the axis range has been limited, from 2048
to 100. As can be seen from the two pushes to the maximum
positive value, the idle value on release cannot be predicted, with
the idle value at position A being negative, while B shows a
positive idle offset, despite a similar push in the positive axis.
To accommodate the need to remove this offset from the input
recordings, a calibration stage was added to the game. In it, the
user is informed to take their hand off the joystick entirely, for a
timed period. Over this time, the mean position is recorded on
each axis, which can then be subtracted from the incoming device
numbers to reduce inaccuracy. This calibration period takes place
after every target is completed, to minimize the effect of the
shifting offset as the joystick is moved.
To compensate for players not paying attention to the instructions,
not lifting their hand off correctly, or simply taking too long to
perform the required action, the derivatives of the joystick values
are recorded during this calibration period only samples with a
low acceleration and jerk are considered for the mean calculation,
within a specified Euclidean distance from the axis origin.
2.5 The Pads
2.6 Pad accuracy
As with the joystick we used weights to measure the accuracy of
the pads. We also correlated information relating to the technical
ability of the pads.
The sensor setup comprises 4 Loadstar iLoad Mini devices,
attached to a Loadstar DQ-4000 device, used to interface between
the sensors and the host computer.
The iLoad Mini sensors are approximately 1.25 inches in
diameter, with a cylindrical 'load button' on top, on which the load
is rested to calculate force. The sensors output a square wave with
a frequency proportional to the applied load to the DQ-4000
device, which converts the signal to millipounds.
The DQ4000 unit communicates to the host computer via a USB
virtual serial connection. This serial link operates at a baud rate of
230400, with data characteristics of 8 signal bits to 1 stop bits.
The millipound rating of the applied load is sent in the form of
ASCII characters, therefore a millipound force of 123 would take
24 data bits and 3 stop bits to send to the host device.
The full format of one data frame is to always send 4 values no
matter how many sensors are attached, separated by tab keycodes,
and terminated by an endline keycode. The DQ-4000 is
communicated with via simple 4byte commands, including one
which will begin a constant stream of data to be sent to the host
pc, requiring no further host commands or polling.
Due to this, the actual update rate of the unit is variable higher
millipounds will take up more characters, therefore fewer
characters can be sent per second. The average update rate is
approximately 140hz under normal conditions.
Upon startup, the device calculates the tare weight of each of its
attached sensors that is, it will calculate the offset required to
return 0 when nothing is rested on the sensors.
A recording of 30 seconds of this idle displays good stability, with
relatively little noise (Mean value -2.46 millipounds, stdev 3.03
millipounds). It is worth noting that the sensors will report
negative values, so no noise is 'lost' by the tare process. A
minimum difference between samples of 1 was noted, suggesting
that the device is capable of detecting differences in load weight
of 1 millipound.
The sensors display a small amount of creeping bias as they are
used. The graph below depicts a recording of the device, in which
the senor was alternated between being left idle for 10 seconds,
and loaded with 5.0bs of force for 5 seconds. Note that at each
point at which the weight is removed from the device, the
recorded reading dips noticeably lower than the samples that
follow it. Note also that the sensor reading begins a slow upward
crawl after the first load. It takes approximately 30 seconds for
this to ease of and return to an idle state.
The graph has been clipped to a range of -50 to 100, but note that
the scale is in millipounds, so each loading was to a value of
5000, and therefore the device tare weight only loses
approximately 0.05lbs of accuracy.
Experimentation was performed to determine the accuracy of
readings when loaded with a number of different weights. The
results of this are collated below:
Due to the small, domed surface area, correct balance of the
weights was difficult, and so these values should be seen as a base
estimate. It appears that the devices are capable of operating
outside of their specifications with a similar degree of accuracy.
2.7 Capturing data
Assuming that each hand will deviate in ability due to stroke the
game calibrates separately for each hand. This allows further
refinement of the coordinate system. The side more severely
affected will not be able to reach the same degree of coordinates
as the less affected side. Therefore, calibration takes this into
account and applies an appropriate ratio to allow the same degree
of “on-screen” movement for both hands. For example, if the left
side can only apply a maximum of 5 kgf whereas the right side
can apply a maximum of 20 kgf, then the left phase of gameplay
will apply a 4:1 ratio multiplier to attain the same degree of
movement on the screen.
Theoretically, when a target (meteorite) appears it is possible to
apply the correct force and direction to the joystick so as to make
the avatar appear immediately over the target. This is because we
are not “moving” in the direction indicated by the joystick, rather
we are placing the avatar on screen as directed by the {x, y}
coordinates produced by the joystick. However, as humans
naturally take time to react and build up force the avatar appears
to move across the screen.
To achieve an appropriate fidelity of sampling to ensure no
significant movement data is missed we sample the joystick at 500
times a second and update the game loop at the same rate (500
times a second). A commercial video game typically runs at 60
frames a second. However, our desire for a much higher rate is
driven by the need to rule out the loss of outlier measurements
(sudden increase or decreases in pressure) that may occur. This
also has the result of allowing a high fidelity of movement data to
be considered within the gameplay itself, allowing the data that is
used clinically to drive the game. Although the monitor/TV
cannot show 500 full frames a second, we can still run the game at
such a rate for accuracy of tracking the joystick and refresh the
screen as and when required.
Accuracy tracking per-target is calculated as the mean distance
the joystick {x, y} is from the actual target coordinates over the
time the target is present. This measurement is returned to the
coordinate system between (-1, 1) to trivialise comparisons across
all targets. Time (as presented in the graphs) is measured in
seconds.
2.8 Analysis system
Prior to starting the game each player undertakes a calibration
procedure. The maximum pressure that the subject is able to
generate by rotating their hand to the right or left and by palmer
flexion is recorded for each hand separately. The pressure
required in the task to reach the target at each position is then
automatically set to be 10% of maximum respective pressure, to
avoid fatigue during game play.
1) Transfer Time - the time between the appearance of the
target on the screen and the avatar reaching the target. This
index reflects predominanty feed-forward generation of
movement with little opportunity to correct errors based on
feedback.
2) Distance - the mean distance between the target and the
avatar during the lock and track phase. This index reflects
predominantly feedback mechanisms and error correction.
For both indexes lower values are associated with higher
performances.
3. METHOD
The ethical committee of Newcastle University approved the
study and written informed consent was obtained from
participants. All participants were naïve to the experimental setup
and objectives of the study.
3.1 Subjects
We compared two groups of 12 right-handed, young adult
subjects (Group 1: mean age, 27 years, range 20-35 years; Group
2: mean age, 27 years, range 20-36 years). The video game and
the controllers used and the environment in which the game was
played were the same for both groups.
Group 1 (Single objective instruction group) were asked to play
the video game with the single instruction to follow the target as
accurately as possible; the feedback score reflected the accuracy
of tracking the target.
Group 2 (Two step instruction group) were asked to play the
video game with the instructions to move the avatar to the target
and then to follow the target as accurately as possible. The
feedback score for this group reflected both the time taken to
transfer the avatar to the target and the accuracy of tracking
thereafter.
3.2 Protocol
Initially the subjects play three trials within the game with each
hand (non-dominant hand first) to assess their pre-training
performance levels. This is followed by a session of training for
the dominant hand, when the player undertakes 15 further trials.
After completing training, the player undertakes a final 3 trails
with their dominant (trained) hand to determine their post training
performance levels. This is followed by 3 trials with their non-
dominant hand (un-trained hand) to assess inter-limb transfer of
skill from their trained to their untrained hand (a measure of
generalisation of learning).
Figure 3 - Graphs derived from game data; learning curves
for transfer time (s) and distance. For analysis the data for
every three sequential trials are grouped. The blue symbols
indicate Group 1 (perceived single objective) and the green
symbols Group 2 (perceived 2 step task). The squares indicate
the trained right hand (TH) and the triangles the non-trained
left hand (nTH). The error bars are SEMs. For both indexes
lower values are associated with higher performances.
3.3 Data analysis
Pre and Post training performance for each hand was assessed as
the mean Transfer Time and the mean Distance for the first and
last 3 trials respectively. Motor learning was assessed as the
difference between pre-training performance and post-training
performance for the trained hand. Inter-limb transfer of training
from the trained hand to the non-trained hand was assessed as the
difference pre and post-training performances for the untrained
hand.
3.4 Statistical analyses
The data was normally distributed. Significance was set at p<0.05,
with Bonferroni correction. A General Linear Model Repeated
Measures Analysis of Variance (ANOVA) was used with
Greenhouse-Geisser correction if required (SPSS 15, SPSS Inc,
Chicago, Illinois, USA). Each hand (trained hand, non-trained
hand) was analysed separately. Time (Pre, Post Training,) was the
within-subject factor; Group (Single, Double instruction) was the
between-subject factor.
4. RESULTS
4.1 Trained hand (TH)
There was a main effect of Group for Distance (p=0.012), with a
significantly better performance in Group 1 (perceived single
objective) than Group 2 (perceived two step task). There was a
main effect for Time for both indexes (p<0.001 for Transfer Time
and Distance), indicating motor learning for both groups. Whilst
there was no main effect of Group for Transfer Time (p=0.256),
there was a significant Group*Time interaction (p=0.015) with a
trend towards a better performance for Group 1 (perceived single
objective) prior to training (p=0.089) but no difference between
groups after training (p=0.831).
4.2 Non-trained hand (nTH)
There was a main effect of Group for both indexes (Transfer
Time, p=0.034; Distance, p=0.001) with significantly better
performance for Group I (perceived single objective) both pre and
post-training. There was also a main effect of Time in both
indexes (Transfer Time, p<0.001; Distance, p<0.001) indicating
inter-limb transfer of dominant hand training occurred in both
groups for both components of the task. A significant
Group*Time interaction was observed in the Transfer Time index
(p=0.040) with greater transfer in Group 2 compared to Group 1.
5. DISCUSSION
In almost any training situation where action goals are to be
learnt, including rehabilitation after stroke, instructions are given.
In the present study we presented to both groups exactly the same
sequences of stimuli and in response the participants performed
the same two sequentially linked action phases. When the
instructions and feedback emphasised the two subcomponents of
the task rather than focusing on the single action goal (tracking
the target), the performance of both action phases was
significantly degraded. These findings add to a growing literature
that the nature of the instructions given has a decisive influence
on performance and/or learning of the action goal. For example,
there is converging evidence that an external focus for instructions
(i.e., a focus on the movement effect) is more effective than an
internal focus (i.e., focus on the muscles activated to achieve
components of the movements themselves) for both performance
and learning (for review see [5]). In this study the two groups
were given an external focus since both instructions focused on
the desired movement effect.
The task studied in the present study involved two sequentially
linked action phases. Although most manual tasks involved in
activities of daily living comprise sequentially linked action
phases, nearly all studies of manual control and learning concern
single actions, such as simple reaction times or moving the hand
between two positions. Thus, our understanding of how action
phases are linked to perform the overall action goal, and how such
linking affects learning, is limited.
Bernstein [6] first argued that action goals correspond to a pattern
to be executed in external space rather than a sequence of
specified muscle patterns. An easily verified demonstration that
action goals are an abstract pattern is that one’s written signature
has the same unique pattern whether generated by shoulder and
arm muscles to write on a blackboard or by forearm and finger
muscles to write on paper [7].
Klapp and Jagacinski [8] summarised recent research that
supports action goals being represented as an abstract code that
does not incorporate details of the sub-components required, but
rather involves a single motor gestalt or chunk that is processed
holistically. Additional findings indicate that the organization of
action goals into chunks can be changed by instructions. For
example, when reaction time was measured prior to the
articulation of pseudowords [9] instructions that encouraged
separate articulation of each of the syllables resulted in reaction
times that increased as a function of the number of syllables.
However, when the instructions favoured combining the syllables
to form a single word, the reaction time did not increase as a
function of the number of syllables implying that combining the
syllables created a single motor gestalt so that the number of
chunks is one, regardless of the number of syllables.
The results of our study mirrors these findings but in
relation to a manipulative task; the transfer time was prolonged
when the instructions favoured viewing the action goal as two
separate tasks rather than as a single action goal. Furthermore, not
only was the transfer timing prolonged but the accuracy of the
tracking task was degraded, when the action goal was viewed as
two separate tasks. These results support the concept that motor
action goals might be represented as a single motor gestalt and
indicate that task boundaries defining a motor chunk or single
motor gestalt are not inherent to the action goal, but are ultimately
determined by participants’ subjective representations of the task,
shaped by the instructions given.
6. CONLCUSION
Video games have been shown to encourage stroke rehabilitation
due to the simple fact that they require concentration of thought
and some form of physical exertion (e.g., [10] [11]). However,
there has not been a clinical view to deriving the required early
markers to inform intervention, as described here, directly from
the gaming hardware itself (in our case the joystick). Our most
recent work [12] demonstrates an ability to determine change
using video game devices benchmarked against therapist
intervention. However, this paper goes further and investigates the
validity of using off-the-shelf hardware coupled with bespoke
video games to gain early response markers in the case of stroke
rehabilitation.
This study provides proof of concept that video games and
automated analysis systems can provide the capability to rapidly
evaluate even subtle aspects of rehabilitation strategies, such as
the content of instructions. We propose to develop a library of
action phases, which can be easily combined and incorporated
into bespoke video games together with automated data extraction
and analysis to provide a flexible system for early response
markers for evaluation of rehabilitation efficacy.
7. REFERENCES
[1] Intercollegiate Working Party, National clinical guideline
for stroke, 4th edition ed. London: Royal College of
Physicians, 2012.
[2] Centers for Disease Control and Prevention, "Prevalence of
stroke: United States, 2006-2010," MMWR Morb Mortal
Wkly Rep, vol. 61, pp. 379-382, 2012.
[3] M. Kelly-Hayes, A. Beiser, C. Kase, A. Scaramucci, R.
D’Agostino, and P. Wolf, "The influence of gender and age
on disability following ischemic stroke: the Framingham
study," J Stroke Cerebrovasc Dis, vol. 12, pp. 119-126,
2003.
[4] J. Krakauer, "Motor learning: Its relevance to stroke
recovery and neurorehabilitation," Curr Opin Neurol, vol.
19, pp. 84-90, 2006.
[5] G. Wulf, "Attentional focus and motor learning: A review of
10 years of research " Bewegung und training, vol. 1, pp. 4-
14, 2007.
[6] N. Bernstein, Coordination and regulation of movement.
New York, NY: Pergamon Press, 1967.
[7] S. Keele, Motor control vol. 2. New York, NY: Wiley,
1986.
[8] S. Klapp and R. Jagacinski, "Gestalt principles in the control
of motor action," Psychol Bull., vol. 137, pp. 443-62, 2011.
[9] S. Klapp, "Reaction time analysis of two types of motor
preparation for speech articulation: Action as a sequence of
chunks," J Mot Behav, vol. 35, pp. 135-150, 2003.
[10] Lange, B., S. Flynn, and A. Rizzo. "Initial usability
assessment of off-the-shelf video game consoles for clinical
game-based motor rehabilitation." Physical Therapy
Reviews 14.5 (2009): 355-363.
[11] Burke, James William, et al. "Optimising engagement for
stroke rehabilitation using serious games." The Visual
Computer 25.12 (2009): 1085-1099.
[12] Shi, J. Q., et al. "Evaluating Functional Ability of Upper
Limbs after Stroke Using Video Game Data." Brain and
Health Informatics. Springer International Publishing, 2013.
181-192.
... Motor learning can be assessed more quickly and robustly than outcomes from rehabilitation. Davison et al. (2014) proposed to utilize a commodity input device to play a bespoke video game to measure the critical components of motor learning. To detect how simple changes in therapist instruction change motor performance and learning, experiment has been conducted by either giving a single objective (single instruction) or by breaking the task down into its two sequential action phases (double instruction). ...
Article
Gaussian process regression (GPR) model is well-known to be susceptible to outliers. Robust process regression models based on t-process or other heavy-tailed processes have been developed to address the problem. However, due to the nature of the current definition for heavy-tailed processes, the unknown process regression function and the random errors are always defined jointly and thus dependently. This definition, mainly owing to the dependence assumption involved, is not justified in many practical problems and thus limits the application of those robust approaches. It also results in a limitation of the theory of robust analysis. In this paper, we propose a new robust process regression model enabling independent random errors. An efficient estimation procedure is developed. Statistical properties, such as unbiasness and information consistency, are provided. Numerical studies show that the proposed method is robust against outliers and has a better performance in prediction compared with the existing models. We illustrate that the estimated random-effects are useful in detecting outlying curves.
... While much medical research is based on quantitative biometric data that can be measured or observed in the patient, it is often the case that the available methods for collecting data are more qualitative. In such cases, medical markers must be identified within the soft data that is accrued [4], [2]. The analysis of soft data requires techniques for understanding qualitative results, which are by definition open to some interpretation [6]. ...
Conference Paper
Full-text available
Mass provision of healthcare through a digital medium can be greatly enhanced by the use of serious games. The accessibility and engagement provided by a serious game to the subject can significantly increase participation. The commercial games industry employs numerous techniques to analyse soft data collected from early users of an application to evolve the application itself and improve the experience of playing it. A game for mass stereoacuity testing of young children is used as a case study in this paper, to illustrate how soft feedback can be used to improve the effectiveness of a clinical trial. The key to the approach is identified as rapid incremental evolution of the application and trial protocol in a manner which increases the amount and usefulness of soft data collected, and reacts to issues identified in the soft data in a timely fashion. It is hoped that the approach can be adopted for a wide range of digital applications for mass health provision.
... Firstly a games for rehabilitation of stroke which asks the patient to replicate upper arm exercise shown by avatars on screen, using motion sensing equipment to measure movement [4]. Figure 1 shows a level from this game. ...
Conference Paper
Full-text available
We describe how developers with extensive experience in the entertainment video game industry approach the creation of rehabilitative games. A key element a game for health shares with a game from the entertainment industry is the ability to engage over long periods of time: the longer a commercial game engages the more popular it becomes, improving its revenue; the longer a game for health is played the more extensive an intervention, improving patient outcomes. As such, having worked on a number of successful titles for over twenty years we explored transferring player engagement techniques in games for entertainment over to games for health. This we achieved in a series of rehabilitative games for different patient conditions and demographics. We present our findings as a number of design and implementation practises that bring a heightened sense of engagement and replay value to games for patients that do not distract from the main health benefit.
Article
The Gaussian process regression (GPR) model is well-known to be susceptible to outliers. Robust process regression models based on t-process or other heavy-tailed processes have been developed to address the problem. However, due to the current definitions of heavy-tailed processes, the unknown process regression function and the random errors are always defined jointly. This definition, mainly owing to mix-up of the regression function modeling and the distribution of the random errors, is not justified in many practical problems and thus limits the application of those robust approaches. It also results in a limitation of the statistical properties and robust analysis. A general robust process regression model is proposed by separating the nonparametric regression model from the distribution assumption of the random error. An efficient estimation procedure is developed. It shows that the estimated random-effects are useful in detecting outlying curves. Statistical properties, such as unbiasedness and information consistency, are provided. Numerical studies show that the proposed method is robust against outliers and outlying curves, and has a better performance in prediction compared with the existing models.
Chapter
Full-text available
Motion detection devices are readily available as part of domestic gaming system. Many rehabilitative regimes require a patient to repeat specific movements in order to exercise the affected limbs. Incorporating these movements into games is of interest as the exercises can take place in the domestic environment without continual medial oversight, and the use of a gaming context encourages engagement with the exercise regime, leading to increased dosage. This chapter discusses the motion capture devices available commercially, presenting a benchmark for comparison, and provides a case study of a game for rehabilitation of the upper limb after stroke. The work is resented in a manner which is intended to be applicable to the development of any game rehabilitation.
Chapter
In today's medical environments, imaging technology is extremely significant to provide information for accurate diagnosis. An increasing amount of graphical information from high resolution 3D scanners is being used for diagnoses. Improved medical data quality become one of the major aims of researchers. This leads to the development of various medical modalities supported by cameras that can provide videos for the human body internal for surgical purposes and more information for accurate diagnosis. The current chapter studied concept of the video processing, and its application in the medical domain. Based on the highlighted literatures, it is convinced that video processing and real time frame will have outstanding value in the clinical environments.
Chapter
Motion detection devices are readily available as part of domestic gaming system. Many rehabilitative regimes require a patient to repeat specific movements in order to exercise the affected limbs. Incorporating these movements into games is of interest as the exercises can take place in the domestic environment without continual medial oversight, and the use of a gaming context encourages engagement with the exercise regime, leading to increased dosage. This chapter discusses the motion capture devices available commercially, presenting a benchmark for comparison, and provides a case study of a game for rehabilitation of the upper limb after stroke. The work is resented in a manner which is intended to be applicable to the development of any game rehabilitation.
Chapter
In today’s medical environments, imaging technology is extremely significant to provide information for accurate diagnosis. An increasing amount of graphical information from high resolution 3D scanners is being used for diagnoses. Improved medical data quality become one of the major aims of researchers. This leads to the development of various medical modalities supported by cameras that can provide videos for the human body internal for surgical purposes and more information for accurate diagnosis. The current chapter studied concept of the video processing, and its application in the medical domain. Based on the highlighted literatures, it is convinced that video processing and real time frame will have outstanding value in the clinical environments.
Article
Full-text available
To detect differences between the mean curves of two samples in longitudinal study or functional data analysis, we usually need to partition the temporal or spatial domain into several pre-determined sub-areas. In this paper we apply the idea of large-scale multiple testing to find the significant sub-areas automatically in a general functional data analysis framework. A nonparametric Gaussian process regression model is introduced for two-sided multiple tests. We derive an optimal test which controls directional false discovery rates and propose a procedure by approximating it on a continuum. The proposed procedure controls directional false discovery rates at any specified level asymptotically. In addition, it is computationally inexpensive and able to accommodate different time points for observations across the samples. Simulation studies are presented to demonstrate its finite sample performance. We also apply it to an executive function research in children with Hemiplegic Cerebral Palsy and extend it to the equivalence tests.
Article
Full-text available
Aims/background: Off-the-shelf games for consoles such as the Nintendo Wii, Nintendo WiiFit and Sony PlayStation 2 EyeToy have been developed and tested for the purpose of entertainment. Many clinics are adopting the use of these off-the-shelf devices for exercise, social interaction and rehabilitation because they are affordable, accessible and can be used within the clinic and home. Our group carried out initial usability evaluations for these off-the-shelf games and a prototype game (using an off-the-shelf device) specifically developed for people with disabilities. Methods: A series of studies have been undertaken through formative and summative evaluation and focus group research with a sample of people recovering from spinal cord injury, traumatic brain injury and stroke. Findings from two studies are presented. Following a demonstration and trial of the devices, observational and questionnaire data were collected to determine participants' perception of each system's usability, appeal and enjoyment. Results: The first study involved evaluation and focus group discussions of seven participants (two females, five males) with SCI (n54) and CVA (n53). Findings indicated that interaction with the EyeToy interface appeared to be more intuitive than the use of the Wii-mote interaction device, although some participants had difficulty navigating the menu of the PlayStation EyeToy. The second study involved evaluation of six participants (SCI54 males, TBI51 male, CVA51 female), aged between 25 and 58 years. The investigator was able to increase or decrease the difficulty of a game (developed specifically for bimanual rehabilitation task), using an off-the-shelf haptic feedback device, for each participant, depending on their skill level so that each participant was able to work at a level that was challenging to them. In both studies, participants reported that they would be more motivated to exercise if playing these types of games in both the clinic and home setting. Conclusion: This series of usability tests is the first phase within a program of work using gaming for a range of physical disabilities. The use of virtual reality and video games for rehabilitation offers potential for motivating patients to perform specific therapy tasks.
Article
Full-text available
We argue that 4 fundamental gestalt phenomena in perception apply to the control of motor action. First, a motor gestalt, like a perceptual gestalt, is holistic in the sense that it is processed as a single unit. This notion is consistent with reaction time results indicating that all gestures for a brief unit of action must be programmed prior to initiation of any part of the movement. Additional reaction time results related to initiation of longer responses are consistent with processing in terms of a sequence of indivisible motor gestalts. Some actions (e.g., many involving coordination of the hands) can be carried out effectively only if represented as a unitary gestalt. Second, a perceptual gestalt is independent of specific sensory receptors, as evidenced by perceptual constancy. In a similar manner a motor gestalt can be represented independently of specific muscular effectors, thereby allowing motor constancy. Third, just as a perceptual pattern (e.g., a Necker cube) is exclusively structured into only 1 of its possible configurations at any moment in time, processing prior to action is limited to 1 motor gestalt. Fourth, grouping in apparent motion leads to stream segregation in visual and auditory perception; this segregation is present in motor action and is dependent on the temporal rate. We discuss congruence of gestalt phenomena across perception and motor action (a) in relation to a unitary perceptual-motor code, (b) with respect to differences in the role of awareness, and (c) in conjunction with separate neural pathways for conscious perception and motor control.
Conference Paper
The aim of this paper is to develop a validated system for remote monitoring by health professionals of home-based upper limb rehabilitation by utilising action-video games, data analysis algorithms and cloud server technology. Professionally-written action-video games designed specifically for upper limb rehabilitation were used and game controllers provided continuous 3D kinematic data of hand and arm position. Assessments were made in the patient’s home when they played a bespoke ‘assessment’ mini game controlled by 40 representative actions. An occupational therapist also undertook a blinded clinical CAHAI assessment. For each move 8 scalar variables were defined from both limbs, giving 320 covariates. There were entered into a multiple linear regression random effects model which identified 15 covariates derived from 12 movements that explained 80% of the variance in the CAHAI scores. We conclude that remote monitoring by health professionals of home-based upper limb rehabilitation is possible using data collected remotely from video game play.
Article
Effective stroke rehabilitation must be early, intensive and repetitive, which can lead to problems with patient motivation and engagement. The design of video games, often associated with good user engagement, may offer insights into how more effective systems for stroke rehabilitation can be developed. In this paper we identify game design principles for upper limb stroke rehabilitation and present several games developed using these principles. The games use low-cost video-capture technology which may make them suitable for deployment at home. Results from evaluating the games with both healthy subjects and people with stroke in their home are encouraging.
Article
Reaction time (RT) prior to speech articulation increased as a function of response complexity. The RT findings formed 2 patterns, each of which was a different Response Complexity x Paradigm (choice RT vs. simple RT) interaction. That result extends previous findings from manual button-pressing tasks (S. T. Klapp, 1995) to a different action modality. Two different types of response programming, INT and SEQ, are assumed in the interpretation. Whereas INT can be identified with response programming within a word, SEQ fits a different interpretation related to timing of onsets of speech units. A critical assumption is that a long response is represented as a sequence of chunks; that organization is subject to manipulation. New findings suggest some modifications of the previous theory.
Article
Much of neurorehabilitation rests on the assumption that patients can improve with practice. This review will focus on arm movements and address the following questions: (i) What is motor learning? (ii) Do patients with hemiparesis have a learning deficit? (iii) Is recovery after injury a form of motor learning? (iv) Are approaches based on motor learning principles useful for rehabilitation? Motor learning can be broken into kinematic and dynamic components. Studies in healthy subjects suggest that retention of motor learning is best accomplished with variable training schedules. Animal models and functional imaging in humans show that the mature brain can undergo plastic changes during both learning and recovery. Quantitative motor control approaches allow differentiation between compensation and true recovery, although both improve with practice. Several promising new rehabilitation approaches are based on theories of motor learning. These include impairment oriented-training (IOT), constraint-induced movement therapy (CIMT), electromyogram (EMG)-triggered neuromuscular stimulation, robotic interactive therapy and virtual reality (VR). Motor learning mechanisms are operative during spontaneous stroke recovery and interact with rehabilitative training. For optimal results, rehabilitation techniques should be geared towards patients' specific motor deficits and possibly combined, for example, CIMT with VR. Two critical questions that should always be asked of a rehabilitation technique are whether gains persist for a significant period after training and whether they generalize to untrained tasks.
Article
The magnitude of disability among elderly stroke survivors is substantial. There have been few community-based estimates of the contribution gender and older age make to stroke-related disability and outcome. Using the original Framingham Study cohort, we documented gender-specific neurological deficits and disability differences in stroke survivors at six months post-stroke. Logistic regression analyses were performed to estimate odds ratios, comparing men and women, and adjusting for age, and age and stroke subtype. Age and gender-matched controls were then compared to distinguish stroke-related disability from disability associated with general aging. Results showed that almost half (43%) of all elderly stroke survivors in the cohort had moderate to severe neurological deficits. In the crude analyses, women were more dependent in ADLs (33.9% vs 15.6%), less likely to walk unassisted (40.3% vs 17.8%), and living in nursing homes (34.9 % vs 13.3%). After adjusting for age and stroke subtype, it was older age that accounted for the severity of disability. When compared to age and gender-matched controls, stroke cases were significantly more disabled in all domains studied. In this elderly cohort, more women experienced initial strokes and were more disabled at 6 months post-stroke than men. However, older age at stroke onset, not gender or stroke subtype, was associated with greater disability. Health care providers need to understand that strokes occur later in life for women and that because of age, women are at greater risk for disability and institutionalization.
Evaluating Functional Ability of Upper Limbs after Stroke Using Video Game Data
  • J Q Shi
  • J. Q.