Content uploaded by Misha Pavel
Author content
All content in this area was uploaded by Misha Pavel
Content may be subject to copyright.
Abstract —Cognitive assessments and the early detection of
dementia are an important component of clinical care. In this
paper we describe an approach to continuous monitoring of
sensory-motor function that corresponds to standard tests of
cognitive function, but measured more frequently and in a
natural home environment. The approach is based on
monitoring human-computer interactions using standard
devices such as keyboard and pointing devices.
Keywords — home health, aging, unobtrusive monitoring,
computer interaction, cognitive assessment.
I. INTRODUCTION
Many nations are facing severe economic and social
problems associated with the health care needs of a rapidly
growing elderly population [1]. With longer life
expectancies come the increased possibility of cognitive
decline and the resulting loss of independence. 10% of
people over age 65 are afflicted with cognitive decline,
increasing to 50% by age 85. Many require some level of
assisted living, as even mild cognitive decline can result in a
loss of independence and significantly reduced quality of
life [2-4].
Maintaining independence is paramount to maintaining
quality of life. With the advent of new medications, early
detection and intervention allow the greatest opportunity for
individuals to delay onset of dementia for elders and, with
the help of their doctors and caregivers, to develop strategies
for compensating and coping with deficiencies, thus
allowing them to maintain independence. Ongoing
cognitive monitoring would also enable detection of
cognitive events due to medication effects, such as drug-
drug interactions, adverse side effects, and missed dosages.
A. Standard Cognitive Assessments
In standard clinical practice, routine cognitive
assessments are uncommon for screening purposes in
advance of symptoms. In fact, screening for dementia or
Alzheimer’s disease is typically delayed until symptoms are
advanced. In addition, follow-up monitoring after
diagnosis occurs at infrequent intervals (often yearly, at
best). Table 1 lists some standard cognitive tests that cover
a spectrum of functions [5-7]. Although there is substantial
overlap in the cognitive dimensions used in each of these
This work was supported by Intel Corporation and the Oregon Roybal
Center for Technology in Aging, Community, and Health.
tests, performance in these areas is predictive of both current
and potentially future cognitive decline. The Finger Tap
Test (highlighted in the table) is a motor task that has
substantial predictive power. The test measures how
quickly a patient is able to tap a pad or keyboard in a 10
second interval. The test is often performed on both the
dominant and non-dominant hand. We have highlighted this
task for our project because it is a simple motor task that
likely corresponds with various computer interactions, such
as typing speed and pointing device interactions.
Cognitive
Domain
Tests
Attention &
Concentration
WAIS-R Digit Span
Crossing-off Test
Speed of
Processing
Simple/Choice Reaction Time
WAIS-R Digit Symbol
Finger Tap Test
Memory Letter-Number Sequencing
CERAD Word List
WMS-R Logical Memory
CERAD Visual Figures Recall
Language WRAT-R
Verbal Fluency
Boston Naming (CERAD)
Executive
Function
Odd Man Out task
Spatial or Visual
Perceptual
WAIS R Block Design
Table 1: Standard tests used to assess cognitive function for various
domains of cognitive performance. The highlighted Finger Tap Test
corresponds to the computer monitoring approach described in this paper.
B. Background on the Finger Tap Test
The Finger Tapping Test was originally developed as part
of the Halstead Reitan Battery of neuropsychological tests.
It is one of the most basic and frequently used measures of
motor speed and motor control [8]. It has been shown to be
a sensitive predictor of cognitive decline [9-10]. In addition
to measuring direct motor performance, the Finger Tap Test
Unobtrusive Computer Monitoring of Sensory-Motor Function
H. B. Jimison1, M. Pavel2, J. McKanna2
1 Department of Medical Informatics, Oregon Health and Science University, Portland, OR, USA
2Department of Biomedical Engineering, Oregon Health and Science University, Portland, OR, USA
measure is affected by levels of alertness, impaired ability to
focus attention, or slowing of responses. A comparison of
performance on dominant and non-dominant hands can be
used as a measure of the integrity of brain function.
Additionally, overall slow finger tap speeds have been
shown to be associated with Alzheimer’s disease [11],
schizophrenia [12], and traumatic brain injury [13].
Several finger-tapping devices are in routine use for
cognitive testing. The most commonly used device is a
tapping lever mounted with a key driven mechanical
counter. There are also electronic and software versions
available in computer-based cognitive tests. However, as
with paper-based tests, these are typically administered in a
clinical setting, often after signs of cognitive decline are
apparent, when opportunities for intervention or remediation
are significantly diminished. Further, they compare subject
performance at one point in time to norms for a similar
population. In the approach to cognitive monitoring that we
describe in this paper, we propose a method for continuous
monitoring of sensory-motor interactions with routine
computer use to observe performance trends and variability.
This approach takes advantage of the growing use of
computers by the elderly.
C. Computer Use by the Elderly
According to findings published in 2003 by Harris
Interactive [14] 32% of adults 65 and over are online.
Additionally, adults 65+ are participating at faster rates, and
that trend is expected to continue. In a recent Pew Internet
and American Life Survey [15], 93% of seniors with
Internet access have sent or received email, and seniors are
more inclined to go online and check email on any given
day than any other group of Internet users. Second only to
email in popularity is researching information, especially
health topics, and over a third of this population goes online
to play games.
This level of computer activity in elders provides an
opportunity for the measurement of sensory-motor
interactions on a regular basis. In this paper we discuss an
approach for measuring keyboard and pointing device
interactions as an indicator of sensory-motor / cognitive
performance.
II. DATA COLLECTION
To test our approach to monitoring sensory-motor
interactions with elders, we recruited 9 elderly residents
(mean age 79.5 ± 8.5) in a senior residential facility, where
they used computers in the facility’s library setting with four
computers. Each of these computers had a trackball as a
pointing device. We installed monitoring software on each
of the four computers. The software recorded all keyboard
entries, all trackball movements, and time spent in each
computer application. For each event, the date, time and
interval since last event was recorded.
The monitoring software was completely invisible to the
elderly users of the computers. It started up by itself
whenever there was a system reboot, and it uploaded the
data to a secure server at Oregon Health & Science
University each night at approximately 2:00 am, after
determining that the system was free. Also, the monitoring
software did not display any new windows, thus minimizing
the chance of being turned off by a user.
The subjects were also given a battery of standard
cognitive tests as part of the experiment. Based on their
performance on Logical Memory II (delayed paragraph
recall) the subjects were classified as normal (n=6) or MCI
(mild cognitive impairment) (n=3).
III. ANALYSIS OF KEYBOARD INTERACTIONS
The goal of our work with an analysis of individuals’
interaction with a keyboard on a day-to-day basis was to
approximate the Finger Tap Test and the assessments
represented with that test. Most computer users spend a
great deal of time in word processing activities, such as
email and document creation. Our objective in this work is
to develop a measure of typing speed that is as reliable and
consistent as possible. Clearly, under normal conditions, a
computer user’s typing speed will vary, depending on
distractions, pausing when thinking of what to type, etc. In
addition, people have varying typing abilities. Our
exploration of measures to minimize this unwanted
variability has led us to considering speed of typing during
consistent periods of word processing and considering
repeated keyboard events, such as logins.
An advantage of measuring login speed is that it is
relatively free of contamination with context, distractions,
language abilities, and typing abilities. Additionally, it is a
short, easily-distinguishable repeated event that can be
quickly selected from the large data files and averaged for
mean estimates.
Figure 1 shows median inter-stroke intervals on logins
for three of the subjects in our experiment. A larger inter-
stroke interval is indicative of slower typing speed. We took
the median value for login speed for each day, as the most
robust measure of typing speed. Another important
variable to consider in predicting cognitive problems is the
variability in scores over time. In many cognitive
monitoring tests we have performed, we have noticed that
subjects with cognitive impairment have both lower scores
on average, and also a much higher variability from day to
day. In Figure 1, the lower two lines represent two
cognitively healthy elders whose data on average is both low
(faster typing speed) and stable from day to day. The top
line is data from a subject with mild cognitive impairment.
The speed of typing for this person is highly variable from
day-to-day and on average slower. Thus, the simple
measure of login speed seems to be a promising indicator of
cognitive performance.
Figure 1: Plot of median interstroke interval on login for 3 elderly subjects.
The lower 2 lines show cognitively healthy subjects with low variability in
their typing speed. The line on top is a plot of a subject with mild cognitive
impairment (slower typing – i.e., greater interstroke interval, and greater
variability in performance).
IV. ANALYSIS OF POINTING DEVICE
INTERACTIONS
Another source of useful sensory-motor information
involves interactions with pointing devices such as mouse or
trackball. These interactions require the user to execute
visually-guided movements. Our initial investigations are
focused on the idea that the trajectories executed by the user
may provide useful information regarding his or her
cognitive processes.
In a similar manner to the keyboard-based interactions,
the context of the interaction may greatly influence the
trajectories and their interpretations. To sidestep the issue
for the purpose of this paper, we focused our analysis on the
interactions with pointing devices during the game of
FreeCell. In this situation, it is possible to assess the context
of the moves from the state of the game.
The basic data consist of point-to-point movements,
where each move is represented by samples in time
corresponding to the locations of the cursor on the computer
screen. An example of a trajectory of a move is shown in
Figure 2.
A key question involves how best to represent the
trajectories in terms of a small number of parameters that
would capture the performance of the subjects in a way that
would most likely relate to their cognitive ability. This
representation should, of course, be rotation and scale
invariant and capture characteristics such as tremor and
inaccurate aiming. In previous research, we and others [16]
used techniques such as the number of “straight” segments,
but those parameterizations depend on the scale of
measurement and the definition of “straight.” For example,
a segment could be deemed to be straight if the maximum
perpendicular distance is less than a given threshold.
In order to avoid the necessity of making such
assumptions, we developed several metrics that are
relatively independent of scale. The first of these is the ratio
of the lengths of the trajectory to the distance between the
endpoints. This metric measures the deviation from a single
straight line. A straight line, however, is not necessarily the
most efficient way for a human to move from one point to
another because of the kinematic and dynamic constraints of
human articulated mechanisms of the arm and hand.
For that reason we developed a novel approach
borrowed from machine vision called Fourier Descriptors
[17]. Intuitively, Fourier Descriptors capture the trajectory
in terms of harmonic functions that capture the various rates
of deviation from the straight line. Formally, if the
trajectory is described in terms of the coordinates in a
complex plane, then
(
)() ()
ut xt jyt=+ (1)
where j designates an imaginary dimension. Given this
formulation, it is natural to express the trajectory in terms of
its harmonic components using the Fourier representation
() ()
1
0
12
exp
N
k
j
kt
ut ak
NN
π
−
=
=
∑ (2)
where the coefficients a(k) are computed
() ()
1
0
2
exp
N
k
j
kt
ak ut N
π
−
=
−
=
∑ (3)
The coefficient corresponding to the k=0 represents the
location of the trajectory, and thus we ignore it for the
purpose of this analysis. The remaining coefficients
represent the trajectory in terms of components that vary
with higher frequencies.
For the purpose of the analysis of the pointing device,
trajectories were sampled and interpolated so that the
sample spacing was approximately 10 pixels along its
Milliseconds
Date
Subject with
mild cognitive impairment
Two cognitively
healthy elders
50 100 150 200 250 300 350 400
0
50
100
150
200
250
300
X
Y
Figure 2. A typical trackball movement during a game of FreeCell.
length. In order to avoid discontinuities, the paths were
extended by their mirror images in each coordinate.
A graph of the first 10 components for the trajectory in
Figure 2 is shown in Figure 3. It is interesting to note that
the global characteristics of each move are represented by
the “low frequency” components, while the small deviation,
e.g., corresponding to tremor, would be characterized by the
higher components. This can be seen by band-limiting the
Fourier Descriptor representation to the low-frequency
components and reconstructing the move.
For the purpose of the trajectory characterization, we
used the total power in the trajectory and in different sub-
bands.
V. CONCLUSIONS
We have presented a new method for continuous in-
home monitoring of sensory-motor function, as a potential
indicator of cognitive performance. We analyzed both
keyboard and pointing device interactions to develop robust
measures of naturally occurring motor effects. The
advantage of this approach over conventional cognitive tests
is that it is inexpensive and continuous. With trend
detection, we are able to use subjects as their own controls
and depend less on referring to data from population norms.
This is a promising approach for the early detection of
cognitive problems, as well as for monitoring cognitive
effects of medication management.
REFERENCES
[1] A. M. Garber, Alan M. 1989. "Long-Term Care, Wealth, and Health
of the Disabled Elderly Living in the Community." In David A. Wise
(Ed.), The Economics of Aging. Chicago: the University of Chicago
Press.
[2] Social Security Online, http://www.ssa.gov/OACT/TR/TR04/
index.html, last update March 23, 2004.
[3] US Census Bureau, www.census.gov/population/www/
socdemo/age.html, last revised: November 04, 2003.
[4] J. D. Kasper, 1988. "Using the Long-Term Care Surveys: Longitudinal
and Cross-Sectional Analyses of Disabled Older People."
Proceedings of the 1987 Public Health Conference on Records and
Statistics. DHHS Pub.No.88-1214. Hyattsville, MD: National Center
for Health Statistics, pp.353-358.
[5] Kiernan, R., Meuller, J., Langston, J., and Van Dyke, C. "The
Neurobehavioral Cognitive Status Examination: A brief but
differentiated approach to cognitive assessment," Annals of Internal
Medicine, vol. 107, pp. 734-739, 1987.
[6] Chen, P., Ratcliff, G., Phil, D., Belle, S., Cauley, J., De Kosky, S., and
Ganguli, M ."Cognitive tests that best discriminate between
presymptomatic AD and those who remain nondemented,"
Neurology, vol. 55, pp. 1847-1853, 2000.
[7] Callahan, C. H. Hendrie, and W. Tierney, "Documentation and
evaluation of cognitive impairment in elderly primary care patients,"
American College of Physicians, vol. 122, pp. 422-429, 1995.
[8] M. N. Mitrushina, K.B. Boone, L.F. D’Elia, Handbook of Normative
Data for Neuroopsychological Assessment. Oxford University Press,
1999.
[9] D. Howieson, S. Bowman, D. Coehlo, S. Lim, and J. A. Kaye, J.
"Natural history of cognitive decline in the old old," Neurology, vol.
60, pp. 1489-1494, 2003.
[10] D. Howieson, A. Dame, R. Camicioli, G. Sexton, H. Payami, and J. A.
Kaye, "Cognitive markers preceding Alzheimer's dementia in the
healthy oldest old," J Am Geriatr Soc, vol. 45, pp. 584-9, 1997.
[11] J. S. Wefel, B. D.Hoyt, and P. J. Massman, (1999).
"Neuropsychological functioning in depressed versus nondespressed
Alzheimer's patients." The Clinical Neuropsychologist, 13, 249-
257.Anderson, RN in Vital and Health Stats. (National Center for
Health Statistics, 1999).
[12] N. C. Andreasen, L. Flashman, M. Flaum, S. Arndt, V. Swayzee, D.
S. O'Leary, J. C. Ehrhardt, W. T. C. Yuh, (1994) "Regional brain
abnormalities in schizophrenia measured with magnetic resonance
imaging." JAMA 272:1763-1769
[13] D. S. Geldmacher, E. C. Hills. "Effect of stimulus number, target-to-
distractor ratio and motor speed on visuospatial search quality
following traumatic brain injury." Brain Injury 1997; 11(1):59-66.
[14] D. Krane, "Who's Online," Harris Interactive, 2003.
[15] Lenhart, A. "The ever-shifting internet population," Pew Internet and
American Life Project, 2003.
[16] I. S. MacKenzie, T. Kauppinen, and M. Silfverberg, "Accuracy
measures for evaluating computer pointing devices," presented at CHI
2001 - ACM Conference on Human Factors in Computing Systems,
New York, 2001
[17] C. T. Zahn and R. Z. Roskies, "Fourier descriptors for plane close
curves," IEEE Trans. Computers, vol. C-21, pp. 269-281, 1972.
[18] S. Keates, F. Hwang, P. Langdon, P.J. Clarkson, P. Robinson, “The
use of cursor measures for motion-impaired computer users.”
Universal Access in the Information Society (UAIS) , Springer-Verlag,
2(1), November, 2002, 18-29.
1 2 3 4 5 6 7 8 9
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
Component Frequency
Magntide of Components
Figure 3: First 9 Fourier Descriptors
