Available via license: CC BY 4.0
Content may be subject to copyright.
J. Sens. Sens. Syst., 8, 95–104, 2019
https://doi.org/10.5194/jsss-8-95-2019
© Author(s) 2019. This work is distributed under
the Creative Commons Attribution 4.0 License.
A study of hand-movement gestures to substitute for
mouse-cursor placement using an inertial sensor
Romy Budhi Widodo1, Reyna Marsya Quita2, Rhesdyan Setiawan1, and Chikamune Wada3
1Informatics Engineering Study Program, Ma Chung University, Malang, 65151, Indonesia
2Department of Mathematics, Faculty of Science, National Central University, Taoyuan City, 32001, Taiwan
3Graduate School of Life Science and Systems Engineering,
Kyushu Institute of Technology, Wakamatsu, Fukuoka, 808-0196, Japan
Correspondence: Romy Budhi Widodo (romy.budhi@machung.ac.id)
Received: 22 October 2018 – Revised: 23 January 2019 – Accepted: 31 January 2019 – Published: 18 February 2019
Abstract. This paper examines the new study of hand orientation as a substitute for computer-mouse movement
and is evaluated based on ISO/TS 9241 part 411: Ergonomics of human–system interaction-evaluation methods
for the design of physical input devices. Two pairs of hand-orientation candidates were evaluated, using, for
example, pitch–roll and pitch–yaw to substitute for up–down and left–right mouse-cursor movements. The up–
down cursor movement was generated from the pitch orientation, while the left–right cursor movement was
generated from the roll or yaw orientation, depending on the evaluation of the proposed gesture. The research
employed a standard computer mouse as a baseline comparison for the study. The empirical study was conducted
to evaluate quantitative performance such as throughput and movement time. The best impression resulted when
the throughput had the greatest value as well as the shortest movement time. The performance test was based
on Fitts’s law using a multi-directional tapping test as suggested by ISO/TS 9241-411. The test was divided into
several levels of difficulty, including high, medium, low, and very low. The other assessment is qualitative and
was performed using the comfort-rating scale questionnaire and rating of perceived exertion of comfortability
and fatigue. The quantitative results show that pitch–yaw throughput is slightly higher than for the pitch–roll
gesture, and that the movement time in pitch–yaw is slightly less than in pitch–roll, although there is no statis-
tically significant difference between the two. We also found that pitch–yaw movements have a higher level of
comfort based on the comfort-rating scale test. Since the test was divided into levels of difficulty, we identified
those gestures suitable for the task with a low and very low level of difficulty based on throughput, movement
time, and error-rate results. Finally, this study suggests that pitch–roll and pitch–yaw movements of the hand can
be used as substitutes for the mouse, and that pitch–yaw movements are superior in regard to causing less fatigue
than pitch–roll movements. Furthermore, this study provides a new suggestion for a suitable level of difficulty
when using an inertial sensor as an emulator for the movement of a mouse cursor in the field of human–computer
interaction.
1 Introduction
A computer mouse’s main function is as a pointing device for
the user to navigate, target, and command execution through
mouse movement and button-clicked action (Lazar et al.,
2017; Natapov et al., 2009). The mouse as a pointing device
could not be used by someone who is disabled for certain rea-
sons. (1) The fingers’ impairment caused by a malfunction of
the sensoric system and congenital disorder; (2) the person’s
difficulty operating a computer in a sitting position. There-
fore, a study of the suitable hand gestures or hand movement
or hand orientation which serve as a substitute for a conven-
tional mouse is needed.
Some research found that a mouse replacement could be
categorized into some groups, for example handglove, grasp-
ing, and optic types. The material used in the handglove
type using an acceleration sensor was introduced in Perng
et al. (2002); an acceleration sensor was also used in edu-
Published by Copernicus Publications on behalf of the AMA Association for Sensor Technology.
96 R. B. Widodo et al.: Hand-movement gestures to substitute for mouse-cursor placement
tainment as a control (Kranz et al., 2010); fiber optic in Zim-
merman et al. (1986); a flexible plastic resistive ink sensor as
in Power Glove by Mattel, Inc. (Sturman and Zeltzer, 1994);
and ultrasonic and magnetic hand position tracking technol-
ogy as in Data Glove (Zimmerman et al., 1986) and Zim-
merman and Lanier (1989). The grasping type as in a Wii re-
mote, GyroPoint, and RemotePoint was discussed and stud-
ied in Natapov et al. (2009), MacKenzie and Jusoh (2001),
and Norman and Norman (2010). The use of optic types such
as a laser pointer as a pointing device has been discussed in
Myers et al. (2002) and Oh and Stuerzlinger (2002). Much
of the current literature on pointing devices pays particular
attention to others evaluating and comparing pointing de-
vices; however, the investigation of gestures has not been
highlighted in those studies.
One of the most significant parts that can be used to em-
ulate the movement of a mouse is a limb, due to its ability
in multi-directional movement. The wrist movement in the
tri-axial plane, such as the frontal, median, and transverse
planes, represents the orientation of roll, pitch, and yaw, re-
spectively. The wrist movement consists of flexion–extension
and radial–ulnar deviation; the forearm movement consists
of forearm pronation and forearm supination as in Gates et
al. (2016) and Nelson et al. (1994). In this paper we relate
those movements to the orientation axis, in which flexion–
extension represents pitch, pronation–supination represents
roll, and radial–ulnar deviation represents yaw. Figure 1 il-
lustrates the wrist and forearm movement. The range of
motion related to these movements reported in Gates et
al. (2016) and Nelson et al. (1994) for wrist flexion and ex-
tension is 38 and 40◦; wrist radial and ulnar deviation: 28 and
38◦; and forearm pronation and supination: 13 and 53◦.
Inspired by Perng et al. (2002), Zimmerman et
al. (1986), Sturman and Zeltzer (1994), and Zimmer-
man and Lanier (1989), and evaluated by Natapov et
al. (2009), MacKenzie and Jusoh (2001), Norman and Nor-
man (2010), MacKenzie et al. (2001), and Widodo and Mat-
sumaru (2013), this study set out to clarify several aspects
of the two candidates of movement gestures: pitch–roll and
pitch–yaw, to substitute the movement of the mouse cursor.
We worked on comparing the performance of pitch–roll and
pitch–yaw quantitatively and qualitatively based on ISO/TS
(International Standards Organization/Technical Specifica-
tion) 9241 part 411: evaluation methods for the design of
physical input devices.
The rest of the paper is organized as follows: Sect. 2 dis-
cusses ISO/TS 9241 related to the evaluation procedure and
Fitts’ formula, Sect. 3 discusses the research methodology,
Sect. 4 presents the experiment results, and Sect. 5 elabo-
rates on the results as a discussion. Lastly, Sect. 6 presents
the conclusion of the study.
Figure 1. Wrist and forearm movement: (a) flexion–extension
represents pitch; (b) radial–ulnar deviation represents yaw, and
(c) pronation–supination represents roll. The angle value is the nor-
mal value based on the previous study.
2 ISO/TS 9241-411
ISO 9241 is a standard used for human-system interaction
(International Organization for Standardization, 2012). ISO
9241 part 411 (ISO/TS 9241-411) discusses the evaluation
methods for the design of physical input devices. The quanti-
tative assessment of performance was measured by through-
put and movement time, as well as using a comfort-rating
scale to assess comfort qualitatively. The dependent measure
of Throughput (TP) defined in ISO was based on Fitts’ law
model. Fitts’ law proposed an index of difficulty of a move-
ment based on the relationship between distance (amplitude),
movement time (duration), and distance variability. The TP
is the index of difficulty (ID) divided by movement time
(tm) (Fitts, 1954; Mackenzie, 2018). Based on the Shannon–
Hartley theorem, the formulation of the ID is in Eq. (1):
ID=log2
d+w
w(bit),(1)
where dis the distance of movement and wis the target
width. The ISO procedure includes the four levels of diffi-
culty (ID), that is, high (ID > 6); medium (4 < ID≤6); low
(3 < ID≤4); and very low (ID≤3).
The tapping coordinates to a user spreading around the tar-
get’s center. Therefore, the scatter data should be used to
adjust the accuracy of each user as suggested in Macken-
zie (2018). The ISO standard dependent measurement,
throughput, was calculated using this adjustment for accu-
J. Sens. Sens. Syst., 8, 95–104, 2019 www.j-sens-sens-syst.net/8/95/2019/
R. B. Widodo et al.: Hand-movement gestures to substitute for mouse-cursor placement 97
Figure 2. (a) Pattern of the multi-directional tapping task:
d=distance of movement; w=target width. (b) Enlargement of
one target circle; (xc,yc) is the actual clicked target; “x” indicates
the clicked-coordinate spreading of each target circle.
racy. Equation (1) was modified to be in Eq. (2):
IDe=log2
d+we
we
;we=4.133sx,(2)
TP =IDe
tm
,(3)
where weis the effective target width and sxis the standard
deviation of the clicked target’s coordinate. The movement
time (tm) was calculated from one target to the other target
in seconds. Finally, the TP is the effective index of difficulty
(IDe) divided by tmresults in bits per second (bps).
The one-directional tapping task as in Fitts (1954) does
not concern the angle of movement in the performance
assessment; therefore ISO 9241-411 recommends a multi-
directional tapping task. The evaluation using the multi-
directional tapping task was used in Norman and Nor-
man (2010), MacKenzie et al. (2001), and Douglas et
al. (1999). The pattern of the multi-directional tapping task
is illustrated in Fig. 2.
The target consists of 25 small circles, which are tapped
sequentially according to the number or color changes as il-
lustrated in Fig. 2a. The actual clicked target in each small
circle is the center of coordinates of the circles; however,
spreading tapping by each subject in each experiment caused
the effective target and standard deviation (sx). Figure 2b il-
lustrates spreading tapping coordinates by each subject, sym-
bolized by x, spread around the center of the circle (xc,yc).
Every clicked coordinate out of the circle will be recognized
as an error.
In the beginning, the IDein Eq. (2) was reserved for a
one-directional tapping task; the IDefor a multi-directional
tapping task was calculated based on the extended Eq. (2) as
in Norman and Norman (2010) and the International Orga-
nization for Standardization (2012). For the calculation con-
ducted in each small circle, all clicked coordinates are ana-
lyzed relative to (xc,yc) and finally will be averaged. Here-
after, all equations are for the multi-directional tapping task.
Equation (4) calculates the mean of the clicked coordinates
and, then in Eq. (5), the subtraction for each xand ycoordi-
nate.
X=1
N
N
X
i=1
xi;Y=1
N
N
X
i=1
yi(4)
ˆx=xi−X; ˆy=yi−Y(5)
The two-dimensional standard deviation is as in Eq. (6).
sx=v
u
u
t
1
N
N
X
i=1
d2
i(6)
The distance dis formulated as in Eq. (7).
d2
i= ˆx2+ ˆy2(7)
The calculation of the effective target width (we) is the same
as in Eq. (2), rewritten in Eq. (8). The effective index of dif-
ficulty is written in Eq. (9).
we=4.133sx(8)
IDe=log2d
we
+1(9)
Finally, the throughput (TP) as in Eq. (3) is rewritten in
Eq. (10) as the performance value of the device.
TP =IDe
tm
(10)
3 Research methodology
3.1 Participants
Nineteen right-handed subjects, 15 males and 4 females,
who were an average of 27.1 years old, standard devia-
tion (SD) =6.2, were recruited from university students and
staffs. All subjects were informed about the procedure before
the experiment began.
3.2 Experiment design
The experiment was conducted using a within-subject experi-
mental design. The learning effect was reduced by two ways:
(1) randomizing the order of experiment based on index of
difficulty level (ID level), and (2) conducting a sufficient ses-
sion for practice until the subject could get used to operating
the evaluation software and experimental apparatus. Every
subject used two devices: a standard mouse and an inertial
sensor. The inertial sensor was used in two ways: pitch–roll
and pitch–yaw gestures; therefore, in this paper we treated
the sensor as two devices; the total number of devices was
three, including the standard mouse. There are four levels of
difficulty: (1) mode 1 is very low level of difficulty; (2) mode
www.j-sens-sens-syst.net/8/95/2019/ J. Sens. Sens. Syst., 8, 95–104, 2019
98 R. B. Widodo et al.: Hand-movement gestures to substitute for mouse-cursor placement
Table 1. Index of difficulty design (the range of ID is recommended
by ISO).
d(pixels) w(pixels) ID (bits) ID level
350 50 3 Very low (mode 1)
600 60 3.459 Low (mode 2)
600 20 4.954 Medium (mode 3)
800 12 6.066 High (mode 4)
2 is low level of difficulty; (3) mode 3 is medium level of dif-
ficulty; and (4) mode 4 is high level of difficulty. Table 1 de-
scribes the design of ID levels using a computer display reso-
lution of 1280×1024 pixels; the dand windicate the distance
of movement and target width, respectively (see Fig. 2a). The
number of blocks are three and three trials per block. There-
fore, for 19 subjects, the design is 19×3×4×3×3; the num-
ber of trials was 2052.
The design for statistical analysis is as follows. First, the
data for TP and tmwere investigated using the Shapiro–Wilk
test to determine whether they represented normality data. A
non-parametric test using the Kruskal–Wallis H test was em-
ployed to determine whether the data deviated significantly
from a normal distribution, followed by the Mann–Whitney
Upost hoc test. For normal data, the homogeneity of vari-
ance test was employed. One-way ANOVA was applied if
the assumption of homogeneity of variances was fulfilled,
and this was followed by a post hoc test. If the data failed the
assumption of homogeneity, we employed Welch’s ANOVA
instead of ANOVA.
3.3 Apparatus/materials
The experiment involves the measurements of three compo-
nents of rotation as independent parameters, namely pitch,
roll, and yaw. The number of independent parameters re-
ferred to as degree of freedom (DOF) defines the config-
uration of the analysis of the system’s bodies. The experi-
ments used 3 DOF tracking InertiaCube 4™to record the
orientation angle such as pitch, roll, and yaw. The manu-
facturer’s accuracy specification: 1◦in yaw, 0.25◦in pitch
and roll at 25 ◦C. The other input device is a standard mouse
(Microsoft®Basic Optical Mouse v2.0) as a baseline condi-
tion. The C# software was developed to record orientation
data, emulate the mouse-cursor movement using the orienta-
tion angle data, and display the multi-directional tapping task
simultaneously. Software specification was designed to ful-
fill Annex B of ISO/TS 9241-411 which consists of (1) four
levels of difficulty; (2) movement time recording, (3) clicked
coordinate recording, and (4) an error count indicator, which
is accompanied by sound feedback when a subject clicks an
area outside the target. The qualitative assessment of com-
fort and fatigue was conducted using the comfort-rating scale
questionnaire and rating of perceived exertion (RPE), as sug-
gested by Annex C of ISO/TS 9241-411.
Figure 3. Illustration of experimental conditions: (1) Subject (0.9 m
from display); (2) Inertial sensor (mounted on the back of the
dominant hand); (3) Click part (grasped with the dominant hand);
(4) Display (computer monitor).
Figure 4. (a) Orientation of the sensor; (b) cursor space axes;
(c) the sensor–cursor mapping: “+” and “−” signs correspond to
the directions in (a) and (b).
3.4 Procedure
The system is illustrated in Fig. 3. A subject sits about 0.9 m
from the display, the forearm resting on the chair armrest
when using the 3 DOF sensor for testing. However, the hand
is normally on the desk when operating a mouse test. The 3
DOF tracking sensor was mounted on the back of the domi-
nant hand, which is the middle part of the dorsal surface. The
right and left mouse click events were the same for all lev-
els of the test. Subjects used a conventional mouse grasped
with the dominant hand as the clicking part by employing
the mouse’s left button. The PC monitor displays the multi-
directional tapping task. The sound speaker gives a warning
when the subject misses the target; the sound speaker is not
shown in the figure. Figure 4 illustrates the orientation of the
axes of the sensor; θy(pitch), θx(roll), and θz(yaw) are the
rotation angles about the y,x, and zaxes, respectively. Fig-
ure 4 also describes the mapping for a sensor and cursor.
J. Sens. Sens. Syst., 8, 95–104, 2019 www.j-sens-sens-syst.net/8/95/2019/
R. B. Widodo et al.: Hand-movement gestures to substitute for mouse-cursor placement 99
Table 2. Experiment results (in detail).
BaMbID (bits) Mouse Pitch–roll Pitch–yaw
weIDetmTP weIDetmTP weIDetmTP
(pixel) (bits) (s) (bits s−1) (pixel) (bits) (ms) (bits s−1) (pixel) (bits) (ms) (bits s−1)
1
1 3.00 43.13 3.19 0.72 4.45 52.34 2.95 2.36 1.25 53.01 2.93 1.82 1.61
2 3.46 53.63 3.61 0.78 4.65 63.43 3.39 2.84 1.19 65.35 3.35 2.18 1.54
3 4.95 19.14 5.02 1.08 4.64 24.46 4.68 4.50 1.04 24.06 4.70 4.21 1.12
4 6.07 12.24 6.06 1.35 4.47 15.76 5.70 7.93 0.72 16.18 5.66 7.41 0.76
2
1 3.00 45.32 3.13 0.66 4.71 54.92 2.88 1.73 1.67 51.47 2.97 1.64 1.81
2 3.46 66.65 3.48 0.73 4.77 63.88 3.38 2.06 1.64 64.33 3.37 1.90 1.78
3 4.95 19.56 4.99 1.00 5.00 23.97 4.70 3.82 1.23 24.27 4.69 3.35 1.40
4 6.07 12.71 6.00 1.29 4.66 15.67 5.70 6.47 0.88 16.00 5.67 5.63 1.01
3
1 3.00 46.25 3.10 0.66 4.68 53.21 2.92 1.62 1.80 53.32 2.92 1.54 1.90
2 3.46 57.10 3.53 0.73 4.85 64.75 3.36 1.89 1.78 63.68 3.38 1.74 1.94
3 4.95 19.75 4.97 0.98 5.07 24.19 4.69 3.34 1.40 24.68 4.66 3.08 1.51
4 6.07 12.62 6.01 1.26 4.76 15.75 5.70 5.43 1.05 16.09 5.67 5.18 1.09
Mean 0.94 4.73 3.67 1.30 3.31 1.46
aB stands for block. bM stands for mode.
Before the experiment began, the purpose and experiment
procedure were explained to every subject. Also, the subject
practiced the task until the speed did not show any improve-
ment. The sequence of the index of difficulty level was ran-
domized, as well as the sequence of the devices. The multi-
directional tapping is a point-and-click task, and each ses-
sion consists of 25 clicked targets, which are indicated by 25
small circles (see Fig. 2a). The movement time was recorded
starting when they clicked the first target until when they
clicked the last one, as well as the clicked coordinates and
the number of errors. For the “pitch–roll” gesture, a subject
moved his wrist flexion–extension and forearm pronation–
supination. The “pitch–yaw” gesture is a movement of wrist
flexion–extension and radial–ulnar deviation.
4 Experiment results
4.1 Throughput (TP) and movement time (tm)
Throughput provides a measurement of speed and accuracy.
Table 2 describes the experiment results for throughput (TP)
and movement time (tm) in detail. The summary of the re-
sults includes the error rate in Table 3 presented in “mean
(standard deviation)” and will be used for further discussion.
The result of the error rate comes from the average number
of errors of all blocks and modes for 19 subjects.
Basic descriptive statistics were conducted; deviation from
the normal distribution or tests of normality were conducted
using the Shapiro–Wilk test; the null hypothesis is the data
from a normally distributed population. Figure 5 describes
the boxplot of all data distributions related to throughput and
movement time.
The Shapiro–Wilk testing for normality indicated that the
TP was normally distributed for the mouse, pitch–roll, and
Table 3. Experiment results.
Measurement Device∗
Mouse Pitch–roll Pitch–yaw
TP (bps) 4.73 (0.18) 1.30 (0.34) 1.46 (0.37)
tm(s) 0.94 (0.25) 3.67 (1.96) 3.31 (1.84)
Error rate (%) 2.81 (0.13) 28.19 (1.85) 34.76 (2.13)
∗Presents in mean (SD).
pitch–yaw device group (p> 0.05). Next, the test of homo-
geneity of variances using Levene’s test yields significance
at p=0.025, meaning that variances of TP categories in de-
vices are not equal. The assumption of homogeneity of vari-
ances is not met. The Welch ANOVA was used to understand
whether there is a difference in mean of throughput value in
all devices. The null hypothesis: all TP value means are equal
(i.e., µTP mouse =µTP pitch−roll =µTP pitch−yaw). The alterna-
tive hypothesis (HA) is that at least one category mean is
different. The Games–Howell post hoc test shows that the
multiple comparison table revealed that there are statistically
significant differences between the mouse and the two other
devices (p< 0.05), but there is no statistically significant dif-
ference between pitch–roll and pitch–yaw.
The test for tmindicates that pitch–yaw is p=0.046,
suggesting evidence of non-normality. The independent
Kruskal–Wallis test is summarized as follows: the mean
ranks of tmvalues were statistically significantly different be-
tween categories (χ2(2) =23.473, p=0.0005). The Mann–
Whitney Upost hoc test using multiple comparisons was
conducted to interpret all pairwise comparisons. The results
indicate that the tmin the pitch–roll category was not statis-
tically higher than in pitch–yaw (U=63, p=0.603). How-
www.j-sens-sens-syst.net/8/95/2019/ J. Sens. Sens. Syst., 8, 95–104, 2019
100 R. B. Widodo et al.: Hand-movement gestures to substitute for mouse-cursor placement
Figure 5. The boxplot of all data distribution related to (a) throughput (in bps) and (b) movement time (in seconds).
ever, the tmin the pitch–roll category is significantly higher
than in mouse (U=0.0005, p=0.0005) and the tmin the
pitch–yaw category is also significantly higher than in mouse
(U=0.0005, p=0.0005).
To deeply analyze the influence of the index of difficulty
(mode), the dependent-ttest was conducted to compare the
means between each mode on TP and tm. The dependent vari-
able is the value of TP and tm, while the independent variable
is the same subject present on two occasions on the same
dependent variable. Table 4 concludes the results of signif-
icance levels of each pair. We could see that in all devices,
mode 3 and mode 4 have statistically significant difference
results. Mode 4 is the most difficult mode, which causes the
difference.
4.2 Error rate
The percentage of clicked coordinates outside the target was
calculated and the average is shown in Table 3. Figure 6
shows the graph of error rate using mode as a repetition. The
error rate is related to the index of difficulties; as previously
mentioned, mode 1 is the lowest level of difficulty and mode
4 is the highest level of difficulty. Therefore, as expected, the
error rate of mode 4 is the highest.
As shown in Fig. 6, the error rates of modes 3 and 4 of the
sensor’s gestures are far above the error rate of the mouse.
The error increment from modes 2 to 3 at pitch–roll and
pitch–yaw is 59 % and 58 %, respectively. The error-rate in-
crement is very large compared to the increment of the mouse
from modes 2 to 3, that is, only 11 %. The huge increment of
the error rate also occurs from modes 3 to 4 for pitch–roll
and pitch–yaw, which is 58 % and 55 %, respectively.
4.3 Qualitative results
We conducted the assessment of comfort and fatigue us-
ing a seven-question questionnaire (α=0.79) and a five-
question questionnaire (α=0.85), respectively. Each ques-
Figure 6. Error rate as a function of index of difficulty modes.
tion in comfort and fatigue assessment was a seven-point
Likert scale from “very low” to “very high” levels of com-
fort; however, in the fatigue test, the scale is from “very high”
to “very low” levels of fatigue; therefore, option 7 is the best
impression. Figure 7 shows the results of the comfort ques-
tionnaire (items 1 to 7) and fatigue questionnaire (items 8 to
12). Table 5 describes the mean result of the questionnaire.
By far, all subjects were most comfortable with the mouse
over the pitch–roll and pitch–yaw in all items. For a represen-
tative report, we take item number 7 (“Overall operation of
input device”) as an indicator (U=27.5, p< 0.05) of mouse
compared to pitch–roll and (U=46, p< 0.05) for mouse
compared to pitch–yaw. Another significant difference is in
items 10, 11, and 12 (arm, shoulder, and neck fatigue): it was
reported that pitch–yaw was less in fatigue than the pitch–roll
gesture was (U=89.5, p=0.006; U=107.5, p=0.029;
and U=109.5, p=0.035).
The other assessment is RPE by using the Borg scale (0,
0.5, 1–10 scale; from “nothing at all” to “very, very strong
(almost max.)”) which is conducted on arm, shoulder, and
neck effort assessment. Table 6 describes the details of the
RPE assessment result. Spearman’s rank-order correlation
J. Sens. Sens. Syst., 8, 95–104, 2019 www.j-sens-sens-syst.net/8/95/2019/
R. B. Widodo et al.: Hand-movement gestures to substitute for mouse-cursor placement 101
Table 4. The result of comparison of means (paired samples test).
Device Pair tdf p Note
Throughput
Mouse
Mode 3–mode 4 5.377 2 0.033 significant
Mode 2–mode 3 1.912 2 0.196 –
Mode 1–mode 2 3.115 2 0.089 –
Pitch–roll
Mode 3–mode 4 30.962 2 0.001 significant
Mode 2–mode 3 −3.968 2 0.058 –
Mode 1–mode 2 −4.106 2 0.055 –
Pitch–yaw
Mode 3–mode 4 19.887 2 0.003 significant
Mode 2–mode 3 −25.058 2 0.002 significant
Mode 1–mode 2 −0.615 2 0.601 –
Movement time
Mouse
Mode 3–mode 4 −62.528 2 0.0005 significant
Mode 2–mode 3 18.257 2 0.003 significant
Mode 1–mode 2 33.223 2 0.001 significant
Pitch–roll
Mode 3–mode 4 −7.028 2 0.020 significant
Mode 2–mode 3 17.598 2 0.003 significant
Mode 1–mode 2 5.804 2 0.028 significant
Pitch–yaw
Mode 3–mode 4 −7.389 2 0.018 significant
Mode 2–mode 3 7.450 2 0.018 significant
Mode 1–mode 2 6.070 2 0.026 significant
Table 5. Qualitative result.
Assessment Device∗
Mouse Pitch–roll Pitch–yaw
Mean of comfort 6.44 3.91 4.51
Mean of fatigue 6.06 4.19 4.89
∗On average using the seven-point Likert scale; 7 is the best impression.
revealed that the shoulder’s effort of the pitch–roll and pitch–
yaw relationship had a strong and positive correlation, which
was statistically significant (rs=0.77, p< 0.05). We found
that the assessment of effort in the arm is superior in all de-
vices: it needs more effort to move the cursor to the targets.
5 Discussion
The results of a performance assessment, shown in Table 3 as
indicated by throughput, revealed that the TP of the mouse
is 4.73 bps. This is in line with prior studies, which have
noted the range of the mouse’s TP as 3.7–4.9 bps (Soukoreff
and MacKenzie, 2004), and in MacKenzie and Jusoh (2001),
where the range is 3.0–5.0 bps. The results of the experi-
ment ensure that the methodology, experimental apparatus,
data collection, etc., are apparently in alignment with other
researchers’ techniques.
The results of the TP of two gestures, pitch–roll and pitch–
yaw, are different with the TP of the mouse. Since the TP is
related to precision and movement time, we found that the
Table 6. The result of rating of perceived exertion (RPE) assess-
ment using the Borg scale.
Device∗RPE score
Mouse
Arm 1.526
Shoulder 1.000
Neck 1.026
Pitch–roll
Arm 5.053
Shoulder 4.368
Neck 4.316
Pitch–yaw
Arm 3.526
Shoulder 2.579
Neck 2.421
∗On average using the Borg scale (0, 0.5,
1–10 scale); 0 is the best impression.
error rate between the mouse and the two gestures is also
statistically different. However, in the case of TP and move-
ment time between the two gestures, it is not statistically dif-
ferent, although the TP of pitch–yaw is larger than the TP
of pitch–roll. To understand which part of the index of diffi-
culty causes the significant difference, we conducted a paired
samples test, as shown in Table 4. The results of this study
indicate that a comparison of mode 3 and mode 4 is statisti-
cally different in TP as well as in tm. Similarly, we found that
comparisons of TP in modes 2 and 3 are statistically differ-
ent. Based on Table 4, we suspect that the level of difficulty
in modes 3 and 4, for both the pitch–roll and pitch–yaw, does
www.j-sens-sens-syst.net/8/95/2019/ J. Sens. Sens. Syst., 8, 95–104, 2019
102 R. B. Widodo et al.: Hand-movement gestures to substitute for mouse-cursor placement
Figure 7. Results of the pointing questionnaire, where option 7 on the Likert scale is the best impression.
not represent a suitable task for the sensor. The discussion
below includes the error rate and will complete the discus-
sion of the effect of the difficulty level of the task.
The error rate of the mouse is 2.81 %; pitch–roll is 28.19 %
and pitch–yaw is 34.76 % (see Table 3). We believe that the
large error rates in those gestures are due to the level-of-
difficulty factor. Next, we omit the highest level of difficulty
(mode 4) and recalculate the average error rate. The aver-
age error rate after omitting mode 4 is 1.95 %, 16.06 %, and
20.90 % for mouse, pitch–roll, and pitch–yaw, respectively.
This means there is a decrease in the error rate of 31 %, 43 %,
and 40 % for mouse, pitch–roll, and pitch–yaw, respectively,
after omitting mode 4. At the same time, the influence of
mode 3 and mode 4 was investigated by omitting both of
them in the analysis. We found that the error rate would be
reduced to 33 %, 63 %, and 59 % for mouse, pitch–roll, and
pitch–yaw, respectively. The research regarding arm jitter us-
ing inertial sensor measurement in Noy et al. (2015) demon-
strates that arm jitter ranges from 0.7 to 1.15 Hz. This means
that, to reach the 2 % error rate of an underdamped response,
a person needs around 0.87 to 1.43 s; relatively speaking, the
time to reach the clicked target would be increased due to
the number of pixels in the smaller target’. The tasks with
medium and high levels of difficulty have only 20 and 12
pixels of target width, respectively (see Table 1); the target’s
width is too small and almost the same as the target width in
the studies by Widodo and Matsumaru (2013) and by Myers
et al. (2002). The previous studies used a laser-pointer spot
interface to emulate the mouse’s cursor, which is also prone
to arm jitter, as in our study; in those studies, the subject ex-
perienced difficulty tapping the target. This result strengthens
our suspicion that the difficulty level, such as in mode 3 and
mode 4, is not in accordance with the task of the orientation
sensor as a pointing device.
The results of two gestures succeeded as a substitute for
the movement of a computer mouse. Despite this improve-
ment, there was a significantly high error rate, especially for
the medium- and high-level tasks (mode 3 and mode 4). The
natural jitter of the arm, as mentioned in Noy et al. (2015),
indicates that people need more time to tap the actual clicked
target, such as a point (xc,yc) in Fig. 3b. It is difficult if the
diameter of the target is small, as in mode 3 and mode 4
tasks. In the future, a more rigorous pointing device using
an inertial sensor needs to be combined with a special filter
to dampen the jitter. Filtering methods such as a complemen-
tary filter and Kalman filter are important to consider in order
to improve accuracy and measurement reliability. Other po-
tential improvements for accuracy may be made by taking a
mechanical approach and using, for example, an elbow band,
shoulder support, and/or wrist support to dampen arm jitter.
The qualitative results were concluded in Table 5 and
Fig. 7; we found that Cronbach’s alpha is 0.79 and 0.85
for comfort items and fatigue assessment items, respectively.
This indicates that all the items have a satisfactory level of
reliability as this research is in the early stage, as stated in
Nunnally and Bernstein (1994). The subjects’ opinions show
that pitch–yaw results in less fatigue in the arm, shoulder, and
neck than pitch–roll does (U=89.5, p=0.006; U=107.5,
p=0.029; and U=109.5, p=0.035). Overall, pitch–yaw
results in less fatigue compared to pitch–roll, and this might
be caused by the muscles involved. The pitch and yaw repre-
sent the wrist movement that results from flexion–extension
and radial–ulnar deviation of the wrist, respectively. How-
ever, the roll movement results from forearm movement,
called pronation–supination of the forearm. Now, we would
like to compare only “roll” and “yaw” since both of them
use different gestures in our study. The “roll” range of mo-
tion is only 13◦to the left (pronation) but 53◦to the right
(supination), as stated in Gates et al. (2016). However, we
J. Sens. Sens. Syst., 8, 95–104, 2019 www.j-sens-sens-syst.net/8/95/2019/
R. B. Widodo et al.: Hand-movement gestures to substitute for mouse-cursor placement 103
observed that the range of motion of “yaw” is 28◦to the left
(radial deviation) and 38◦(ulnar deviation) to the right. The
pronation and radial deviation results in the cursor moving to
the left of the monitor display and in the opposite direction
for supination and ulnar deviation. The subject experiences
greater exertion caused by the limitation of the left “roll”
range of motion, which is only 13◦; the subject might use
effort to move above the normal limit of his/her range of
motion to attempt to move the cursor to the far left of the
monitor display. However, for “yaw,” the range of motion is
greater and reaches 28◦. Thus, the pitch–yaw results in less
fatigue than the pitch–roll does.
Through the rating of perceived exertion using the Borg
scale of perceived exertion, another finding revealed that
pitch–roll and pitch–yaw gestures have a strong and posi-
tive correlation with shoulder effort. These gestures have the
same effect of fatigue on the shoulder due to the position of
the forearm during experiments; i.e., the forearm rests on the
chair’s armrest.
6 Conclusions
The aim of the present research was to examine the hand ori-
entation to substitute the computer mouse movement; it was
evaluated based on ISO/TS 9241 part 411: Ergonomics of the
human-system interaction standard. Two pairs of hand orien-
tation candidates were evaluated in terms of pitch–roll and
pitch–yaw, by substituting up–down and left–right mouse-
cursor movements.
Although almost all the scores of pitch–yaw overpass the
scores of pitch–roll, surprisingly, no statistically significant
differences were found in throughput and movement time.
Perhaps the most important finding was that the significant
difference among the index of difficulty is fulfilled. There-
fore, the statistical analysis revealed the index of difficulty
(ID) of very low and low tasks (ID≤4); in our experiment
this is marked by mode 1, and mode 2 is a suitable ID when
using the orientation sensor as a cursor emulation. The sec-
ond major finding was that in terms of fatigue of arm, shoul-
der, and neck, the pitch–yaw gesture has a lower significance
of fatigue than the pitch–roll gesture.
This study provides the first comprehensive assessment of
hand gestures, i.e., pitch–roll and pitch–yaw to emulate a
mouse for human–computer interaction based on ISO 9241-
411 evaluation procedures. The empirical findings in this
study provide a new suggestion for a suitable level of diffi-
culty when using an orientation sensor to emulate the move-
ment of a mouse cursor.
Data availability. The data are available in a shared document.
The link is available at https://drive.google.com/drive/folders/
1doULNPc33mwuklKcJd1DJ-E2YtZ4DEtX?usp=sharing (last ac-
cess: 13 February 2019).
Author contributions. RBW, RMQ, and RS carried out the ex-
periment. RBW wrote the manuscript with support from RS. CW
helped supervise the manuscript and gave advice during the project.
RBW conceived the original idea and performed the analytic calcu-
lations.
Competing interests. The authors declare that they have no con-
flict of interest.
Acknowledgements. A very special thank you goes out to
all students and colleagues in Ma Chung University and alumni
who became subjects in this research and made this research
possible. I am also grateful to Rhesdyan Wicaksono Setiawan and
Septian Amrizal for continued support and patience.
Edited by: Rosario Morello
Reviewed by: three anonymous referees
References
Douglas, S. A., Kirkpatrick, A. E. A. E., and MacKenzie, I. S.:
Testing pointing device performance and user assessment with
the ISO 9241, Part 9 standard, Proc. SIGCHI Conf. Hum. factors
Comput. Syst., 15–20 May 1999, Pittsburgh, Pennsylvania, USA,
15, 215–222, https://doi.org/10.1145/302979.303042, 1999.
Fitts, M.: The Information Capacity of the Human Motor System in
Controlling the Amplitude of Movements, J. Exp. Psychol., 47,
381–391, 1954.
Gates, D. H., Walters, L. S., Cowley, J., Wilken, J. M., and
Resnik, L.: Range of motion requirements for upper-limb ac-
tivities of daily living, Am. J. Occup. Ther., 70, 7001350010,
https://doi.org/10.5014/ajot.2016.015487, 2016.
International Organization for Standardization: Technical Specifi-
cation ISO, Switzerland, 2012.
Kranz, M., Holleis, P., and Schmidt, A.: Embedded interaction: In-
teracting with the internet of things, IEEE Internet Comput., 14,
46–53, https://doi.org/10.1109/MIC.2009.141, 2010.
Lazar, J., Feng, J. H., and Hochheiser, H.: Research Methods in
Human-Computer Interaction, Elsevier Inc., Cambridge, MA
02139, United States, 2017.
Mackenzie, I. S.: Fitts’ Law, in: Handbook of human-computer
interaction, 1, 349–370, Wiley, Waltham, MA 02451, United
States, 2018.
MacKenzie, I. S. and Jusoh, S.: An Evaluation of Two Input Devices
for Remote Pointing, Eng. Human-Computer Interact., 2254,
235–250, https://doi.org/10.1007/3-540-45348-2 2001.
MacKenzie, I. S., Kauppinen, T., and Silfverberg, M.: Ac-
curacy measures for evaluating computer pointing de-
vices, Proc. SIGCHI Conf. Hum. factors Comput. Syst.,
Seattle, Washington, USA, 31 March–5 April, 9–16,
https://doi.org/10.1145/365024.365028, 2001.
Myers, B. A., Bhatnagar, R., Nichols, J., Peck, C. H., Kong, D.,
Miller, R., and Long, A. C.: Interacting at a distance: measuring
the performance of laser pointers and other devices, in: Proceed-
ings of the SIGCHI conference on Human factors in computing
www.j-sens-sens-syst.net/8/95/2019/ J. Sens. Sens. Syst., 8, 95–104, 2019
104 R. B. Widodo et al.: Hand-movement gestures to substitute for mouse-cursor placement
systems Changing our world, changing ourselves, 20–25 April
2002, Minneapolis, Minnesota, USA, 33, 2002.
Natapov, D., Castellucci, S. J., and MacKenzie, I. S.: ISO 9241-
9 Evaluation of Video Game Controllers, Proc. Graph. Interface
Conf., 25–27 May 2009, Kelowna, British Columbia, Canada,
223–230, 2009.
Nelson, D. L., Mitchell, M. A., Groszewski, P. G., Pennick, S. L.,
and Manske, P. R.: Wrist Range of Motion in Activities of Daily
Living, in: Advances in the Biomechanics of the Hand and Wrist,
NATO ASI Series (Series A: Life Sciences), edited by: Schuind
F., An, K. N., Cooney, W. P., and Garcia-Elias, M., 256 pp.,
Springer, Boston, MA, 1994.
Norman, K. L. and Norman, K. D.: Comparison of Relative Versus
Absolute Pointing Devices, Human-Computer Interact. Lab, 11-
5-10, 1–17, 2010.
Noy, L., Alon, U., and Friedman, J.: Corrective jitter motion shows
similar individual frequencies for the arm and the finger, Exp.
Brain Res., 233, 1307–1320, https://doi.org/10.1007/s00221-
015-4204-1, 2015.
Nunnally, J. C. and Bernstein, I. H.: Psychometric Theory,
McGraw-Hill, USA, 1994.
Oh, J.-Y. and Stuerzlinger, W.: Laser Pointers as Col-
laborative Pointing Devices, Proc. Graph. Interface
Conf., 27–29 May, Calgary, Alberta, Canada, 141–149,
https://doi.org/10.20380/GI2002.17, 2002.
Perng, J. K., Fisher, B., Hollar, S., and Pister, K. S. J.: Acceleration
sensing glove (ASG), Wearable Computers, 178–180, 18–19 Oc-
tober, San Francisco, CA, USA, 2002.
Soukoreff, R. W. and MacKenzie, I. S.: Towards a standard for
pointing device evaluation, perspectives on 27 years of Fitts’
law research in HCI, Int. J. Hum. Comput. Stud., 61, 751–789,
https://doi.org/10.1016/j.ijhcs.2004.09.001, 2004.
Sturman, D. J. and Zeltzer, D.: A survey of glove-based input, IEEE
Comput. Graph. Appl., 14, 30–39, 1994.
Widodo, R. B. and Matsumaru, T.: Measuring the performance of
laser spot clicking techniques, in: 2013 IEEE International Con-
ference on Robotics and Biomimetics, ROBIO 2013, 12–14 De-
cember, Shenzhen, China, 2013.
Zimmerman, T. G. and Lanier, J. Z.: Computer data entry and
manipulation apparatus and method, available at: https://www.
google.com/patents/US4988981 (last access: 1 August 2018),
1989.
Zimmerman, T. G., Lanier, J., Blanchard, C., Bryson, S., and
Harvill, Y.: A hand gesture interface device, ACM SIGCHI Bull.,
17, 189–192, https://doi.org/10.1145/30851.275628, 1986.
J. Sens. Sens. Syst., 8, 95–104, 2019 www.j-sens-sens-syst.net/8/95/2019/
