Conference PaperPDF Available

# Consumed Endurance: A Metric to Quantify Arm Fatigue of Mid-Air Interactions

Authors:

## Abstract and Figures

Mid-air interactions are prone to fatigue and lead to a feeling of heaviness in the upper limbs, a condition casually termed as the gorilla-arm effect. Designers have often associated limitations of their mid-air interactions with arm fatigue, but do not possess a quantitative method to assess and therefore mitigate it. In this paper we propose a novel metric, Consumed Endurance (CE), derived from the biomechanical structure of the upper arm and aimed at characterizing the gorilla-arm effect. We present a method to capture CE in a non-intrusive manner using an off-the-shelf camera-based skeleton tracking system, and demonstrate that CE correlates strongly with the Borg CR10 scale of perceived exertion. We show how designers can use CE as a complementary metric for evaluating existing and designing novel mid-air interactions, including tasks with repetitive input such as mid-air text-entry. Finally, we propose a series of guidelines for the design of fatigue-efficient mid-air interfaces. More Information: http://hci.cs.umanitoba.ca/projects-and-research/details/ce
Content may be subject to copyright.
Consumed Endurance:
A Metric to Quantify Arm Fatigue of Mid-Air Interactions
Juan David Hincapié-Ramos, Xiang Guo, Paymahn Moghadasian, Pourang Irani
University of Manitoba
{hincapjd, umguo62, umpaymah, irani}@cc.umanitoba.ca
ABSTRACT
Mid-air interactions are prone to fatigue and lead to a feeling
of heaviness in the upper limbs, a condition casually termed
as the gorilla-arm effect. Designers have often associated
limitations of their mid-air interactions with arm fatigue, but
do not possess a quantitative method to assess and therefore
mitigate it. In this paper we propose a novel metric,
Consumed Endurance (CE), derived from the biomechanical
structure of the upper arm and aimed at characterizing the
gorilla-arm effect. We present a method to capture CE in a
non-intrusive manner using an off-the-shelf camera-based
skeleton tracking system, and demonstrate that CE correlates
strongly with the Borg CR10 scale of perceived exertion. We
show how designers can use CE as a complementary metric
for evaluating existing and designing novel mid-air
interactions, including tasks with repetitive input such as
mid-air text-entry. Finally, we propose a series of guidelines
for the design of fatigue-efficient mid-air interfaces.
Author Keywords
Gorilla-arm, mid-air interactions, mid-air text-entry,
endurance, consumed endurance, SEATO mid-air keyboard.
ACM Classification Keywords
H.5.2. Information interfaces and presentation (e.g., HCI):
Evaluation/Methodology.
INTRODUCTION
The proliferation of low-cost gestural tracking systems has
warranted the investigation of mid-air interaction as a new
class of natural user interface (NUI) [20, 19]. This style of
interaction has shown particular value in sterile medical
rooms [7, 25], in educational settings [12], and in gaming
environments [22]. Nonetheless, users engaged with mid-air
input often report fatigue and a feeling of heaviness in the
arm [9, 20], a condition coined as the gorilla-arm effect [9].
Gorilla-arm was first reported with the introduction of touch-
screens, and was one reason for the early dismissal of such
systems [1, 2]. Ignoring this factor in the design of mid-air
interactions can also lead to the demise of this form of NUI.
Current approaches to assess arm fatigue include obtrusive
measurements of bodily variables (heart-rate [30], oxygen
level [16] or EMG [28, 32]) or the collection of subjective
assessments (Borg [8], NASA-TLX [21] or Likert ratings).
However, these methods have limited practical value for
evaluating mid-air interactions as they require specialized
equipment or have high variance. We propose a method for
quantitatively characterizing the gorilla-arm effect based on
the concept of endurance
(Figure 1 and equation 1) [29,
17]. Endurance is the amount
of time a muscle can maintain
a given contraction level
before needing rest. Using a
skeleton-tracking system we
capture users’ arm motions
and compute endurance for the
shoulder muscles. Consumed
Endurance (CE), our novel
metric, is the ratio of the
interaction time and the
computed endurance time.
We validate CE against fatigue ratings as obtained using the
Borg CR10 scale of perceived exertion. Further, we
demonstrate CE’s value as a complementary metric for
evaluating mid-air interactions. For mid-air pointing and
selection on a 2D plane, we used CE to identify the most
suitable interaction parameters, such as arm extension, plane
location and plane size. For example, users consumed the
least amount of endurance when the arm was bent and
operating on the interaction plane located midway between
the shoulder and the waist. Dwell selections have the lowest
CE for single hand interactions. We also demonstrate the
value of using CE to inform the design of an endurance-
efficient text-entry layout, SEATO (Figure 7-left). Users
entering text with SEATO had lower CE than with
QWERTY without compromising text-entry speed. Finally,
we describe how our results inform the design of mid-air
menus and other interactive systems. CE and other fatigue-
related metrics are publicly available in a software toolkit
(http://hci.cs.umanitoba.ca/projects-and-research/details/ce).
Our contributions include: 1) CE, a metric for characterizing
shoulder fatigue or gorilla-arm effects resulting from mid-air
interactions; 2) the use of CE to inform the choice of various
mid-air interaction parameters; 3) an endurance-efficient
mid-air text-entry layout, SEATO; and 4) guidelines for
designing endurance-efficient mid-air interactions.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not
made or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for components
of this work owned by others than ACM must be honored. Abstracting with
credit is permitted. To copy otherwise, or republish, to post on servers or to
redistribute to lists, requires prior specific permission and/or a fee. Request
permissions from permissions@acm.org.
CHI 2014, April 26 - May 01 2014, Toronto, ON, Canada
Copyright 2014 ACM 978-1-4503-2473-1/14/04…15.00. http://dx.doi.org/10.1145/2556288.2557130 Figure 1. Endurance metrics for the mid-air interactions include Arm Strength, Endurance and Consumed Endurance (CE). RELATED WORK This section gives a summary of the existing qualitative and quantitative tools for assessing muscular fatigue. Qualitative Assessment of Fatigue Qualitative methods for assessing arm fatigue include Likert- scale questions [9], the NASA Task Load Index (NASA- TLX) [21] and the Borg RPE and CR10 scales [8]. Likert- scales reduce the users’ subjective ratings to whether or not they experienced fatigue in an interaction. The NASA-TLX questionnaire captures workload along categories such as Physical Demand and Effort [21], rated on a 20-point scale. However, as pointed by Bustamante and Spain, the TLX lacks “scalar invariance, thereby biasing the estimation of mean scores and making the examination of mean differences misleading [10]. Subjectivity is further reinforced as each participant can weigh the various TLX categories differently. The Borg CR10 scale [8] is tailored to physical exertion. It maps numeric ratings to carefully chosen verbal cues and provides scalar invariance. While qualitative assessments provide a coarse estimation of fatigue, a finer characterization is required, particularly for repetitive tasks. Subjective assessments cannot give an account of the small yet significant differences, and are prone to confounding variables such as the participant’s fitness, comfort level or general state of mind. Complementing such methods with objective metrics of fatigue can provide a more holistic handle over gorilla-arm effects. Objective Assessment of Fatigue Fields such as the sports sciences, ergonomics and physiology have long studied the relationship between muscular exertion and fatigue. Their methods range from external measurements such as monitoring muscle swelling [5], muscle oxygenation [16], heart rate (mentioned in [6]), and blood flow and pressure [30]; to invasive techniques such as measuring the intra-arterial levels of lactate and potassium [30]. Morris et al. [26] show a strong relationship between different fatigue-related factors obtained externally (rate of force exertion and relaxation) and those captured invasively (lactate, oxygen level), indicating that the former are highly reliable measures of fatigue. However, these approaches require specialized equipment (such as EMG and NIRS devices, dynamometers or exoskeletons) or are invasive, limiting how users engage with mid-air systems, and thus impractical for the design of interactive systems. Unlike previous approaches [5, 16, 30, 4], our method provides an objective and non-invasive metric of shoulder fatigue, calculated using a low-cost gestural tracking system. FATIGUE IN MID-AIR INTERACTIONS Gorilla-arm is a manifestation of fatigue in the arm muscles. Fatigue is defined as the ability to maintain a given muscular contraction level [17] and depends on the amount of blood flow, and thus oxygen, that reaches the muscle cells. A contracted muscle hardens arteries and restricts blood flow. With low levels of oxygen muscle cells switch their energy source from aerobic to glycolytic metabolism. Given the limited amounts of stored glucose, the muscle cells can produce energy and maintain the contraction only for a short period of time. Fatigue occurs when this energy is used up. A well accepted result in human physiology is Rohmert’s study of the impact of fatigue on endurance: the maximum amount of time that a muscle can maintain a contraction level before needing rest [29]. Figure 1 illustrates Rohmert’s formulation of endurance E(F) as a function of the value of force applied (F) in relation to the maximum force (Fmax) of the muscle (see equation 1). An important observation is that equation 1 is asymptotic at 15% of the maximum force, meaning that forces exerted below that level could be sustained for long time periods. The presence of fatigue in mid-air interactions [9, 20] suggests that current interaction techniques require arm forces above the 15% mark.     We derive metrics based on this mathematical formulation of endurance to study and guide the design of mid-air interact- tions. A higher endurance time for an interaction implies that it triggers lower amounts of fatigue in users and thus allows for longer engagements with a system. Ultimately, this should foster a broader adoption of such technologies. CHARACTERIZING SHOULDER ENDURANCE This section details a non-intrusive method for determining endurance for mid-air interactions. Although multiple body parts are involved in mid-air arm interactions we focus on the shoulder joint as it largely dominates the forces required for moving the arm. Measuring endurance using Rohmert’s formulation requires capturing the two variables in equation 1: the maximum force of the shoulder (Fmax), and the force acting on the shoulder at a given time (F). For the first variable, we rely on values of maximum force as determined previously by others [31, 14]. For the second variable, we use a biomechanical model of the arm where it is represented as a compound rotational system encompassing the upper arm, the forearm and the hand, and with its pivot on the shoulder joint (see Figure 2-left). This compound system can be simplified as a single-part system where all forces are applied at the arm’s center of mass (CoM) (see Figure 2-right). Rohmert’s formulation assumes that and  are com- parable, that is, they are applied at the same distance from the shoulder joint. However, given that  is determined at the elbow [31] and at the CoM, these two forces are not comparable. A solution is to express all forces in terms of Figure 2. Left: The primary forces acting on the arm. Right: The forces aggregated at the CoM. torque. Appendix A shows the relationship between   and  ; and therefore endurance in terms of torque is:       Equation 3 formalizes the sum of torques ( ) acting on the system at a given time. The first torque pulls the arm downwards and is due to the interaction of gravity (g) and the mass of the arm (m) at the CoM (distance r from the shoulder joint). The second torque is provided by the shoulder muscles and it compensates for the effects of gravity and moves the arm. The final torque is due to the arm’s inertia and its angular acceleration (), and represents the tendency of the arm to maintain its rotational movement once in motion.        When the arm is static the and α components are equal to zero, and therefore    . That is, the shoulder has to match the gravity torque. Conversely, when the arm is in rotation the resulting and  are not equal to zero. The next section shows how to calculate . Measuring Torque at the Shoulder Joint By following the process described in Appendix B, a skeleton tracking system can compute the CoM location r at time t. Tracking the CoM allows us to determine its velocity and acceleration. Knowing the arm mass it is possible to determine the force acting at the CoM:      Thus, the total torque of the system can be expressed as1:    Equaling equations 3 and 5, we can derive the actual torque exerted by the shoulder muscles at time t:              The final term in equation 6, , represents the tendency of a rotating body to continue its rotational movement. Angular acceleration ( ) at time t can be calculated as   . See Appendix C for a detailed account on how to calculate the moment of inertia (I) of a moving arm. Maximum Torque of the Shoulder Tan et al. studied the force output range at different joints in the body [31], and established the maximum controllable force of the shoulder applied at the elbow to be 87.2 N for females and 101.6 N for males. In a similar study, Edmunds et al. found that such maximum force varies slightly according to the movement direction (x,y,z), with an approximate 100 N as the most common value [14]. We use Tan et al.’s maximum force to get the maximum torque  1 The cross product in equation 5 indicates that only the forces tangential to the rotation axis are relevant for the torque. When (  , then    . of the shoulder muscles; based on their experimental set-up, we remove the effect of the arm weight from the max torque. Endurance Metrics Based on the above, we can calculate the following metrics: Strength (S): The ratio between the average torque applied in the interaction and the maximum torque. This metric corresponds to the     term of equation 5. Endurance (E): The time, in seconds, the participant could sustain such interaction before needing to rest the arm. Consumed Endurance (CE): The ratio of the interaction time and the computed endurance time (see equation 7). We interpret CE as the percentage of the energy used or as the amount of fatigue.   ENDURANCE METRICS IMPLEMENTATION We used Microsoft’s .NET to implement the CE equations. We used the Microsoft Kinect, an off-the-shelf human skeleton tracking system, to capture the arm joints needed. We applied a noise reduction filter averaging the last 10 skeleton frames. The system operates at approximately 32 frames/sec, which is sufficient to support the small delta () assumption made for equation 5. We used the 50th percentile male or female values2 for weight, length, center of mass, and inertia of the upper arm, lower arm, and hand (a standard approach [18]) as compiled by Freivalds [17]. Using these values and the captured skeleton, our system determines the arm’s CoM and normalizes it to the 50th percentile (we use the skeleton’s upper arm’s length as a frame of reference). The system calculates all metrics from the normalized CoM. VALIDATING THE CONSUMED ENDURANCE METRIC In this section, we assess the validity of CE as a measure of fatigue by 1) comparing CE measurements to Borg CR10 ratings [8] and 2) analyzing the effects of gender-specific constants. The Borg CR10 scale provides a ratio-scale measure of physical exertion which values are matched to verbal anchors. Borg CR10 values range from 0 to 10, where 0 corresponds to "Nothing At All" and 10 to "Very Very Hard (Maximal)". Electromyograms (EMG) measure muscle cell activations and is the bases for several objective metrics (see [32]). Studies have shown that Borg CR10 ratings and EMG-based metrics for shoulder muscles strongly correlate and therefore either method can be used to assess shoulder fatigue [28, 32]. More importantly, they showed that Borg CR10 can be more reliable than EMG metrics. At low levels of physical exertion (such as the ones in optimized mid-air gestures) EMG metrics are not valid fatigue indicators [27]. Also, EMG metrics have lower repeatability than Borg CR10 [13] and their validity is task-dependent [15]. We asked 16 participants (8 female) to hold their dominant arm at different angles from the vertical axis (90°, 60°, 30°, 2 An alternative method measures the length of a volunteer’s upper limbs and uses this data to retrieve the average corresponding weights based on data provided by the Visible Human Project [3]. and at rest, controlled within ±2 degrees) and for different periods of time (15, 30, 45 and 60 seconds). We used a Latin- square design on angle with time and gender as random factors. We captured 3 trials per condition per participant for a total of 16×16×3 = 768 CE measurements. Participants rated each condition (3 trials) for a total of 16×16 = 256 Borg CR10 ratings. Figure 3 shows the overall results. We used linear regression analysis and the Mixed Factors ANOVA test with angle and time as within-subject factors and gender as a between-subjects factor. We used Bonferroni correction for post-hoc tests. Linear regression analysis between CE and Borg CR10 ratings (see Figure 3-bottom) revealed a significant correlation (F1 = 1902.722, p < 0.001) with R2 = 0.716. For the ANOVA, normality tests showed a normal distribution for all conditions (p < 0.001). Results showed a main effect in angle for CE (F3,42 = 8543.719, p < 0.001) and Borg CR10 (F3,42 =89.806, p < 0.001); a main effect in time for CE (F3,42 = 23323.431, p < 0.001) and Borg CR10 (F3,42 = 68.460, p < 0.001); and no main effect in gender for CE (F1,14 = 0.951, p = 0.346) and Borg CR10 (F1,14 = 2.379, p = 0.145). Results showed interaction effects for angle × time in CE (F9,126 = 4874.636, p < 0.001) and Borg CR10 (F9,126 = 10.301, p < 0.05). Post-hoc analysis of angle and time showed significant differences between all conditions for both CE and Borg CR10. Our results show that CE and Borg CR10 ratings present a “very strong to perfect association” (R = 0.846) where the value of CE is used to predict 72% of the variability of Borg CR10 ratings (R2 = 0.716). The remaining 28% can be explained by differences in fitness level of the participants and the subjective nature of the Borg CR10 scale (affected by factors such as tiredness, comfort, and general state of mind). Moreover, CE and Borg CR10 ratings are equally capable of yielding significant differences for changes in angle and time (main effects on both factors). Furthermore, results show that CE is gender neutral, suggesting that the different sets of constants for arm metrics (weights, lengths, and max force) do not affect CE. In other words, given Borg CR10’s correlation to objective measurements of fatigue (such as EMG), these results show that CE is a valid objective fatigue metric for the shoulder muscles. CONSUMED ENDURANCE AS AN ANALYTICAL TOOL We first demonstrate the use of our model to evaluate different mid-air interaction factors. In the first experiment we investigate the effects of different plane locations and arm extensions on CE. In the second experiment we study the effects of plane size and selection method on CE. Experiment 1: Plane Location and Arm Extension Methods Apparatus The system ran on a Windows 7 PC connected to a 4×2.3 meters projector screen with a resolution of 1366× 768 pixels (1 pixel = 3 mm) and a Microsoft Kinect. The Kinect was in front of the screen and 1 meter above the floor; participants stood 3.3 meters from the screen. We used the same set-up for all experiments. Subjects 12 participants (3 female) volunteered, ages 18-40 (mean 26), right handed. All participants had previous experience (mean: 0.6 years) with mid-air interaction systems (Wii, Kinect, etc.) and were familiar with mid-air selection. Task Participants had to select 20 fixed targets (one after the other) in a square 2D plane (35 cm sides) by moving the cursor (small red circle) with their right arm from the current position to the target. Participants were asked to select using a mouse button held in the left hand. We relegated selection to a mouse to avoid any overhead. All targets were solid squares organized in a 6×6 matrix (black border, white background). Upon selection, the target was highlighted in red, and the next target turned blue. The task finished when the participant selected the 20 targets in the order presented. The 20 targets were randomly distributed across all positions and no position was repeated. A landing error was marked when the user left the target before selecting it. This measure describes the level of control a user has over the cursor, i.e., how precise the movements are. Design Independent variables were plane location and arm extension (see Figure 4). We used a 2×2 within-subject design to compare CE in each condition. We considered two 2D plane locations relative to the body (all planes aligned to the right side of the shoulder): Shoulder: is a vertical plane with the vertical center at the shoulder joint. Center: is a vertical plane with the vertical center located halfway between the shoulder and the waist. We considered two arm extensions: Extended and Bent. The system detects the arm as extended when the hand is at least Figure 4. Two location planes (shoulder, center) with two arm configurations (extended, bent) Figure 3. Top Both CE and Borg CR10 present a main effect for angle and time but not for gender . Bottom Linear correlation between Borg CR10 and CE show a strong correlation (R = 0.846) where CE predicts 72% of the variability in Borg CR10 ratings (R2 = 0.716). 35 cm away from the body, and as bent when the hand is 35 cm or closer to the body plane. The system ignores the arm (removes the cursor from the screen) when it is under or beyond these limits, forcing the participant to stretch out or bend it as necessary. We settled on these measures after iterative pilot testing. Participants were trained with each condition after the experimenter demonstrated the task. With a total of 2×2 = 4 conditions and 4 trials per condition, we registered 2×2×4 = 16 trials per participant, or 192 trials in total (each trial consisted of 20 selections). Participants had a mandatory 3 minute break between conditions. All participants completed the experiment in one session lasting approximately 30 minutes. The trials were counter-balanced using a Latin- square on plane location and arm extension. Measures We collected values for CE, completion time, and landing error rate. Participants filled a Borg CR10 rate scale questionnaire after each condition. Results None of the dependent variables comply with the ANOVA assumptions (normality and equal variances) and therefore we applied the Aligned Rank Transform for nonparametric factorial analysis [33] with a Bonferroni correction for pairwise comparisons. Figure 5 presents the results. Consumed Endurance (CE) Results showed a main effect of plane location (F1,11 = 102.249, p < 0.001) and arm extension (F1,11 = 86.959, p < 0.001). There were not significant interaction effects for plane location × arm extension (p = 0.637). CE was lowest in the center plane location with an average of 27.23% (standard deviation or std = 15.33) and the bent arm extension at 23.44% (std = 13.45). Borg CR10 Results showed main effects of plane location (F1,11 = 7.111, p < 0.05) and arm extension (F1,11 = 21.082, p < .001). There were no significant interaction effects for plane location × arm extension (p = 0.134). Borg CR10 was lowest on the center plane location at 3.00 (std = 1.87) and the bent arm extension at 2.75 (std = 1.51). Completion Time Results did not show a main effect of plane location (p = .092) or arm extension (p = .223). There were no significant interaction effects for plane location × arm extension (p = .893). Average completion time was 45 seconds (std = 13.81). Landing Error Rate Results did not show a main effect of arm extension (p = .619) or plane location (p = .357). There were no significant interaction effects for plane location × arm extension (p = .220). Average landing error was 0.93 (std = 2.22). Discussion We first observe that both CE and Borg CR10 yield similar main and interaction effects, highlighting CE’s capacity to reveal the same fatigue effects as Borg CR10. On the other hand, differences in completion times and error rates are not significant. This is an important observation because it suggests that differences in fatigue emerge even when other measurements are flat. Therefore, completion time and landing error rate were limited in determining the optimal combination of plane location and arm extension for mid-air input, given our conditions. With equivalent accuracy to Borg CR10, a system can calculate CE unobtrusively and in real-time by simply tracking arm movements. Interactions in the shoulder plane consume more endurance as the arm is higher up from its resting position. Similarly, interactions with arm extended also consumed more endurance as the center-of-mass is further extended from the body, thus requiring a higher torque. Interactions with the bent arm consumed the least endurance, the lowest being in the center plane at 15.55%. We select the center + bent condition as the optimal area for interaction and use it in the next experiments which evaluate CE for other factors. Experiment 2 - Plane Size and Selection Method The goal of this experiment is to examine the effect of larger arm movements and different selection methods on CE. Methods Subjects 12 participants (4 female), ages 18-40 (mean 22.3), volunteered. All participants were right handed and half had no experience with in air interactions. Task & Design The experimental task was the same as in experiment one. The independent variables were plane size, and selection method. We used a 2×4 within-subject design. We tested two plane sizes: 35x35 cm and 25x25 cm. Selection method indicates the mechanism by which participants select a target. We designed four methods: Click: as in experiment one; participants click a mouse held in their left hand. Swipe: is a quick horizontal arm movement to both sides at min 50 cm/sec and for a movement of at least 15 cm. Dwell: participants highlight a target for 1.5 seconds (threshold determined through iterative pilot testing). Second Hand: participants move the left arm 20cms away from its resting position (i.e., from the hips). Figure 5. Consumed Endurance, Borg CR10 ratings, completion time and landing error rate for experiment one. The experimenter demonstrated each selection method and participants had an initial training with each condition, testing each selection method until they had control over it. The experiment had a total of 2×4 = 8 conditions and each condition had 3 trials, yielding 2×4×3 = 24 trials per participant, or 288 trials in total. All participants completed the experiment within approximately 45 minutes. The trials were counter-balanced with a Latin-square approach on selection method and plane size appeared in a random order. Measures We collected values for CE, completion time, and landing error rate. Participants filled in a Borg CR10 scale after each condition. Results We used the same statistical tests as in experiment one (ART ANOVA). Figure 6 shows an overview of the results. Consumed Endurance (CE) Results showed a main effect of selection method (F3,33 = 19.612, p < 0.001) and plane size (F1,11 = 16.165, p = 0.002). There were no significant interaction effects for plane size × selection method (p = 0.323). Post-hoc pair-wise comparisons on selection method yielded significant (p < 0.04) differences between all pairs except between dwell and swipe, and dwell and second hand. In general, CE was lowest for click at 8.18% (std = 9.06) and the small plane at 10.80% (std = 12.00). Borg CR10 Results showed a main effect of selection method (F3,33 = 8.425, p < 0.001) but not for plane size (p = 0.837). Results did not show interaction effects for plane size × selection method (p = 0.586). Post-hoc analysis on selection method yielded significant differences for all pairs (p < 0.04) except second hand and dwell. In general, Borg CR10 was lowest for click at 1.188 (std = 0.845). Completion Time Results showed main effects of selection method (F3,33 = 58.076, p < 0.001) and plane size (F1,11 = 5.143, p = 0.044). Analysis also revealed interactions effects for plane size × selection method (F3,33 = 3.167, p = 0.037). Post-hoc analysis on selection method revealed significant differences (p < 0.017) between all pairs. Click was the fastest selection method at 40.76 seconds (std = 10.62). The larger plane had a lower mean completion time of 54.52 seconds (std = 17.92). Landing Error Rate Results showed a main effect of plane size and selection method (all p<0.003) on error rate with F1,11 = 17.112 and F3,33 = 22.167 respectively. There were no interaction effects between plane size and selection method (p = 0.131). Post-hoc pair-wise analysis showed a significant difference between swipe and all other selection methods (all p<0.001). Error rate was lowest for dwell at 0.66 (std = 0.59) and the big plane at 0.83 (std = 0.86). Discussion This experiment highlights the capacity of CE for uncovering differences where subjective ratings cannot. A larger plane requires stretching and lifting the arm which clearly results in increased effort. CE reveals a significant difference between plane sizes which Borg CR10 hides due to the high variance and small size of the sample. Selection methods which do not require movement of the selecting hand perform best across all metrics. Swipe, which performs worst, sees its CE increased due to the greater amount of movement it requires due to the gesture design and to tracking errors. Tracking errors, more noticeable in the small plane, are due to problems of distinguishing the arm from the body and to follow the hand back (such as in swipe). This results in poorly controlled gestures which miss the target, leading to repetition, and therefore higher completion time and CE. A better tracking technology would increase the controllability of the gesture, reducing the need to correct and flattening error rates and their effect on CE. The best plane in terms of CE is the small plane. However, the best performance in terms of completion time and landing error rate is the big plane. A designer may have to choose the larger plane to reduce errors which could quickly lead to fatigue and a bad user experience. As expected, Click outperforms all other selection methods in terms of CE and therefore it should be used when possible, else Dwell and Second Hand use similarly little CE. ENDURANCE AS A DESIGN PARAMETER The previous experiments demonstrate the use of CE as a tool to assess various design alternatives. In this section we use another endurance-related metric, strength (defined earlier), as a design parameter for a mid-air text-entry system. We choose text-entry because it is a common task and one that involves repetition. From our previous experiments we know that: (a) interactions consumed the least endurance when they occur on the center plane with a bent arm; (b) a 25x25 cm plane size consumes lower CE; and (c) for single hand situations dwell selections are recommended. In this section we propose a new text-entry layout optimized for such a set of interaction parameters (see Figure 7). We collected data from 4 participants (all male) who held the cursor at each position of the 6x6 grid for 10 seconds (center Figure 6. Consumed Endurance, Borg CR10, completion time and error rate for experiment two. plane, arm bent, 25x25 cm plane size). We recorded arm strength (   ) for each cell because, unlike endurance and CE, it is not affected by the asymptote of equation 2 and thus reveals differences even when the physical effort is low. Figure 7-left shows the resulting heat- map for strength throughout the grid: on average. The cell on the lower-left corner requires 9.2% of the maximum strength, while the cell on the upper-right corner required 20.46%. All bluish cells in Figure 7-left are below the 15% threshold. Figure 7-right shows the resulting SEATO text-entry layout for mid-air interactions. We obtained the SEATO layout by mapping the cells with the lowest strength demands to the characters with the highest probability in the English language, ideally resulting in a less physically demanding interaction than with other text-entry layouts like QWERTY. Experiment 3 - Text Entry Layout In this experiment we compare the SEATO and QWERTY layouts in terms of CE, text-entry speed and error rate. Methods Subjects 12 participants (5 female), ages 18-40 (mean 24), volunteered. All participants were right handed and all but three had previous experience with mid-air interactions. Task Participants had to type a sentence that was shown on the screen. For typing a character participants had to move the cursor to the cell with the character and use the dwell gesture for selection. We selected a list of 53 sentences between 19 and 23 characters long from MacKenzie et al.’s set [24]. When the wrong character was selected the system would not allow any more typing until the wrong character is deleted by selecting the DEL key; this is counted as an entry error. The task finishes when the correct phrase is typed in and the participant selects the ENTER key. Design The independent variable is layout: SEATO and QWERTY. We used a within-subjects design to compare CE between layouts. Participants had an initial training with the SEATO layout and with the mechanics of selecting a letter. Participants were trained by typing sample sentences with both layouts, terminating a phrase with the ENTER key. There were 2 conditions, and each condition had a total of 4 blocks and 3 trials per block, yielding 2×4×3 = 24 trials per participant, or 288 trials in total. Measures We measured CE, words per minute (WPM), and error rate. Users filled a Borg CR10 scale after each block. Results We used the same analysis tools as for experiments one and two. Figure 8 shows an overview of the results. Consumed Endurance (CE) Results showed a main effect for layout (F1,11 = 51.332, p < 0.001) and block (F3,33 = 14.285, p < 0.001). Results did not show significant interaction effects (p = 0.174). Post-hoc analysis showed a significant difference between the first and second block (p = 0.003). SEATO had a lower average CE compared to QWERTY at 6.43% (std = 11.08). Borg CR10 Results did not show a main effect of layout (p = 0.258) or block (p = 0.257). Interactions effects were also not significant (p = 0.300). Words Per Minute Results showed no main effect of layout on words-per-minute (WPM) (p = 0.124), but a main effect for block on WPM (F3,33 = 6.120, p=0.002). Results showed no layout × block interaction effect (p = 0.581). Post-hoc analysis revealed significant differences between the first and third (p < 0.02) and last (p = 0.003) blocks. The last block had the highest WPM at 4.55 (std = 1.26). Typing Error Results showed a main effect of layout (F1, 11 = 15.868, p = 0.002) and block (F3,33 = 8.572, p < 0.001), but no interaction effects (p = 0.378). Post-hoc analysis revealed the first block to be significantly different from all other blocks (p < 0.022). The last block had the lowest mean error rate at 0.05 (sd = 0.07) and the Qwerty layout had a lower mean error rate at 0.06 (sd = 0.1). Discussion Our data shows that layout has an effect on CE, with our pro- posed SEATO layout consuming significantly less endurance than QWERTY (a quarter), at no cost in terms of words-per- minute and only slightly higher error rate. Moreover, results show no significant difference in the Borg CR10 rankings, outlining the added value of our metric for situations where differences do not surface with subjective ratings. Finally, the similar typing speed we observed reinforces the notion that designers could also look at other factors beyond interaction time for making interface choices. Figure 7. Left: Heat-map of strength. Cells in blue require the least strength and those in red require the most. Right: SEATO key layout based on character probability in the English language and strength. Figure 8. Consumed Endurance, Borg CR10, words per minute and error rate for experiment three. GENERAL DISCUSSION We discuss our findings in light of mid-air interactions. Applications of endurance-based metrics Our results demonstrate the value of adopting CE as a complementary guide for evaluating the impact of mid-air input parameters like plane size or selection mechanisms on fatigue. In a similar vein, CE can be used to evaluate alternative sets of mid-air gestures for controlling an interactive system (as in Barclay et al. [6]). Aside from designing endurance-efficient text-entry layouts, our metrics can be used in the design of mid-air menus, document navigation controls and arm gestures. Based on the heat-map shown in Figure 7-left, when selecting a menu with a pointer, the most frequently used menu items should be in the lower left corner (or lower right if interacting with the left arm): buttons on the top or the right side of the interaction plane should be avoided (Figure 9-middle). Similarly, navigation controls, if used frequently, should appear in those regions marked in blue in Figure 7 (see image in color). Our results also suggest that when possible mid-air gestural interactions should consider relative movements rather than absolute ones that have fixed positions in the air (Figure 9- right). In this manner, gestures could take place in regions of least effort. For example, gesturing the letter B could take place by allowing users to start the gesture by moving the arm from its rest position without having to lift it up to an absolute start position of engagement. Finally, to control for arm position (bent or extended) in our experiments, our application did not allow the user to operate outside a certain distance region. While we do not advise enforcing such restrictions in mid-air interactions, application designers could include guidelines to users, in the form of a quick image or video clip, to reduce fatigue during use. While we demonstrated the use of our metric to minimize CE, other applications may choose to increase it or adjust it dynamically. For example, mid-air gaming applications could introduce CE for better control over game balancing. Dynamic game-balancing is possible by gradually shifting the need for selecting or interacting with different positions within the interaction plane or by requiring the user to use different arm positions (switching between extended and bent). This could have direct benefits in virtual therapy applications where movements can become increasingly demanding as the patient’s upper limb functions improve, or conversely if the patient’s progress is slow. To support the different explorations and usages of CE, researchers and designers of mid-air interfaces can download our implementation here http://hci.cs.umanitoba.ca/projects- and-research/details/ce. Our findings complement existing guidelines Our results, obtained with a view on reducing fatigue, empirically confirm and further complement human interface guidelines proposed by some manufacturers of gestural tracking systems (for details see: www.microsoft.com/en- us/kinectforwindows/develop/learn.aspx). Such guidelines mainly provide designers with parameters for optimal tracking efficiency. For example, the Kinect guidelines suggest using Dwell to avoid inadvertent selections (page 55, in above document) and recommend that gestural systems allow seamless hand switching or provide alternative gesture sets to reduce fatigue (page 22). Our results further provide specific insight on how such alternative gesture sets should be designed to reduce effort, such as for text-entry. Our findings in light of previous results Our results justify the fatigue-related findings of prior work. Harrison et al.’s participants preferred a position with “elbows tucked in, hands held front, and palms up” [20]. In light of our results that position seems natural as it closely resembles the center bent arm position. Similarly Boring et al.’s participants who did not move their whole arms and relied on tilt reported less fatigue [9]. This result also seems natural as the arm was not fully extended and thus all of its mass did not have to be moved by the shoulder muscles. Finally, our results can explain why Cockburn et al.’s ray- casting technique was ranked the least physically demanding [11], as the upper-arm was held in a resting position. Our results can also be used to re-consider existing interactions. For example, Li et al.’s VirtualShelves introduce mobile interactions across the horizontal and vertical axis in front of the user [23]. As these movements require full arm extension their CE is high. An endurance-efficient alternative can use only movements of the forearm, with the upper arm in rest position, i.e. with bent elbow. Similarly, Cockburn et al.’s 2D plane technique can improve in terms of CE by fixing their interaction plane at the center plane location, i.e. between the hip and shoulders, and by reducing the size of the plane to one where less arm extension is needed. Lessons learned We take away these lessons from our initial exploration: The center + bent arm position for selections on a 2D plane is the least tiring of all positions we tested. The regions at the bottom of the interaction plane improve CE. Interacting in the lowest possible region should be dictated by the tracking systems accuracy. In the center bent arm position a bigger plane can be used to reduce tracking-induced errors. A clicking device for selection minimizes fatigue. When only one arm is available, the dwell method is best. Figure 9. Design implications of consumed endurance. Left, menu items should be located in the bottom of the UI. Right, for some applications, such as free gesturing, designers may consider relative input which is possible anywhere in the interaction plane instead of a fixed location. Time-based metrics are incomplete indicators of fatigue. Strength can be used to inform the design of endurance- efficient interactions techniques. The SEATO layout supports endurance-efficient mid-air text-entry, without compromising efficiency. LIMITATIONS Our CE implementation for Microsoft Kinect presents two main limitations. First, it requires line of sight to the user’s complete body in order to form a complete skeleton. Second, the skeleton measurements become noisy due to difficulties differentiating between the user’s arm and body (especially when the arm is close to the body). These difficulties can be avoided in future versions of the sensors (higher resolution, improved tracking) or using alternative tracking systems. In future work, we will extend our model to capture other arm-segments and use individual body metrics (length and mass). Moreover, while this paper shows a strong correlation between CE and Borg CR10 during simple mid-air arm movements; further research is needed into highly dynamic settings and the effects of experience and accumulated fatigue. Finally, as advances in the fields of sport sciences and ergonomics refine the notion of muscle fatigue in an objective manner, the definition and validity of CE should also be revisited against such objective metric. CONCLUSIONS In this paper we introduce consumed endurance; a metric to characterize shoulder fatigue in mid-air interactions. CE only requires the tracking system used to interact with the NUI itself, and thus it is a real-time, objective, non-invasive and non-obtrusive approach to assess gorilla-arm. Through an initial study, we showed CE’s validity as a metric of fatigue and its gender neutrality. Using CE, researchers do not need to ask participants about their perceived physical effort due to the strong correlation between CE and the Borg CR10 scale. We showed how CE can be used as an evaluation tool for selecting suitable mid-air interaction parameters. We focused our exploration on item selection in a 2D plane and investigated the suitable variables for plane location, arm extension, plane size and selection method. Our results show that the combination of plane location and arm extension with the least endurance demands (i.e., creating the least fatigue) is at the vertical center of the body, on the side of the moving arm, and with a bent posture. Finally, selections by the dwell method are most appropriate when only one hand is available. Our results along with a related metric, strength, guided the design of the SEATO text-entry layout for mid-air interactions. Results show that SEATO is on par with QWERTY in terms of words per minute and typing error rate, and consumes only a quarter of endurance. APPENDIX A FORCE AND TORQUE CALCULATIONS From the definition of torque ( ) we know that   . Where is the distance from the shoulder joint to where the force is applied, is the angle between the force vector and the axis, and is the measured force at distance . Given that all forces are tangential to the distance vector, we know that   . Therefore, the equivalent  at the CoM at distance r () is:       The   ratio at the CoM can be expressed as:          APPENDIX B - ARM CENTER OF MASS CALCULATION The CoM of a two segment body is located along the vector linking the CoMs of each segment, at a distance from the first segment’s CoM equal to the ratio between the second segment’s mass and the combined masses of both segments. Figure 10 shows the arm as a three segments body composed of upper arm (ShEb), forearm (EbWr), and hand (WrHa). Applying the process described above for a two segment body, and using the values presented by Freivalds [17], we calculate the CoM of the forearm + hand combination as:          Then, we apply a similar process for the CoM of the upper arm + (forearm + hand) combination as:         APPENDIX C - ARM INERTIA CALCULATION The inertia of a multi-segment body like the arm ( ) is a vector of magnitude equal to the sum of each segment’s inertia, and in the direction (unit vector =  ) of the cross product of the movement of its CoM:     Where the unit vector of the direction ( ) is equal to:       And the magnitude ( ) is equal to:          Figure 10. Arm segments involved in calculating its CoM. REFERENCES 1. Gesture recognition. http://en.wikipedia.org/wiki/Gesture_recognition. 2. Gorilla arm. www.computer- dictionaryonline.org/index.asp?q=gorilla+arm. 3. Visible human project. www.nlm.nih.gov/research/visible/visible human.html. 4. Bachynskyi, M., Oulasvirta, A., Palmas, G., and Weinkauf, T. 2013. Biomechanical simulation in the analysis of aimed movements. In CHI '13 EA. ACM. 5. Bakke, M., Thomsen, C., Vilmann, A., Soneda, K., Farella, M., and Miller, E. 1996. Ultra sonographic assessment of the swelling of the human masseter muscle after static and dynamic activity. Archives of Oral Biology 41, 2, 133 140. 6. Barclay, K., Wei, D., Lutteroth, C., and Sheehan, R. 2011. A quantitative quality model for gesture based user interfaces. In Proc. OZCHI '11, ACM, 3139. 7. Bigdelou, A., Schwarz, L., and Navab, N. 2012. An adaptive solution for intra-operative gesture-based human-machine interaction. In Proc. IUI '12, ACM. 8. Borg, G. 1998. Borg’s Perceived Exertion and Pain Scales. Human Kinetics. 9. Boring, S., Jurmu, M., and Butz, A. 2009. Scroll, tilt or move it: using mobile phones to continuously control pointers on large public displays. In Proc. OZCHI '09. ACM. 10. Bustamante, E.A., and Spain, R.D. 2008. Measurement invariance of the Nasa TLX. In Proc. of the Human Factors and Ergonomics Society Annual Meeting 52, 19. 11. Cockburn, A., Quinn, P., Gutwin, C., Ramos, G., and Looser, J. 2011. Air pointing: Design and evaluation of spatial target acquisition with and without visual feedback. Int. J. Hum.-Comput. Stud. 69, 6, 401414. 12. Cuccurullo, S., Francese, R., Murad, S., Passero, I., and Tucci, M. 2012. A gestural approach to presentation exploiting motion capture metaphors. In Proc. AVI ’12, ACM. 13. Dedering, Å., Gnospelius, Å., and Elfving, B. 2010. Reliability of measurements of endurance time, electromyographic fatigue and recovery, and associations to activity limitations, in patients with lumbar disc herniation. Physiotherapy Research International, 15(4), 189-198. 14. Edmunds, T., Gentner, R., d’Avella, A., and Pai, D. 2012. Feasible wrench space and its estimation for isometric haptic interaction. In Proc. HAPTICS '12. IEEE. 15. Elfving, B. and Dedering, Å. 2007. Task dependency in back muscle fatigue Correlations between two test methods. Clinical Biomechanics, 22(1), 28-33. 16. Ferguson, S.A., W. Gary, A., Le, P., Rose, J.D. and Marras, W.S. 2011. Shoulder Muscle Oxygenation During Repetitive Tasks. Human Factors and Ergonomics Society Annual Meeting 55,1, 1039-1041. 17. Freivalds, A. 2004. Biomechanics of the Upper Limbs: Mechanics, Modeling, and Musculoskeletal Injuries. 1 ED. CRC Press. 18. Garner, B.A., and Pandy, M.G. 2001. Musculoskeletal model of the upper limb based on the visible human male dataset. Computer Methods in Biomechanics and Biomedical Engineering 4, 2, 93126. 19. Garzotto, F., and Valoriani, M. 2012. Don’t touch the oven: motion-based touchless interaction with household appliances. Proc. AVI’12, ACM. 20. Harrison, C., Ramamurthy, S., and Hudson, S.E. 2012. On-body interaction: armed and dangerous. In Proc. TEI'12, ACM. 21. Hart, S.G., and Stavenland, L.E. 1998. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Human Mental Workload, P. A. Hancock and N. Meshkati, Eds. 139-183. 22. Huang, M.-C., Chen, E., Xu, W., and Sarrafzadeh, M. 2011. Gaming for upper extremities rehabilitation. In Proc. WH '11, ACM. 23. Li, F.C.Y., Dearman, D., and Truong, K. N. 2009. Virtual shelves: interactions with orientation aware devices. In Proc. UIST ’09, ACM. 24. MacKenzie, I.S., and Soukoreff, R.W. 2003. Phrase sets for evaluating text entry techniques. In CHI EA '03, ACM. 25. Mentis, H. M., O’Hara, K., Sellen, A., and Trivedi, R. 2012. Interaction proxemics and image use in neurosurgery. In Proc. CHI ’12, ACM. 26. Morris, M. G., Dawes, H., Howells, K., Scott, O. M., and Cramp, M. 2008. Relationships between muscle fatigue characteristics and markers of endurance performance. Sports Science and Medicine 7, 431-436. 27. Öberg, T., Sandsjö, L., and Kadefords, R. 1994. Subjective and objective evaluation of shoulder muscle fatigue. Ergonomics, 37(8), 1323-1333. 28. Peres, S.C., Nguyen, V., Kortum, P.T., Akladios, M., Wood, S.B., and Muddimer, A. 2009. Software ergonomics: relating subjective and objective measures. In CHI EA09, ACM. 29. Rohmert, W. 1960. Ermittlung von erholungspausen fr statische arbeit des menschen. European Journal of Applied Physiology and Occupational Physiology 18. 123164. 30. Sjgaard, G., Savard, G., and Juel, C. 1988. Muscle blood flow during isometric activity and its relation to muscle fatigue. European Journal of Applied Physiology and Occupational Physiology 57, 327335. 31. Tan, H., Radcliffe, J., Ga, B.N., Tan, H.Z., Eberman, B., Srinivasan, M.A., and Cheng, B. 1994. Human factors for the design of force-reflecting haptic interfaces. Dynamic Systems and Control, 55(1), 353-359. 32. Troiano, A., Naddeo, F., Sosso, E., Camarota, G., Merletti, R., and Mesin, L. 2008. Assessment of force and fatigue in isometric contractions of the upper trapezius muscle by surface EMG signal and perceived exertion scale. Gait & Posture, 28(2), 179-186. 33. Wobbrock, J.O., Findlater, L., Gergle, D., and Higgins, J.J. 2011. The aligned rank transform for nonparametric factorial analyses using only ANOVA procedures. In Proc. CHI '11. ACM, 143-146. ## Supplementary resource (1) ... We also made sure that the light condition was consistent and in good condition during the experiment to ensure the tracking was stable. Therefore, we believe that the high workload for hand-based methods might be due to arm/hand fatigue [17], which would have made the task more complicated and unnecessarily cumbersome [2]. ... ... If correcting errors during the session is not an issue, users can use Controller+Click. However, users should be comfortable with a relatively higher total error rate due to hand mobility limitations [17]. Head+Dwell should be considered as the first option in both device-free and hands-free scenarios. ... ... Head+Dwell should be considered as the first option in devicefree scenarios since it provides a better overall user experience and produces a lower workload than hand-based techniques (i.e., Hand+Dwell and Hand+Pinch). Hand+Dwell and Hand+Pinch should be minimized in rapid pointing-based text selection tasks since they generate a high overall workload and low overall user experience, with the potential to lead to the gorilla arm syndrome [17]. However, they can be considered when the user has an issue with rotating their neck. ... Preprint Full-text available Text selection is an essential activity in interactive systems, including virtual reality (VR) head-mounted displays (HMDs). It is useful for: sharing information across apps or platforms, highlighting and making notes while reading articles, and text editing tasks. Despite its usefulness, the space of text selection interaction is underexplored in VR HMDs. In this research, we performed a user study with 24 participants to investigate the performance and user preference of six text selection techniques (Controller+Dwell, Controller+Click, Head+Dwell, Head+Click, Hand+Dwell, Hand+Pinch). Results reveal that Head+Click is ranked first since it has excellent speed-accuracy performance (2nd fastest task completion speed with 3rd lowest total error rate), provides the best user experience, and produces a very low workload -- followed by Controller+Click, which has the fastest speed and comparable experience with Head+Click, but much higher total error rate. Other methods can also be useful depending on the goals of the system or the users. As a first systematic evaluation of pointing*selection techniques for text selection in VR, the results of this work provide a strong foundation for further research in this area of growing importance to the future of VR to help it become a more ubiquitous and pervasive platform. ... We also made sure that the light condition was consistent and in good condition during the experiment to ensure the tracking was stable. Therefore, we believe that the high workload for hand-based methods might be due to arm/hand fatigue [17], which would have made the task more complicated and unnecessarily cumbersome [2]. ... ... If correcting errors during the session is not an issue, users can use Controller+Click. However, users should be comfortable with a relatively higher total error rate due to hand mobility limitations [17]. Head+Dwell should be considered as the first option in both device-free and hands-free scenarios. ... ... Head+Dwell should be considered as the first option in devicefree scenarios since it provides a better overall user experience and produces a lower workload than hand-based techniques (i.e., Hand+Dwell and Hand+Pinch). Hand+Dwell and Hand+Pinch should be minimized in rapid pointing-based text selection tasks since they generate a high overall workload and low overall user experience, with the potential to lead to the gorilla arm syndrome [17]. However, they can be considered when the user has an issue with rotating their neck. ... Conference Paper Text selection is an essential activity in interactive systems, including virtual reality (VR) head-mounted displays (HMDs). It is useful for: sharing information across apps or platforms, highlighting and making notes while reading articles, and text editing tasks. Despite its usefulness, the space of text selection interaction is underexplored in VR HMDs. In this research, we performed a user study with 24 participants to investigate the performance and user preference of six text selection techniques (Controller+Dwell, Controller+Click, Head+Dwell, Head+Click, Hand+Dwell, Hand+Pinch). Results reveal that Head+Click is ranked first since it has excellent speed-accuracy performance (2nd fastest task completion speed with 3rd lowest total error rate), provides the best user experience, and produces a very low workload\textemdash followed by Controller+Click, which has the fastest speed and comparable experience with Head+Click, but much higher total error rate. Other methods can also be useful depending on the goals of the system or the users. As a first systematic evaluation of pointing\times\$selection techniques for text selection in VR, the results of this work provide a strong foundation for further research in this area of growing importance to the future of VR to help it become a more ubiquitous and pervasive platform.
... Further, it supports the automatic adaptation of UIs so that interactive elements remain within easy reach while the user moves about in a changing physical environment. The ergonomics metrics currently supported in XRgonomics are RULA [38], Consumed Endurance [24], and muscle activation [3]. Prior research has explored ergonomics [3,24,38] and while the resulting metrics help evaluate existing UIs, it is difficult to use them for generating novel UI layouts. ...
... The ergonomics metrics currently supported in XRgonomics are RULA [38], Consumed Endurance [24], and muscle activation [3]. Prior research has explored ergonomics [3,24,38] and while the resulting metrics help evaluate existing UIs, it is difficult to use them for generating novel UI layouts. Further, the formulated design recommendations can be challenging to interpret and apply, particularly if the ideal interaction space is unavailable, e. g., due to the user's physical environment. ...
Article
Full-text available
Adaptive visualization and interfaces pervade our everyday tasks to improve interaction from the point of view of user performance and experience. This approach allows using several user inputs, whether physiological , behavioral, qualitative, or multimodal combinations , to enhance the interaction. Due to the multitude of approaches, we outline the current research trends of inputs used to adapt visualizations and user interfaces. Moreover, we discuss methodological approaches used in mixed reality, physiological computing, visual analytics, and proficiency-aware systems. With this work, we provide an overview of the current research in adaptive systems.
... Over the last years, HCI researchers have proposed various methods to adapt interface elements in XR applications. They were concerned with the visibility and integration of virtual elements into the physical environment [9,23,37,52] and their reachability or ergonomics [16,30]. When considering criteria to adapt, these typically address independent aspects of the interface, such as position and content [37]. ...
Conference Paper
Full-text available
Adaptive user interfaces can improve experiences in Cross Reality (XR) applications by changing interface elements according to the user's context. Although extensive work explores diferent adaptation policies, XR creators often struggle with their implementation, which involves laborious manual scripting. The few available tools are underdeveloped for realistic XR settings where is often necessary to consider conflicting aspects that affect an adaptation. We fill this gap by presenting AUIT, a toolkit that facilitates the design of optimization-based adaptation policies. AUIT allows creators to flexibly combine policies that address common objectives in XR applications, such as element reachability, visibility, and consistency. In contrast to using rules or scripts, specifying adaptation policies via adaptation objectives simplifies the design process and enables creative exploration of adaptations. After creators decide which objectives to use, a multi-objective solver finds appropriate adaptations in real-time. A study showed that AUIT allowed creators of XR applications to quickly and easily create high-quality adaptations.
Article
Interaction in mid-air can be fatiguing. A model-based method to quantify cumulative subjective fatigue for such interaction was recently introduced in HCI research. This model separates muscle units into three states: active (MA) fatigued (MF) or rested (MR) and defines transition rules between states. This method demonstrated promising accuracy in predicting subjective fatigue accumulated in mid-air pointing tasks. In this paper, we introduce an improved model that additionally captures the variations of the maximum arm strength based on arm postures and adds linearly-varying model parameters based on current muscle strength. To validate the applicability and capabilities of the new model, we tested its performance in various mid-air interaction conditions, including mid-air pointing/docking tasks, with shorter and longer rest and task periods, and a long-term evaluation with individual participants. We present results from multiple cross-validations and comparisons against the previous model and identify that our new model predicts fatigue more accurately. Our modeling approach showed a 42.5% reduction in fatigue estimation error when the longitudinal experiment data is used for an individual participant’s fatigue. Finally, we discuss the applicability and capabilities of our new approach.
Article
The increasing availability of portable handheld mobile Augmented Reality technology is revolutionising the way digital information is embedded into the real world. As this data is embedded, it enables new forms of cross-device collaborative work. However, despite the widespread availability of handheld AR, little is known about the role that device configurations and size play on collaboration. This paper presents a study that examines how completing tasks using a simple mobile AR interface on different device sizes and configurations impacts key factors of collaboration such as collaboration strategy, behaviour, and efficacy. Our results show subtle differences between device size and configurations that have a direct influence on the way people approach tasks and interact with virtual models. We highlight key observations and strategies that people employ across different device sizes and configurations.
Article
Over the past decade, augmented reality (AR) developers have explored a variety of approaches to allow users to interact with the information displayed on smart glasses and head-mounted displays (HMDs). Current interaction modalities such as mid-air gestures, voice commands, or hand-held controllers provide a limited range of interactions with the virtual content. Additionally, these modalities can also be exhausting, uncomfortable, obtrusive, and socially awkward. There is a need to introduce comfortable interaction techniques for smart glasses and HMDS without the need for visual attention. This paper presents StretchAR, wearable straps that exploit touch and stretch as input modalities to interact with the virtual content displayed on smart glasses. StretchAR straps are thin, lightweight, and can be attached to existing garments to enhance users' interactions in AR. StretchAR straps can withstand strains up to 190% while remaining sensitive to touch inputs. The strap allows the effective combination of these inputs as a mode of interaction with the content displayed through AR widgets, maps, menus, social media, and Internet of Things (IoT) devices. Furthermore, we conducted a user study with 15 participants to determine the potential implications of the use of StretchAR as input modalities when placed on four different body locations (head, chest, forearm, and wrist). This study reveals that StretchAR can be used as an efficient and convenient input modality for smart glasses with a 96% accuracy. Additionally, we provide a collection of 28 interactions enabled by the simultaneous touch-stretch capabilities of StretchAR. Finally, we facilitate recommendation guidelines for the design, fabrication, placement, and possible applications of StretchAR as an interaction modality for AR content displayed on smart glasses.
Article
Gestural interaction has evolved from a set of novel interaction techniques developed in research labs, to a dominant interaction modality used by millions of users everyday. Despite its widespread adoption, the design of appropriate gesture vocabularies remains a challenging task for developers and designers. Existing research has largely used Expert-Led, User-Led, or Computationally-Based methodologies to design gesture vocabularies. These methodologies leverage the expertise, experience, and capabilities of experts, users, and systems to fulfill different requirements. In practice, however, none of these methodologies provide designers with a complete, multi-faceted perspective of the many factors that influence the design of gesture vocabularies, largely because a singular set of factors has yet to be established. Additionally, these methodologies do not identify or emphasize the subset of factors that are crucial to consider when designing for a given use case. Therefore, this work reports on the findings from an exhaustive literature review that identified 13 factors crucial to gesture vocabulary design and examines the evaluation methods and interaction techniques commonly associated with each factor. The identified factors also enable a holistic examination of existing gesture design methodologies from a factor-oriented viewpoint and highlighting the strengths and weaknesses of each methodology. This work closes with proposals of future research directions of developing an iterative user-centered and factor-centric gesture design approach as well as establishing an evolving ecosystem of factors that are crucial to gesture design.
Article
Full-text available
This paper presents PinchText, a mid-air technique with a condensed keys-based keyboard, which combines hand positions and pinch gestures, enabling one-handed text entry for Head-mounted displays (HMDs). Firstly, we conduct Study 1 to collect and analyze the typing data of PinchText with two arm postures and two movement directions, obtaining the range of hand position corresponding to the middle key set. Then, we conduct Study 2, a 6-block experiment, finding that PinchText with Hand-Up Vertical (UpV) and Hand-Down Vertical (DownV) modes could achieve a speed of 12.71 words-per-minute (WPM) and 11.14 WPM respectively with both uncorrected error rates less than 0.5%, which is 71% faster than the index finger pinch-based technique. Finally, Study 3 is conducted to explore the potential of reducing the size of the decoupled visual keyboard of PinchText, verifying that the occlusion of the virtual keyboard can be decreased. Overall, PinchText is an efficient, easy-to-learn, and comfortable text entry technique for HMDs.
Article
Full-text available
Article
Full-text available
The purpose of this study was to quantify shoulder muscle oxygenation during repetitive shoulder exertions that were similar to motions found in automobile assembly tasks. Ten subjects participated in the study. There were three independent variables: 1) shoulder flexion angle; 2) frequency; and 3) force. The dependent measure was percentage change in muscle oxygenation for the anterior deltoid and trapezius. The results showed significant muscle oxygenation decreases for each of the main effects (shoulder flexion angle, frequency and force). The interaction of force and repetition was significant for the anterior deltoid, indicating that, as repetition increased the magnitude of the differences between the force levels increased. The interaction of repetition and shoulder angle was also significant. The results of this research illustrate that ergonomists need to consider the interaction of injury risk factors that may trigger musculoskeletal disorders of the shoulder.
Conference Paper
Full-text available
For efficient design of gestural user interfaces both performance and fatigue characteristics of movements must be understood. We are developing a novel method that allows for biomechanical analysis in conjunction with performance analysis. We capture motion data using optical tracking from which we can compute performance measures such as speed and accuracy. The measured motion data also serves as input for a biomechanical simulation using inverse dynamics and static optimization on a full-body skeletal model. The simulation augments the data by biomechanical quantities from which we derive an index of fatigue. We are working on an interactive analysis tool that allows practitioners to identify and compare movements with desirable performance and fatigue properties. We show the applicability of our methodology using a case study of rapid aimed movements to targets covering the 3D movement space uniformly.
Article
Full-text available
The aim of this study was to examine the relationship of a range of in-vivo whole muscle characteristics to determinants of endurance performance. Eleven healthy males completed a cycle ergometer step test to exhaustion for the determination of the lactate threshold, gross mechanical efficiency, peak power and VO2max. On two separate occasions, contractile and fatigue characteristics of the quadriceps femoris were collected using a specially designed isometric strength-testing chair. Muscle fatigue was then assessed by stimulating the muscle for 3 minutes. Force, rate of force development and rates of relaxation were calculated at the beginning and end of the 3 minute protocol and examined for reliability and in relation to lactate threshold, VO2max, gross mechanical efficiency and peak power. Muscle characteristics, rate of force development and relaxation rate were demonstrated to be reliable measures. Force drop off over the 3 minutes (fatigue index) was related to lactate threshold (r = -0.72 p ¼ 0.01) but not to VO2max. The rate of force development related to the peak power at the end of the cycle ergometer test (r = -0.75 p ¼ 0.01). Rates of relaxation did not relate to any of the performance markers. We found in-vivo whole muscle characteristics, such as the fatigue index and rate of force development, relate to specific markers of peripheral, but not to central, fitness components. Our investigation suggests that muscle characteristics assessed in this way is reliable and could be feasibly utilised to further our understanding of the peripheral factors underpinning performance. Key pointsParticipants with a high lactate threshold displayed greater fatigue resistance in the electrical stimulation test.Muscle performance characteristics relate to specific components of endurance performance.The electrical stimulation protocol could be a useful technique, alongside other established measures, when constructing a physiological profile of a participant.
Article
Mental workload is one of the most important constructs of interests for Human Factors researchers. Adequately assessing the amount of mental workload that people experience while performing tasks under specific conditions is essential for the design of safe and efficient systems. Due to its ease of use, the NASA TLX has become the most widely used method of measuring mental workload. However, its psychometric properties are still questionable. The purpose of this study was to examine the extent of measurement invariance of the TLX and raise awareness in the Human Factors community. Two hundred participants reported the amount of mental workload they typically experience while driving in urban and rural areas and across the country. Results indicated that the TLX lacked scalar invariance, thereby biasing the estimation of mean scores and making the examination of mean differences misleading. These findings suggest that researchers should first examine the extent of measurement invariance of the TLX before they proceed to make inferences about mean differences in the amount of mental workload reported by participants under different conditions.
Article
Standard approaches to mapping force to displacement or force to velocity with multi-DoF isometric haptic devices typically ignore the directional variability in a user's feasible wrench. We demonstrate that such a directionally-uniform mapping tends to either over-sensitize the interaction in some directions or under-utilize the user's operational range. To increase the effective use of the user's operational range it is necessary to model that range across all directions; for high-dimensional devices that measure wrenches (i.e, forces and torques) the space of directions is non-trivial to model by sampling. We present an approach that uses in-depth measurement of the feasible wrench space of a small number of users to extract a generic model for a given device interaction context; the generic model can then be automatically fitted to other users through a small number of measurements. In a user study comparing our method against the standard directionally-uniform assumption we show that our method generates a significantly better estimation of a user's output range, while requiring only a few measurements.
Article
Computerized medical systems play a vital role in the operating room, however, sterility requirements and interventional workflow often make interaction with these devices challenging for surgeons. Typical solutions, such as delegating physical control of keyboard and mouse to assistants, add an undesirable level of indirection. We present a touchless, gesture-based interaction framework for the operating room that lets surgeons define a personalized set of gestures for controlling arbitrary medical computerized systems. Instead of using cameras for capturing gestures, we rely on a few wireless inertial sensors, placed on the arms of the surgeon, eliminating the dependence on illumination and line-of-sight. A discriminative gesture recognition approach based on kernel regression allows us to simultaneously classify performed gestures and to track the relative spatial pose within each gesture, giving surgeons fine-grained control of continuous parameters. An extensible software architecture enables a dynamic association of learned gestures to arbitrary intraoperative computerized systems. Our experiments illustrate the performance of our approach and encourage its practical applicability.
Article
Motion-based touchless interaction empowers users to interact using movements and gestures, and without the burden of physical contact with technology (e.g., data gloves, body markers, or remote controllers). Most motion-based touchless applications are designed for interaction "in-the-large", where users engage with medium or large displays. Our research explores motion-based touchless interaction "in-the- small" that involves only small displays (of the size, for example, of smart phone screens). We have applied this novel paradigm to develop interactive applications for household appliances, discovering that the "in-the-small" feature raises a number of challenging design issues, exemplified in the paper through a case study.