Content uploaded by Yuri Quintana
Author content
All content in this area was uploaded by Yuri Quintana on Aug 10, 2019
Content may be subject to copyright.
A Usability Evaluation of the
InfoSAGE App for Family-Based
Medication Management
Yuri QUINTANA1,2, Darren FAHY1, Bradley CROTTY3, Maxwell GORENBERG1,
Ruchira JAIN1, Eli KALDANY1, Lewis LIPSITZ2,4, Yi-Pei CHEN1, Juan HENAO1,
Charles SAFRAN1,2
1Division of Clinical Informatics, Beth Israel Deaconess Medical Center, Boston, MA
2Harvard Medical School, Boston, MA, U.S.A
3Medical College of Wisconsin, Milwaukee, WI, U.S.A
4Institute for Aging Research, Hebrew Senior Life, Boston, MA, U.S.A
Abstract. The design of a mobile medication manager within a broader family and
elder-centric collaboration platform faces challenges of usability and wide
applicability. To inform the development and use cases of eldercare apps, we present
the preliminary results of a usability study of an iOS and Android app intended for
both family members and aging adults for the mobile management of medication
lists. Seven participants were recorded during the performance of eight typical use-
case scenarios of the medication portion of the InfoSAGE app. Audio and video
recordings were analyzed for themes and events. The aim of this paper is to help
inform future design choices for eldercare mobile apps.
Keywords. health information technology (HIT), elderly, smartphone, usability
1. Introduction
There is a rising life expectancy of populations in both developed and developing
countries. The number of people aged 60 years and over has tripled since 1950 and is
projected to reach 2.1 billion by 2050 [1]. Many informal caregivers of frail elders have
high levels of stress that can lead to physical, mental and financial problems [2].
Technology enabled support systems may help improve the care coordination between
patient and family members [3-4] but acceptance of technology and sustained usage for
elder care application remains a challenge [5-11], particularly for mobile applications
[5,6,11]. We present the preliminary results of a usability evaluation of the InfoSAGE
[3] family-based care coordination mobile app, and propose recommendations for the
development of future mobile applications for medication management for elders.
2. InfoSAGE
InfoSAGE (https://www.infosagehealth.org) is a communication and collaboration
platform for family caregivers and elders navigating the aging process developed by the
Improving Usability, Safety and Patient Outcomes with Health Information Technology
F. Lau et al. (Eds.)
© 2019 The authors and IOS Press.
This article is published online with Open Access by IOS Press and distributed under the terms
of the Creative Commons Attribution Non-Commercial License 4.0 (CC BY-NC 4.0).
doi:10.3233/978-1-61499-951-5-352
352
Division of Clinical Informatics at the Harvard Medical Faculty Physicians at Beth Israel
Deaconess Medical Center (BIDMC) in conjunction with aging researchers at the
Institute for Aging Research at Hebrew Senior Life [3]. It consists of a web-app and
mobile-app for the sharing of tasks, calendars, contacts, medication lists, and associated
care discussions. InfoSAGE is primarily intended as a family-centric tool coordinating
care around a single elder, called a keystone, but is frequently utilized by single users for
the management of medication lists. InfoSAGE is open to the public and free to use. To
date, 321 users have signed up across 180 family networks.
Currently, the primary focus of the research surrounding the InfoSAGE platform is
in the observation of changing medication lists, the discrepancies between family
maintained lists and medications recorded in health records, and medication safety. A
novel, family-curated medication manager was developed for InfoSAGE, leveraging
online National Library of Medicine databases, and versatile recording methods to
capture name, type, dose, strength, route, indication, schedules, and administration
directions in a readily accessible and shareable format. Owing to the diversity of age
within our target user population, InfoSAGE was developed to be approachable and
usable by both aging older adults, their baby-boomer children, and younger users.
Despite increasing smartphone use in those 75 or older, achieving the desired usability
across this age range can be challenging and benefits from iterative, systematic design.
3. Methods
A convenience sample of subjects was recruited from the Boston area using
advertisements and referrals from collaborating facilities at Hebrew Senior Life, and
through a grass-roots campaign focused on BIDMC. Online local message boards were
also utilized to increase the reach and diversity of the sample. Persons age 18 or older
who are involved in the care of an elder aged 75 or older, or the elder themselves were
invited to participate in the study. This study was conducted under the review and
oversight of the Beth Israel Deaconess Medical Center institutional review board
(#2014P000296) using observational usability methods [13-14]. All study visits took
place at BIDMC facilities, using the iOS InfoSAGE app (version 2.8) on a provided
Apple iPad Pro. Participants’ interactions with the screen were recorded by a camera as
well as screen capture software. During the testing process, participants were asked to
‘think-aloud’ as they completed the study scenarios, in order to capture any reasoning
behind their specific actions. Spoken comments were recorded and transcribed.
Eight scenarios of varying level of complexity were employed to capture common
use cases for the medication manager feature of the InfoSAGE app. Participants were
asked to add multiple medications with varying strengths, doses, and schedules. Both
generic and name brand medications were included. Participants were also asked to
utilize other primary features of the app such as drug-drug interaction notifications. No
specific instruction was given from study staff about the study scenarios, nor were any
participants shown how to use any feature of the app. If a scenario could not be completed
due to difficulty, participants were given guidance on how to proceed. Participants used
standardized instructions and scenarios, and did not enter any of their own, or elder’s,
personal health information.
Demographic and subject-reported comfort with the Internet was collected prior to
beginning the study scenarios and an after-study survey was collected to evaluate the
system and process. Participants were asked to rate on a Likert-type scale, with one
Y. Quintana et al. / A Usability Evaluation of the InfoSAGE App 353
corresponding to ‘Strongly Agree’ and seven to ‘Strongly Disagree’, the ease of use,
satisfaction with entry speed, the usefulness of in-app help, the difficulty of completion,
and how useful the app would be as part of their informal caregiving. Semi-structured
interviews were used to obtain further feedback from participants.
4. Results
All audio recordings were transcribed and thematically coded using an iterative
process by a study staff member in consultation with the rest of the team. Codes and
themes were categorized into five groups: Software response, user actions, user feelings,
usability process, and task completion. Video recordings were analyzed using the
Behavioral Observation Research Interactive Software (BORIS, v. 7.0.8), coded for
events and themes and correlated to associated audio events. Time to complete each
scenario was recorded, along with metadata such as the number of taps, use of in-app
help, and frequency of mistaken ‘clicks.’ Observations were captured from seven
participants involved in the care of an elder. Median age was 50 (2.5 IQR), and 71%
were female. Demographics and self-reported skills are shown in table .
Table 1: Demographics, Survey Responses, Time to Complete (n=7)
Age (Median, IQR) 50 (2.5)
Female (%) 5 (71%)
Ethnicity
Hispanic or Latino 1 (14%)
Not Hispanic or Latino 6 (86%)
Race
Black or African American 1 (14%)
White 6 (86%)
Level of education
4-year college graduate 5 (71%)
Masters or doctoral degree 2 (29%)
What is your comfort level with using the Internet?
Very comfortable 6 (86%)
Comfortable 1 (14%)
Do you currently care for an elderly family member? Yes 7 (100%)
On average, I access InfoSAGE: Never 7 (100%)
I would personally rate my skills with online websites as
Intermediate 3 (43%)
Expert 4 (57%)
I would personally rate my skills with mobile apps as
Beginner 1 (14%)
Intermediate 4 (57%)
Expert 2 (29%)
Questions (1=Strongly Agree, 7=Strongly Disagree)
Mean (SD)
Overall, I am satisfied with the ease of completing these tasks 1.3 (0.5)
Overall, I am satisfied with the amount of time it took me to complete these
tasks 1.3 (0.5)
Overall, I am satisfied with the usefulness of the help for completing these
tasks 1.8 (1.6)
How would you rate the difficulty of completing the task scenarios 1.7 (0.5)
Overall, after completing these tasks, I feel that this could potentially be used
on a regular basis as part of my patient care/loved one's patient care and
communicating my current list of medications with my care provider/loved
one's care provider 1.2 (0.4)
Time to complete 14m04s
1
Y. Quintana et al. / A Usability Evaluation of the InfoSAGE App354
All participants were naïve users, having never used either the InfoSAGE website
or mobile applications. Except for one participant, all self-rated their skills with mobile
apps as intermediate or expert. All were familiar with the general use of the iPad. Each
participant successfully completed all eight scenarios, with an average time to
completion of 14 minutes 04 seconds. Familiarity and comfort with the app increased for
each subsequent scenario after the first despite increasing complexity, and the average
time to complete scenario two, the addition of warfarin (5mg tablet, by mouth, once
daily) to the medication list, decreased by 37% when compared to scenario one, the
addition of lisinopril (5mg tablet, once daily).
Audio and video coding uncovered confusion in the entering of medication details
surrounding doses and strengths, and was the most consistent theme observed in all
participants. Two participants suggested that the current method of entering medication
(figure 1) “feels redundant.” The app currently employs several fields that seem
repetitive (use of dose and schedule, naming conventions) but allow for maximum
flexibility in the recording of a wide range of medications, prescribing practices, and
delivery methods (aerosol, injection, by mouth).
Figure 1. Adding a medication to the InfoSAGE mobile app, process flow
Terminology was a consistent source of questions, ambiguity, and hesitation
throughout the scenarios. Two participants confused medication side effects with drug-
drug interactions, and all but one participant had difficulty discovering where medication
side effects were listed. The difference between generic and name-brand medications,
and how they are listed through the app, also proved to be challenging for participants.
Medications are loaded by searching the National Library of Medicine’s RXNorm
database, which also contains information about dose and strength. These strings are
shown to the user for selection of the appropriate medication, but often contain details,
including generic names, that must be confirmed on further screens. Lastly, the
difference between deactivating and deleting a medication was a consistent point of
clarification. Deactivating is used when a user stops taking a medication but wants to
keep it on record, and deleting removes it. Despite these difficulties, participants also
remarked on the ease of use of the app, and several expressed that the utility may be
useful in their own informal caregiving. To the question ‘Do you think that this system
could possibly improve your mother’s care’, one participant stated:
Y. Quintana et al. / A Usability Evaluation of the InfoSAGE App 355
“Yes, but not my mother’s care. It would improve my sister and I -- our sanity. I
think it would be more useful for us than for her, which in essence would benefit
her. More for everybody to figure out what is going on.”
Another participant indicated that the thoroughness of the medication entry process
made her feel secure. As the user enters additional details, the app automatically
constructs a summary sentence for that medication. Others requested additional features,
or changes to existing features, another commonly observed theme:
“I’d like to see a planner, or daily schedule, or something. That would be useful,
so what we’ve just done is only useful to the extent that it feeds into that. But you
have to take nothing for granted.”
Overall, participants were satisfied with the ease of completing the scenarios (1.3,
SD: 0.5), the amount of time each task took (1.3, SD: 0.5), difficulty of each task (1.7,
SD: 0.5), and the utility in their own caregiving (1.2, SD: 0.4).
5. Discussion
Our findings indicate that while users were able to use the app, there were difficulties in
understanding both the content and the wide range of options to enter medications. The
comments regarding the ability of users to understand the terminology, side effects and
drug interactions suggest that more improvements are needed to respond to health
literacy issues [15]. The drug information for elders and their caregivers will need to be
further developed at a language level that more users can understand [16].
Design may help users distinguish differences between side effects and drug-
interaction, such possibly using different colors and labels. Some users also expressed
confusion on the various options for entering medications. By adding more functionality
to the app we can support more possible scenarios of medication schedules, but we may
be increasing the difficulty of the app for users that have simpler medication regimens.
We will need to balance feature rich but usability poor tradeoffs. We may consider in
future having a simpler entry form for rapid entry and a link to more advanced fields.
A few users also had some issues recognizing which items were buttons. This is in
part due to the flat graphical style of the most recent release of the mobile operating
system. Frequent changes by mobile computer makers to user interface layouts will cause
problems for users with less computer experience and may be particularly problematic
for older users that may have vision problems.
While there are some evaluations of mobile apps with elders [17-19], however the
existing guidelines for user interfaces used by elder populations from international
organizations such as The World Wide Web Consortium [20-21], The European
Commission [22], and the US Government [23-24] have focused on web-based interfaces.
Future research could strive towards a consensus recommendation on user interface
guidelines for mobile apps for elder health applications with insights from academic
groups, industry groups, elders and their families.
Acknowledgements: We acknowledge support from AHRQ grant R18HS24869-01.
References
[1] United Nations. World Population Ageing. United Nations Geneva, Switzerland, 2001.
Y. Quintana et al. / A Usability Evaluation of the InfoSAGE App356
[2] M. Lopez-Hartmann, J. Wens, V. Verhoeven, R. Remmen. The effect of caregiver support interventions
for informal caregivers of community-dwelling frail elderly: A systematic review. IJIC 12 (2012), e133.
[3] Y. Quintana, B. Crotty, D. Fahy, A. Orfanos, R. Jain, E. Kaldany, L. Lipsitz, D. Engorn, J. Rodriguez, F.
Pandolfe, A. Bajracharya, W. V. Slack, C. Safran. InfoSAGE: Use of Online Technologies for
Communication and Elder Care. Stud Health Technol Inform. 234 (2017), 280-285.
[4] V. Cammack, M. Byrne. Accelerating a network model of care: Taking a social innovation to scale. Tech
Innov Manage Review. 7 (2012), 26-30.
[5] R. McCloskey, P. Jarrett, C. Stewart, L. Keeping-Burke. Recruitment and retention challenges in a
technology-based study with older adults discharged from a geriatric rehabilitation unit. Rehabil Nurs.
Jul-Aug 40(4) (2015), 249-59
[6] D. M. Levine, S. R. Lipsitz, J. A. Linder. Changes in everyday and digital health technology use among
seniors in declining health. J Gerontol A Biol Sci Med Sci. Mar 14 73(4) (2018), 552-559.
[7] I. Vedel, S. Akhlaghpour, I. Vaghefi, H. Bergman, L. Lapointe. Health information technologies in
geriatrics and gerontology: A mixed systematic review. JAMIA. Nov-Dec 20(6) (2013), 1109-19. doi:
10.1136/amiajnl-2013-001705. Epub 2013 May 10.
[8] T. Heart, E. Kalderon. Older adults: Are they ready to adopt health-related ICT? Int J Med Inform, 82(11)
(2013), e209–e231.
[9] S. H. Fischer, D. David, B. H. Crotty, M. Dierks, C. Safran. (2014). Acceptance and use of health
information technology by community-dwelling elders. Int J Med Inform. 83(9) (2014), 624–635.
[10] S. Yusif, J. Soar, A. Hafeez-Baig. Older people, assistive technologies, and the barriers to adoption: A
systematic review. Int J Med Inform. Oct 94 (2016), 12-6.
[11] R. Hoque, G. Sorwar. Understanding factors influencing the adoption of mHealth by the elderly: An
extension of the UTAUT model. Int J Med Inform. May 101 (2017), 75-84.
[12] A. Anderson, A. Perrin. Pew Research Center. Tech adoption climbs among older adults. May 17, 2017.
URL: http://www.pewinternet.org/2017/05/17/tech-adoption-climbs-among-older-adults/ Last Accessed
September 17, 2018.
[13] A. Kushniruk, E. Borycki. Low-cost rapid usability testing: Its application in both product development
and system implementation. Stud Health Technol Inform. 234 (2017), 195-200.
[14] E. M. Borycki, J. Griffith, H. Monkman, C. Reid-Haughian. Isolating the effects of a mobile phone on
the usability and safety of eHealth software applications. Stud Health Technol Inform. 234 (2017), 37-
41.
[15] H. Monkman, A. W. Kushniruk. All consumer medication information is not created equal: Implications
for medication safety. Stud Health Technol Inform. 234 (2017), 233-237.
[16] H. Monkman, A. W. Kushniruk. All consumer medication information is not created equal: Implications
for medication safety. Stud Health Technol Inform. 234 (2017), 233-237.
[17] A. S. Lum, T. K. Chiew, C. J. Ng, Y. K. Lee, P. Y, C. H. Teo. Development of a web-based insulin
decision aid for the elderly: Usability barriers and guidelines. Universal Access in the Information
Society, 16 (2016), 775-791.
[18] K. A. Grindrod, M. Li, A. Gates. Evaluating user perceptions of mobile medication management
applications with older adults: a usability study. JMIR Mhealth Uhealth. Mar 14 2(1) (2014), e11.
[19] K. A. Grindrod, A. Gates, L. Dolovich, R. Slavcev, R. Drimmie, B. Aghaei, C. Poon, S. Khan, S. J. Leat.
ClereMed: Lessons learned from a pilot study of a mobile screening tool to identify and support adults
who have difficulty with medication labels. JMIR Mhealth Uhealth. Aug 15 2(3) (2014), e35.
[20] World Wide Web Consortium (W3C). Older users online: WAI guidelines address older users’ web
experience. User Experience Magazine, 8(1) (2009), URL: https://www.w3.org/WAI/posts/2009/older-
users-online, Last Accessed Sept. 17, 2018,
[21] World Wide Web Consortium (W3C). Older users and web accessibility: Meeting the needs of ageing
web users. URL: http s://www.w3. org/WAI/older-users/ Last Accessed Sept. 17, 2018.
[22] European Commission. Web accessibility initiative: Ageing education and harmonisation (WAI-AGE),
European Commission Project, 6th Framework, IST 035015, 30 September 2010, URL:
www.w3.org/WAI/WAI-AGE/ Last Accessed Sept. 17, 2018,
[23] B. Bailey. Age-related research-based usability guidelines. URL: https://www.usability.gov/get-
involved/blog/2005/11/age-related-guidelines.html November 1, 2005, Last Accessed Sept. 17, 2018.
[24] C. Sibley. Web usability and aging. December 1 2008, URL: https://www.usability.gov/get-
involved/blog/2008/12/aging-and-usability.html Last Accessed Sept. 17, 2018
Y. Quintana et al. / A Usability Evaluation of the InfoSAGE App 357
