Content uploaded by Michelle Sublette
Author content
All content in this area was uploaded by Michelle Sublette on May 02, 2018
Content may be subject to copyright.
Structured Brainstorming Helps Home Network Managers
Transcend Technical Language Barriers to Express Their Needs
Cathy D. Emery
1
, Michelle Sublette
1
, C. Melody Carswell
1
, and K.L. Calvert
2
University of Kentucky, Departments of Psychology
1
and Computer Science
2
,
And the UK Center for Visualization and Virtual Environments
Managing home networks is fraught with usability challenges due, in part, to the adoption
of conventions, tools, and terminology meant for professional network administration.
Eliciting user needs is challenging because of the gulf between the language used by
network professionals and the less precise, unstandardized terms used by home
administrators. Here we modify a traditional Structured Brainstorming (SB) technique to
help circumvent these thorny communication problems. We describe our specific
implementation and provide a sample of the information obtained from our SB sessions.
INTRODUCTION
Structured brainstorming (SB) is a method
for eliciting user needs that combines work by
individuals, dyads, and larger groups in a single
session (Byrne & Barlow, 1993). The rationale for
using SB instead of other group-based methods
(e.g., traditional focus groups) is that it encourages
discussion of higher-level user goals while avoiding
a discussion mired in the details of current product
features. Byrne and Barlow’s original SB
procedure was used with domain experts; however,
we argue that the technique can be adapted for use
with novice participants who have limited facility
with domain vocabulary.
The modified version of SB we describe
below was designed to elicit needs from individuals
who, often reluctantly, serve as home network
administrators. These users may have little formal
training in network administration and have scant
knowledge of the domain’s challenging
terminology. Our application of SB in this context
is based on the assumption that working in dyads
will help participants express their ideas in much
the same way that co-discovery learning is used to
increase fluency during “think aloud” protocols. In
short, we believe that the social structure of SB can
facilitate users’ communication of their system
needs.
Numerous researchers have recounted the
challenges faced by most people when setting up,
configuring, monitoring, and repairing their home
networks (e.g., Franzke & McClard, 1996; Teger &
Waks, 2002; Poole et al., 2009; Grinter et al., 2009).
Many of the tools used for home network
administration were originally designed to meet the
needs of professional network administrators. The
terminology is particularly difficult for home users
who may find it necessary to substitute less precise
but easier to recall terms in order to set up and
maintain their networks. Poole et al. (2009)
recognized the difficulty of acquiring meaningful
verbal reports from users to learn about how they
conceptualization home networks. Instead of
relying on interviews alone, these researchers
collected system sketches from participants to
capture their understanding of network structure and
function. Although participant sketches were not a
formal part of our modified SB procedure, some
participants spontaneously used them to
communicate with partners or with the researchers.
PRACTICE INNOVATION
In this case study, we explore the use of a
relatively simple SB procedure, one that could be
completed in a single day, to learn about the needs
of those household members who manage a home
network. We modified the typical SB procedure in
several ways in order to create an environment that
encouraged discussion without concern for “correct
terminology.” Changes to the procedure were
necessitated by problems we encountered during
pilot sessions -- participants had difficulty
generating responses to our brainstorming exercises,
their responses were too few, and the comments that
were made were often off topic due to the “misuse”
of terminology. After several iterations, we had a
Copyright 2016 by Human Factors and Ergonomics Society. DOI 10.1177/1541931213601298
Proceedings of the Human Factors and Ergonomics Society 2016 Annual Meeting 1284
procedure in place that elicited adequate
commentary by participants to allow us to identify
and prioritize user needs. Our modifications are
highlighted in the following description of the full
protocol.
IMPLEMENTATION
Recruitment. Our SB application differed
from that of Byrne and Barlow (1993) because we
wanted to include participants with no formal
training in the domain of interest. However, finding
participants with the right level of familiarity with
networking tasks and technologies proved
challenging and, as a result, instructive.
We recruited participants through informal
social networks and a formal subject pool of 700
college undergraduates. All participants were paid
$50 for completing a 2-hour, IRB-approved session.
Thirteen men and 13 women (19 – 65 years old)
took part in SB sessions.
All potential participants were given a
screening questionnaire that allowed us to
disqualify individuals who had formal training or
professional experience as a network administrator.
Participants in pilot studies were selected based on
whether they reported having a home network of
sufficient complexity. However, we quickly
realized that selection needed to be based on the
strategies potential participants used for dealing
with network problems. In other words, we needed
active managers rather than simply network users.
Test environment and materials. The SB
sessions required few material resources.
Participants were run in small groups of 2 – 6. Thus,
a small conference room with a single large table,
six movable seats, and a white board was sufficient.
Participants were provided with structured response
sheets for each exercise (steps 5 thru 8 in next
section) during the session.
Test Protocol. Six sessions were
conducted, each with 2 - 6 participants. Most
sessions were conducted by a facilitator and an
assistant; however, sessions could also be
conducted by a single researcher. There were 8
major steps in the procedure:
1) Obtaining informed consent. Each
participant was given an overview of the
tasks they would be asked to perform,
and were given the opportunity to read
and sign an IRB-approved consent form.
2) Encouraging creativity. As with most
SB procedures, we encouraged
participants to be creative in their
discussion. We reviewed five common
creativity tips (i.e., avoiding habitual
responses, generating multiple ideas,
being tolerant of ideas, avoiding
functional tunnel vision, and having
fun). These tips were posted in a
prominent place in the seminar room for
the rest of the session.
3) Providing context. Participants were
given a depiction of a complex home
network with a number of connected
devices, such as an electric car, a
thermostat, a water heater, a security
system, and various computers (see
Figure 1).
4) Defining scope. Our SB protocol
differed from the typical approach by
carefully delineating the scope of the
discussion. In pilot tests, participants
mainly focused on networked devices
and service providers. Our goal was to
focus their discussion on the process of
managing a home network. Thus, we
gave them the following list of
management tasks as a framework –
setting up, diagnosing problems,
monitoring usage, troubleshooting
Figure 1. Illustration used to provide participants with a sha
r
model of the range of possible devices that could comprise a
home network.
Proceedings of the Human Factors and Ergonomics Society 2016 Annual Meeting 1285
efficiency problems, optimizing network
performance, and securing the network.
5) Listing likes and dislikes. The first
exercise was performed as a group.
Participants were asked to describe what
they liked and disliked about current
home networking systems, keeping in
mind the major tasks they might
perform. Each new like/dislike was
added to the white board. When the
group could think of no more issues to
add to the list, they were told to
independently pick their most important
3 likes and dislikes. Participants were
also asked to generate a list of their
concerns about network security and
then to rate their degree of concern in
two ways – 1) the degree of threat to
society, and 2) the degree to which they
were concerned for themselves.
Elicitation of security concerns was
added to the protocol after pilot testing
revealed, somewhat surprisingly, that
few participants spontaneously
mentioned security.
6) Imagining the ideal system. The group
was broken into dyads for the second
major exercise. When an odd number of
participants attended a session, one
participant completed this phase
independently. Participants were asked
to imagine an ideal network system for a
home environment and think about how
it would work. Each dyad completed a
response sheet that asked for brief
statements describing their preferences.
The sheet also included a reminder of
the overarching tasks that comprise
home network administration (discussed
above in the “defining scope” section).
The information from the response
sheets were integrated into an overall list
that was added to the white board by the
facilitator during a five-minute break.
7) Creating a “Top 10” list. The two final
exercises were performed independently
by participants. Participants were asked
to study all lists generated in previous
exercises and choose the 10
characteristics they believed were most
important.
8) Rating satisfaction and importance. In
the final exercise, participants rated the
importance of each characteristic or
issue in their own top 10 list. They also
rated their satisfaction with the way their
current system dealt with the issue. Both
ratings were on an 11-point scale (0 to
10 with lower numbers meaning less
importance/satisfaction).
FINDINGS
Security concerns. Security ratings were
treated descriptively because the manner in which
the individual concerns were elicited and rated (at
the group level) resulted in too few ratings to justify
statistical treatment. Table 1, however, reveals an
important trend. The check mark in either the
‘society’ or ‘personal’ column indicates whether
each concern was more frequently seen as a society-
level problem or a personal one. Most concerns
were seen as societal rather than personal (i.e.,
representing the sentiment that, “It’s a problem, but
I don’t worry about it myself.”). The security
concerns that were believed by participants to
directly pose a personal threat were basically related
to internet and device access (“forgetting
complicated passwords” and “Wi-Fi goes down”)
and privacy (“shared devices and display of
personal information” and “router ‘memorizing’
personal data”). Participants were also concerned
about the transfer of malware via Wi-Fi. The
concern with malware was not due to mistrust in
security software, but rather to uncertainty about
whether participants had properly used it.
Home networking requirements. The SB
data from participants’ “Top 10” list were submitted
to a scoring procedure in which their indicated
needs were categorized by basic network
management tasks. We found it necessary to add
three additional categories to accommodate
statements concerning specific devices, service
providers, and “other.”
Table 2 shows the mean satisfaction and
importance ratings associated with the needs
Proceedings of the Human Factors and Ergonomics Society 2016 Annual Meeting 1286
Society Personal
Complicated passwords x
Concern for shared passwords x
Shared devices and display of
personal information x
ip address saved on router
Monitoring of internet use x
Remote help may be malicious x
Wifi is not private x
Passwords linked x
wifi goes 'down' x
Malware transfers via wifi x
Internet provider tracks use x
Illegal downloads x
Unprotected wifi x
Hacking x
Router memorizing personal
information x
Router reveals number of persons
accessing wifi x
Table 1. Security concerns identified by participants for home
networks. Check marks indicate whether the concern is seen
as personal or societal.
Code Importance
Satisfaction
Efficiency 9 5.3
Service 8.7 4.7
Security 8.7 6.1
Devices 8.5 5.1
Monitoring/Usage 8.4 3.8
Connectivity 8.3 5.3
Customer Service 8.3 3.3
Other 8 4.8
Troubleshooting 7.6 4.3
Setup 7.5 5.7
Optimization/maintenance 7.5 3.9
Visualization 6.8 4.2
Diagnosis 5 6.8
Table 1. Mean importance and satisfaction ratings as a
function of task cluster. Red clusters are those identified as
important but unsatisfied (design opportunities).
grouped with each management task. In other
words, each category is associated with specific
suggestions that are not reported here. Rather, our
focus is on identifying clusters of suggestions, here
identified with management tasks, that should be
the focus of future development. Not surprisingly,
almost all of the tasks were rated as important or
highly important given that participants were
focusing on their “Top 10” requirements. Notably,
only “diagnosis” functions were deemed as
anything other than important, possibly because
participants saw these functions as something that
home network managers should not need to do
themselves. The key variability among the tasks
was with the current satisfaction expressed by
participants. Needs associated with those tasks
viewed as both important and low in satisfaction are
the ones that should be prioritized for future
development. As an example, participants
expressed dissatisfaction with their current ability to
monitor usage patterns within their networks,
something they also saw as highly important. This
finding supports a variety of current efforts to
develop displays of bandwidth use and network
traffic (e.g., Chetty et al., 2010).
Other topics rated as important and low in
current satisfaction were items related to enhancing
efficiency and optimization. Participants desired
network administration that provided for auto-
connection of devices, device synchronization, and
password memorization. As optimization
characteristics, initial network setup by a trained
person was deemed important. Generally, there was
a desire for a passive approach to maintenance with
characteristics such as push notifications for time-
critical maintenance tasks.
Customer service also showed up as an
important cluster of needs that was currently unmet.
This may reflect, among other things, the
miscommunications that occur in support calls and
that result from the language barrier for home
administrators who do not use formal network
management terms. Service dissatisfaction was
characterized by limited Wi-Fi range in the home
and having to re-connect devices.
Note that the privacy cluster is rated
relatively high on satisfaction. This may once again
reflect participants’ lack of personal (as opposed to
societal) concern about security on their own home
networks.
DISCUSSION
After a number of initial SB pilot sessions,
we found that we could obtain useful information
from less than expert home network user-managers,
although we were forced to modify the typical
procedure used with domain experts. First, we
needed to provide context about the system we were
Proceedings of the Human Factors and Ergonomics Society 2016 Annual Meeting 1287
exploring, in our case by providing participants with
a simplified diagram of the interaction among
devices on a hypothetical home network. Second,
we needed to provide guidance about the scope of
the discussion. In our particular case, we asked
participants to consider a number of core home
network management tasks and what they liked and
disliked about the way they currently accomplished
each.
Perhaps most importantly, screening
participants required a careful assessment of the
role played by individuals with respect to home
network management. Although many people will
endorse statements about having and using home
networks with various numbers of connected
devices, fewer users find themselves in the positon
of managing it, by choice or default. With these
users that do serve as home administrators, there are
also difference in the motivation of individuals to be
proactive, engaged managers. For the SB technique
to be useful, individuals must at least have
attempted some of the common home networking
tasks; it is not enough to simply be the person who
calls tech support when something goes awry.
Our data, which could have been collected
in a single day once the protocol was well
developed, highlighted user needs similar to those
elicited through more time-intensive field methods,
including a need to support communication in
customer service encounters, a desire to monitor
home network usage patterns, and a desire to
increase network efficiency.
Not all identified needs required expensive
fixes. Participants identified some very low-
hanging fruit when it came to actual network device
usability. For example, we repeatedly found that
participants had difficulty identifying their routers.
This problem was likely due to some users being
unfamiliar with the label itself, even if they had
their own informal name for the device and
understood its function. For other users, the
physical form of the device did not meet their
expectations and it was also unlabeled. Thus,
developing a label, one that communicated the
device’s function and connected its function to the
standard device name, should be a priority.
ACKNOWLEDGMENT
This research was supported by NSF Grant IIS-
0904350, “
NetSE: Medium: Collaborative Research:
Towards Human-Network Interaction (HNI) for the Home.”
K.L. Calvert was the Principle Investigator.
REFERENCES
Byrne, G., & Barlow, T. (1993). Structured
brainstorming: A method for collecting user
requirements. Proceedings of the Human
Factors and Ergonomics Society Annual
Meeting, 37, 427-431.
Chetty, M., et al. (2010, April). Who's hogging the
bandwidth: the consequences of revealing
the invisible in the home. In Proceedings of
the SIGCHI Conference on Human Factors
in Computing Systems (pp. 659-668). ACM.
Franzke, M., & McClard, A. (1996). Winona gets
wired: technical difficulties in the
home. Communications of the ACM, 39(12),
64-67.
Grinter, R. E., et al. (2009). The ins and outs of
home networking: The case for useful and
usable domestic networking. ACM
Transactions on Computer-Human
Interaction (TOCHI), 16(2), 8.
Poole, E. S., Edwards, W. K., & Jarvis, L. (2009).
The home network as a socio-technical
system: Understanding the challenges of
remote home network problem
diagnosis. Computer Supported Cooperative
Work (CSCW), 18(2-3), 277-299.
Teger, S., & Waks, D. J. (2002). End-user
perspectives on home networking.IEEE
Communications Magazine, 40(4), 114-119.
Proceedings of the Human Factors and Ergonomics Society 2016 Annual Meeting 1288
