Content uploaded by Brad A. Myers
Author content
All content in this area was uploaded by Brad A. Myers on Jan 08, 2015
Content may be subject to copyright.
Future of End-User Software Engineering:
Beyond the Silos
Margaret M. Burnett
School of Electrical Engineering and Computer Science
Oregon State University
Corvallis, OR 97331 USA
burnett@eecs.oregonstate.edu
Brad A. Myers
Human Computer Interaction Institute
Carnegie Mellon University
Pittsburgh, PA 15213 USA
bam+@cs.cmu.edu
ABSTRACT
End-user software engineering (EUSE) is a research area that
aims to invent new kinds of technologies that collaborate with end
users to improve the quality of their software. The practice that
EUSE research aims to support is end users using new tools and
methods to improve the quality of the software that they and other
end users have created. There is a need for this outcome because
research shows both that the number of end users creating their
own software greatly exceeds the number of professional software
developers, and that the software they create is riddled with errors.
In this paper, we describe the present state of EUSE, and
challenges in moving forward toward a bright future. We show
how the future of EUSE may become over-siloed, restricting
future researchers’ vision of what can be achieved. We then show
that focusing on the in-the-moment intents of end-user developers
can be used to derive a number of promising directions forward
for EUSE researchers, and how theories can help us further de-
silo future EUSE research. Finally, we discuss how overcoming
challenges for the future of end-user software engineering may
also bring direct benefits to the future of “classic” software
engineering.
Categories and Subject Descriptors
D.2 [Software Engineering]. H.1.2 [Models and Principles]:
User/Machine Systems.
General Terms
Reliability, Human Factors, Theory
Keywords
End-user software engineering (EUSE), end-user development
1. INTRODUCTION
Software engineering is no longer solely applicable to
professional software engineers. Anyone who develops software,
whether as a professional developer or not, may well need to
employ beyond-coding processes to help ensure that the software
they develop is of sufficiently high quality for the purpose for
which they intended it.
Who, besides professional developers, actually develops
software? According to data from the U.S. Bureau of Census and
Bureau of Labor, more than 12 million people in the U.S. say they
do programming at work, and more than 50 million use
spreadsheets and databases—compared to only about 3 million
professional programmers in the United States [54]. Thus,
depending on how conservatively one differentiates “use” from
“program in”, the number of end-user programmers in the United
States falls somewhere between 12 million and 50 million
people—several times the number of professional programmers.
Clearly then, end-user programming empowers—it has already
empowered millions of end users to create their own software.
Nardi defined end-user programmers as being distinct from
professional developers in that end users’ programs are not the
end in itself, but rather a means to accomplish their own tasks
[39]. There are a variety of research systems and tools aimed at
this audience; examples include an accountant creating
spreadsheet formulas (which are computational instructions) to
keep track of a budget in Excel or a web-literate end user building
a quick mashup to help sort real estate based on custom criteria.
Note that, although Nardi’s definition does not define all end-user
programmers as lacking formal computer science training, many
subpopulations of end-user programmers do lack such training.
In this paper, we consider the present and future of an area of
software engineering that addresses populations like these. The
future of this area that we envision breaks down the silos that have
begun to emerge. We focus on where these silos are, and the
benefits that could ensue to end-user software engineering
research in the absence of these silos.
1.1 End-User Programming versus End-User
Software Engineering
Some examples of the “programming environments” used by end
users are shown in Figure 1. For example, CoScripter [38] (Figure
1(a)) allows web users to automate web-based tasks by
demonstrating the desired sequence of actions on the web pages.
AgentSheets [46] (Figure 1(b)) is aimed mainly at children and
teachers. These AgentSheets users demonstrate “rules of
behavior” for different characters in a game or simulation by
placing the characters onto the game board and demonstrating
both the circumstance that requires the new behavior and the
behavior itself. LabVIEW [25] (Figure 1(c)) aims at a different
audience: scientists and engineers doing sensor-based
computation. LabVIEW users wire together nodes representing
various sensors and measurement instruments to construct
dataflow diagrams defining the desired computations. Finally,
TouchDevelop [36] (Figure 1(d)) enables end users to create
mobile apps of their own by choosing from applicable actions
from a continually updating “smart” keyboard.
Using systems like these, end users create a variety of programs to
use on their desktops, phones, and the web. (Following Ko et al.,
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
FOSE'14, May 31 – June 7, 2014, Hyderabad, India.
Copyright 2014 ACM 978-1-4503-2865-4/14/05…$15.00.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from Permissions@acm.org.
FOSE’14, May 31 – June 7, 2014, Hyderabad, India
Copyright 2014 ACM 978-1-4503-2865-4/14/05...$15.00
http://dx.doi.org/10.1145/2593882.2593896
201
we define a program as “a collection of specifications that may
take variable inputs, and that can be executed (or interpreted) by a
device with computational capabilities” [29]. The variability of
input values requires that the program be able to execute on future
inputs, which is one way it is different from simply doing a
computation once manually.) Some ways in which end users
create these programs include writing and editing mathematical
formulas, dragging and dropping objects onto a logical
workspace, typing and editing textual code, sketching user
interface states and indicating “hot spots” and transitions,
connecting objects in a diagram, or demonstrating intended logic
to the system.
Unfortunately however, the software that end users are creating
with these techniques is riddled with errors. Evidence abounds of
the pervasiveness of errors in the software that end users create
(e.g., [19]). These errors can have significant impact. For
example, after Hurricane Katrina, dozens of end users created web
sites to allow updates about survivors’ last known locations [52].
Confusion reigned with these sites. Data on one site were often
missing or inconsistent with data on the next site. Leveraging data
across sites was not possible, due to differing data formats and
uncertain quality [52]. Turning to a 2011 example from the
European Spreadsheet Risks Interest Group’s huge collection of
spreadsheet errors, a formula error caused the village of West
(a) CoScripter [38], used for web automations. (Left): script. (Right):
Web context for the script. (Original screen shot.)
(b) AgentSheets [46], used for agent-based games and
simulations. (Top): Program state that includes some truck
objects. (Bottom): A visual rule that controls truck behavior.
(Excerpted from http://sgd.cs.colorado.edu/wiki/Frogger_Tutorial
under Creative Commons License
http://creativecommons.org/licenses/by/3.0/)
(c) A dataflow diagram in LabVIEW [25], which is used primarily for
lab applications. (Excerpted from
http://upload.wikimedia.org/wikipedia/commons/b/b6/WikipediaFPandB
D.png, under Wikipedia Commons License
http://commons.wikimedia.org/wiki/Main_Page.)
(d) The mobile version of TouchDevelop [36] for developing
mobile apps.
Figure 1: Several end-user programming environments.
202
Baraboo, Wisconsin to underestimate the cost of a borrowing
decision by about $400,000 [19, case FH1218). Even when errors
in end-user-created software are non-catastrophic, however, their
effects can matter. For example, web applications created by
small-business owners to promote their businesses do just the
opposite if they contain broken links or pages that display
incorrectly, resulting in client frustration, loss of revenue, and/or
loss of credibility [1].
Given that so much of end-user-created software suffers from
quality too low for the purposes for which it was created, there is
a clear need for researchers and tool makers to address this. From
much prior research in classic software engineering, we know that
to improve quality, the software lifecycle beyond just the “create”
stage must be supported. That is, end-user developers, like
professional developers, must be supported beyond the coding
stage of their software development.
To help end-user developers do just that, a new research area
emerged about a decade ago to address this problem. The area is
known as end-user software engineering (EUSE). As Ko et al.
summarize, end-user software engineering technologies are
“technologies that collaborate with end users to improve software
quality” [29]. Thus, the differences between end-user
programming and end-user software engineering are that end-user
programming focuses mainly on how to allow end users to create
their own programs, whereas end-user software engineering
considers how to support the entire software lifecycle and its
attendant quality issues. Differences between end-user software
engineering and traditional software engineering are several, but
they all stem from a difference in population.
When taking this lifecycle view of end users’ software, the term
end-user developer becomes more appropriate than end-user
programmer, drawing upon Lieberman et al.’s definition of end-
user development: “a set of methods, techniques, and tools that
allow users of software systems, who are acting as non-
professional software developers, at some point to create, modify
or extend a software artifact” [37]. This terminology points out
that end users involved in software development may or may not
actually create new code as part of what they do in the software
development process. For example, they might participate in
activities like configuring their software, co-designing their
software side-by-side with professional programmers, and
tailoring their software [15, 17, 20, 26, 61]. Because the term
“end-user developer” implies participation in any software
development lifecycle stage—not restricted to just creating
code—for the remainder of this paper we will use the term end-
user developers .
1.2 A Future of End-User Software
Engineering: Challenges in De-siloing
EUSE research is at a crossroads. Following too closely in the
footsteps of its classic software engineering predecessors raises a
risk of over-siloing the area, to the detriment of the end-user
developers it aims to serve. When we refer to siloes, we mean “a
system, process, department, etc. that operates in isolation from
others” [40] (italics added).
In one sense, end-user software engineering de-silos already: it
removes the silo that fenced off software engineering benefits
from everyone except professional software developers. However,
end-user software engineering has not gone much further in de-
siloing, and we believe the area could benefit from a great deal
more. For example, it primarily considers end-user developers
only as (atomic) people, rarely anything smaller or larger;
considers software development tasks only one lifecycle phase at
a time; and suffers from siloing individual tools and findings
rather than building connective tissue between them. Such silos
raise risks that an area can become overly narrow, building upon
research in one silo or another, but losing the potential for big
steps forward that can enable a research area to transform.
Thus, this paper sets out a number of de-siloing challenges that we
hope will influence the future of end-user software engineering.
We consider them from the perspectives of who, when, why and
how.
2. END-USER SOFTWARE
ENGINEERING’S “WHO”
A key difference between end-user software engineering and
traditional software engineering is the kind of developers served.
A premise of classical software engineering research is that
professional developers have some training and/or experience in
software engineering methods, and also at least a little interest in
following good software engineering practices. In contrast, end-
user software engineering research cannot assume the presence of
any of these.
From this difference, two requirements become clear for end-user
software engineering tools and techniques. First, EUSE tools and
techniques must be designed for audiences that are usually
unfamiliar with professional software engineering practices (even
though they may have programmed before). Second, these tools
and techniques must fit well with their audience’s non-software-
engineering-oriented interests, motivations, and practices.
2.1 Today’s End-User Developer Target
Audiences
Fulfilling these requirements requires considering more deeply
who the target audiences are. The term “end-user programmers”
was once used synonymously with “novice programmers” to
mean anyone with a lack of experience programming regardless
of their motivations, but that trend largely disappeared after
Nardi’s clarifying book on end-user programming [39]. Still, in
today’s end-user software engineering literature, assumptions
about the population of end-user developers to be served are
sometimes left implicit, and differ from paper to paper. For
example, some researchers use the term to mean people who have
neither experience nor interest in software development per se
(e.g., [50]), others use it to mean anyone using a particular
programming environment that targets end users (e.g., [6, 36]),
and others use it to mean people who can program in one
particular application area only (such as end-user webmasters
[47]).
Such assumptions are true of some end-user developer
populations, but not all of them. For example, counterexamples to
many of the above assumptions can be seen in Scaffidi et al.’s
survey responses from 831 information workers (subscribers to
Information Week) [51]. As Table 1 shows, these users—77% of
whom were in non-technical jobs—did a surprising amount of
programming, often using only ordinary office productivity tools.
One reasonable way to establish what assumptions to make about
a target end-user developer population is to empirically study the
target audience as part of designing EUSE tools for that audience,
and some EUSE research has moved in this direction (e.g.,
accountants using spreadsheets [59] and web designers [41]). An
advantage of informing EUSE tool designs by empirically
studying a particular population is that it enables EUSE
researchers to fulfill the two requirements enumerated at the
203
beginning of Section 2: namely, not relying upon familiarity with
software engineering, and fitting with the target audience’s
motivations and practices. However, a disadvantage is that it can
encourage siloing EUSE research by population. Further, each silo
(population) must be pre-identified and researched on its own,
without including other end-user developer populations.
2.2 Challenge: Targeting a More Nuanced
“Who”
A de-siloing alternative can be derived from Nardi’s original
definition. Recall that, according to Nardi [39], the distinguishing
factor of professional programmers vs. end-user developers is a
difference in intent, not a difference in experience. Indeed, Nardi
goes on to explain that end users are not necessarily “casual,”
“novice,” or “naive” users. Ko et al. emphasized this notion by
defining end-user programming (and end-user development) as a
role (a close cousin to high-level intent) rather than a person’s
semi-permanent identity [29].
Defining end-user developers based on intent allows a more
nuanced version of “who”, thus widening the population of those
who could be served by end-user software engineering research.
Roles can include not only traditional populations of end-user
developers like teachers and spreadsheet users engaging in
software development, but also populations who, although they
program in various ways fairly regularly, still have little interest in
software engineering in at least some of their software
development tasks.
As an extreme example, a professor of software engineering might
be systematic and rigorous in validating software created for
distribution to other researchers, but ad hoc about validating a
grant budget spreadsheet. Indeed, an ad hoc view of beyond-
coding software engineering tasks has been shown to arise often
across a wide span of end-user developers1, from scientists and
mathematicians who program often in traditional languages [56]
to spreadsheet users [59] to teachers [63] to museum exhibit
designers [7].
The decoupling of intent from personal identity helps to de-silo
end-user software engineering because it removes the need to
separate end-user developers as people into groups: instead,
researchers can focus on intent, which slices across many
different classes of end-user developers. Intent as a central
concept ties end-user software engineering research directly to the
end-user developer’s motivations and values in that particular
software development task, which in turn can affect their software
engineering processes.
2.2.1 Intent as a basis for addressing the challenge
Moving to intent as the key definer of when someone is acting as
an end-user developer suggests the possibility of raising intent to a
first-class factor in defining future EUSE tools and techniques.
One way focusing on intent can help advance EUSE research is
by revealing flawed assumptions in EUSE work. One common
assumption is “if we build it they will come.” That is, if a new
tool or feature becomes available that can improve users’
correctness or efficiency, then surely conscientious users will use
it. However, a focus on intent clarifies that this assumption may
not hold for end-user developers.
For example, what might a laboratory scientist’s intent be, when
working on a laboratory application to collect and measure data
from a variety of sensors? Some possibilities might be “build my
lab application” or “fix my broken lab application” or “add new
capabilities to last year’s lab application.” Note that none of these
intents sounds much like “find a new tool that might give me
some useful new ways to guard the quality of my application.”
Since intent resides in people’s heads, understanding its
implications on EUSE is facilitated by turning to applicable
theories of how people think. Blackwell’s Model of Attention
Investment [5] is a model applicable to this situation; it helps to
explain how end-user developers make choices about what
problem-solving tools and features to use in computer systems.
Blackwell’s model rests on three constructs: the users’ perceptions
of the costs, benefits, and risks they will face in completing their
task (accomplishing their intent) one way versus some other way.
According to Blackwell, users expect that exploring an unknown
tool or feature will have a cost, some benefits, and some risks. For
example, in the lab example above, suppose the scientist’s usual
way of accomplishing such tasks is to use the environment’s
editor. If something goes wrong, the scientist can fix it using the
editor. This has no learning cost, and if the scientist believes s/he
is succeeding, it seems to have the benefit of accomplishing the
task with no risk. Only if the scientist realizes that there are
problems too great to fix that way (i.e., there is high risk that the
benefits will not be realized) is s/he likely to consider switching to
a new unknown tool.
Now suppose the scientist decides that a more powerful problem-
solving tool is needed and, in looking around the environment,
sees a tool that might help. The scientist ventures to open it up,
and is now faced with a first impression of the cost of learning the
tool and of the cost of using it for the particular problem at hand.
1 From now on, we will use this phrase as an abbreviation
meaning “developers operating in an end-user development
role.”
Table 1. Programming-like features, in a variety of
common software applications, used by at least 33% of the
information workers [52].
Application
Used by
(% of
users)
Feature
Feature
used by
(% of
users)
Slide Editors
& Word
Processors
96.1
Create document templates
73.3
Make inter-document hyperlinks
53.7
Record desktop editor macros
33.3
Spreadsheets
93.1
Use functions (e.g., “sum”)
89.3
Create charts in spreadsheets
80.1
Inter-spreadsheet references
66.2
Create spreadsheet templates
50.4
Record spreadsheet macros
42.3
Create/edit spreadsheet macros
38.6
Databases
79.3
Reference records by key
74.0
Create tables
71.7
Create database views
68.1
Create stored procedures
49.5
Web Pages
68.6
Create hyperlinks
65.2
Create web forms
52.2
Create web page scripts
42.9
Create JavaScript functions
34.3
204
This first impression is the perceived cost the scientist is
imagining. S/he is also considering the perceived risk
(probability) that going down this path will be a waste of time or,
worse, will leave the application in a mess from which it would be
difficult to recover. If users perceive an EUSE tool’s costs or risks
to be greater than the perceived benefits, they are unlikely to use
the tool [5, 50].
Viewing intent in light of Blackwell’s Model of Attention
Investment points out that EUSE tools and techniques can target
an intent-nuanced “who” by (1) connecting to a target end-user
developer’s intent whenever it may arise, and (2) using that
moment to entice them to take appropriate software engineering
steps when the need arises by making clear from the outset the
upfront and ongoing costs, benefits, and risks that are likely.
One approach to accomplishing these two steps is to explicitly use
inferences of users’ intentions as starting points of EUSE support
tools. Another approach is to use curiosity-based enticement to
draw end users into engaging at an appropriate moment with
software engineering features. A third is to base tools on past
intents of the target audience as predictors of “common” intents.
Although possible intent-focused approaches to EUSE techniques
are relatively unexplored, the next section describes a special case
with promising beginnings in that direction.
2.2.2 A special-case “who”: EUSE & learning
An interesting case study in de-siloing by moving from “who” as
people to “who” as roles has occurred in the realm of learning.
Originally, tools for helping people learn software development
skills aimed at teaching programming (e.g., syntax, semantics, and
basic programming skills) to people who wanted to learn
programming. However, a relatively recent trend is to reach out to
people who may not be interested in programming to try to
convince them to want to program—i.e., to manipulate intent, or
in the terminology of the Blackwell model, by manipulating users’
perceptions of benefits. A system can manipulate intent in (at
least) three ways: (1) it can build new intent, (2) it can serve
existing intent (if intent is known), and/or (3) it can attempt to
infer intent, often as a precursor to (1) or (2).
The aim of a system building new intent is to reach out to users
who are motivated by some benefit other than “a program of my
own.” Such systems present programming as a means to that other
benefit, with the hoped-for result that the users then decide to use
the programming system in order to attain that benefit. One
example is to “gamify” the process of learning programming (e.g.,
Pex4Fun [60] (Figure 2(a)) and Gidget [35]), leveraging the
drawing power of a game to engage users in learning
programming as a side effect of playing the game.
A system can build intent to learn other software development
skills too, not just coding. For example, a few researchers have
worked to draw users toward higher success at design. Besides
offering a non-software-development benefit (like telling stories,
or designing their own web pages), these works feature
scaffolding to reduce users’ perceptions of the costs of achieving
the benefit they seek. For example, Kerr et al. provided story
templates to help users with story design ideas in the setting of
StoryTelling Alice’s successor [27], and Lee et al. provided
manipulable examples to help end users explore web page design
ideas [34]. Approaches like these can help improve end-user
developers’ software development skills.
Approaches like the ones discussed so far have been shown to be
effective for users who are at least open to expanding their
software development skills. However, many end users are
reluctant to do so—resistance to various software development
tasks has been reported for a variety of end-user developers (e.g.,
[14, 42, 56, 63]). As these reports show, to end-user developers
like these the benefits seem low, and/or the costs and/or risks
seem high.
Fortunately, the keys to attracting even reluctant learners to learn
software development skills become clear from the ideas of intent
and the cost/benefit/risk factors of the Attention Investment
model. That is, a system can introduce appropriate software
development practices to end-user developers by serving their
actual intents (if known) to provide these users with the benefits
they seek or to reduce their risks or costs in a way that also shapes
their software development behaviors.
For example, the Whyline research began by first researching
common debugging intents empirically. End user developers’
most common debugging intents were harvested by investigating
their most common questions. The investigation showed that their
(a) Pex4Fun’s gamification [60] can build intent to program in
people who like to play games. (Original screen shot.)
(b) Whyline (Alice version) [31] directly serves an existing intent
that has been reported empirically. Here, the user asks “why
didn’t Pac resize 0.5?” (Original screen shot)
(c) Idea Garden [10] first infers intent and cost, and then offers
help to reduce the cost of achieving that intent.
Figure 2: Three ways systems can build upon intent.
205
most common questions were “why” and “why not”, so the
Whyline was developed to allow users to ask those questions
directly [31], and to answer them through scaffolded views of the
kinds of problem-solving approaches professional developers use,
such as tracing through calling sequences with their parameter
values. See Figure 2(b) for a Whyline view of a calling sequence
with parameter values.
Serving existing intents can also be combined with building new
intents. For example, Surprise-Explain-Reward [64] is a strategy
to entice users toward desirable software engineering behaviors.
The strategy attempts to arouse the end-user developer’s curiosity
about some aspect of their software that seems to be erroneous
(such as a faulty cell formula), by decorating the cell if the tool
has something to say about it. If the software’s accuracy is
important to that developer, they may become curious and follow
up. The decoration (notification that the software has something to
say) is the surprise, the information they get by following up (e.g.,
by hovering over the decoration) is a brief explanation, and the
reward is (ideally) a found error and a sense that there is less risk
of the software producing incorrect answers. Surprise-explain-
reward has been used successfully in several EUSE settings; one
example was to entice end users into incrementally entering
assertions to guard spreadsheet correctness [8].
Another intent-oriented strategy is to target the end-user
developer’s intent by inferring it or asking about it in context [10,
12]. For example, the Idea Garden (Figure 2(c)) infers intent from
context, and also infers the costs of behaviors the end-user
developer may be exhibiting, and then provides help that will
reduce the user’s difficulties (cost) of accomplishing that intent
[10].
The approaches discussed here to de-siloing the learning of EUSE
practices share a set of common attributes: (1) targeting end-user
developer intent, often as a “right now” moment, (2) drawing end-
user developers into doing just a little of the desired behavior at a
time when it can benefit them, and (3) giving them the supports
they need, just in time and in their own environment, so that they
can succeed at the software engineering behaviors they are
engaging in.
These approaches generalize beyond learning settings, as some of
the above examples have already demonstrated. That is, building
new intent by serving an existing intent does not have to be
leveraged only in “learning systems” or “learning modes.”
Approaches like these can be included in any system that end-user
developers use, to encourage incremental, life-long improvement
of their software development skills, in the contexts of doing their
ordinary work.
3. WHEN EUSE CAN HAPPEN
From a lifecycle perspective, end-user software engineering
begins, by definition, by moving beyond the coding phase silo to
connect it with other phases of the software lifecycle.
3.1 Today’s EUSE: Beyond Coding
Researchers in end-user software engineering have devised a
variety of techniques to support software engineering activities
beyond coding. Perhaps the first such example of explicit support
beyond coding was the “What You See Is What You Test”
(WYSIWYT) research to help end-user developers systematically
test their spreadsheets [49, 21]. WYSIWYT incrementally tracks
test coverage of the spreadsheet formulas (via definition-use
association coverage of the formula subexpressions), and
colorizes the cell borders on a continuum from red (no test
coverage yet) to blue (fully covered). At any time, if the user
notices that a value (like the “Max of...” cell in Figure 3(a)) is
correct, she can check it off, and the coverage is automatically
recalculated so that corresponding cell border colors update
immediately in the spreadsheet.
There has also been research to support other stages of the end
users’ software development lifecycle, such as design [48],
assertions [8, 45], reverse-engineering [16], refactoring [3, 58],
and explicitly guarding against or detecting errors and software
“smells” [2, 18, 24, 53]. Figure 3(b) shows an example of the
Topes approach [53], which guards against errors.
These approaches are steps forward in bringing to end-user
developers some of the benefits of software engineering
methods—but they do not go far enough. Most of these
approaches address each lifecycle stage in isolation (e.g., just
refactoring, or just designing) so, while they break out of the
“coding” silo, they then erect new silos for each lifecycle phase.
3.2 Challenge: Beyond “Beyond Coding”
Such silos for each lifecycle phase do not match the opportunistic
workstyle of many end-user developers [7, 56]. In developing
their software, exactly when are end-user developers doing
design, when are they testing, and so on?
For example, consider a teacher who wants to create a grades
spreadsheet, and suppose she begins by entering a few column
headings across the top (say, “ID” and “name”), and then typing
in all her students’ IDs and names into the corresponding
columns. So far she has not entered a single “line of code”
(formula)—so is she in the design stage? In one way she is,
because she has begun to specify the structure of the spreadsheet,
(a) The WYSIWYT testing approach for end-user developers [49,
21]. (Original screenshot.)
(b) Topes allow end-user developers to define and error-check
data inputs [53]. (Original screen shot )
Figure 3: Examples of beyond-coding approaches.
206
and indeed, according to design theorist Schön, she is doing
design according to the reflection-in-action model [11, 55].
However, from a traditional software engineering standpoint, she
is not, because she has not specified any relationships or logic, yet
has already skipped ahead to “data entry.” Suppose she next
“implements” a little of the program (enters several formulas), and
simultaneously “tests” a couple of the formulas (glances at their
values based on the inputs she put in), and decides things are
going well.
This opportunistic mix of several elements from the software
lifecycle is the way many of today’s end-user developers with no
prior software engineering training work [7, 56]. Without support
for navigating seamlessly among lifecycle phase “moments” in
their own environments, these end-user developers may never
choose to engage in tools that could help them. (Recall our
emphasis on developers’ intents in Section 2.) Yet, as the software
they create grows in complexity or criticality (e.g., [19, 57]),
sometimes they need these supports to attain the quality their
software needs.
At a minimum, then, EUSE tools need to be integrated into the
work styles and environments these developers use. To do so
requires tool builders to ensure that the tools they build for
various lifecycle stages are interactive and incremental, so that
end-user developers can easily do a little using one lifecycle
stage’s tool, then switch to another to do a little, then switch back.
This may require entirely different reasoning algorithms than have
used in the lifecycle phase counterparts in classic software
engineering. For example, some EUSE efforts have been to
develop new EUSE static and dynamic reasoning algorithms that
are highly incremental and responsive so as to support exactly this
kind of incremental phase switching (e.g., [23, 32, 49]).
Still, from a de-siloing perspective, this approach is somewhat
incremental. It is supportive of the fact that end-user developers
will need to rapidly and intermittently switch in and out of their
tools, but still silos the EUSE tools and research efforts by
lifecycle phase.
An alternative approach would be a more dramatic departure from
classic software engineering approaches. We believe it has more
potential to transform but also brings more challenges. In
essence, it is to promote end-user developers’ work styles and
intents as a way to define our vision, giving EUSE researchers the
potential to transcend one-lifecycle-phase-at-a-time research silos
in favor of bigger visions.
One way to go about such an approach would be to frame the
EUSE problem more along the lines of the problem-solving
activity rather than the software engineering lifecycle. To see
how this can be done, we describe an emerging EUSE research
project (in which we are involved) that characterizes its target as
supporting “exploratory programming” by end-user developers
(http://www.exploratoryprogramming.org/).
The project targets opportunistic end-user developers who are
exploring possible ways to achieve their development goals,
working to understand as much as needed of unknown code
fragments, and exploring ways to fit some of the pieces together.
Some code fragments may come from the user’s own files, and
others discovered on the web. Some may have been written by
this user, some by other end-user developers, and some by
professional developers. As things go wrong, the end-user
developer may need to revisit some variants tried before that were
working.
For example, consider “Frieda“, an office manager in charge of
her department’s budget tracking. (Frieda was a participant in a
set of interviews with spreadsheet users that the first author
conducted. Frieda is not her real name.) Every year, the company
she works for produces an updated budget tracking spreadsheet
with the newest reporting requirements embedded in its structure
and formulas. But this spreadsheet is not a perfect fit to the kinds
of projects and sub-budgets she manages, so every year Frieda
needs to change it. She does this by working with four variants of
the spreadsheet at once: the one the company sent out last year
(we will call it Official-lastYear), the one she derived from that
one to fit her department’s needs (Dept-lastYear), the one the
company sent out this year (Official-thisYear), and the one she is
trying to put together for this year (Dept-thisYear).
Using these four variants, Frieda exploratively mixes reverse
engineering, reuse, programming, testing, and debugging, mostly
by trial-and-error. She begins this process by reminding herself of
ways she changed last year’s by reverse engineering a few of the
differences between Official-lastYear and Dept-lastYear. She then
looks at the same portions of Official-thisYear to see if those
same changes can easily be made, given her department’s current
needs.
She can reuse some of these same changes this year, but copying
them into Dept-thisYear is troublesome, with some of the
formulas automatically adjusting themselves to refer to Dept-
lastYear. She patches these up (if she notices them), then tries out
some new columns or sections of Dept-thisYear to reflect her new
projects. She mixes in “testing” along the way by entering some
of the budget values for this year and eyeballing the values that
come out, then debugs if she notices something amiss. At some
point, she moves on to another set of related columns, repeating
the cycle for these. Frieda has learned over the years to save some
of her spreadsheet variants along the way (using a different
filename for each), because she might decide that the way she did
some of her changes was a bad idea, and she wants to revert to try
a different way she had started before.
To support this style of development, the exploratory
programming project is creating a variety of visualizations, search
facilities, and interaction techniques to provide effective ways for
users to find, understand, assess, reuse and create variants, and be
able to ask “why” questions to understand differences between
variations of a system. It will also include supports for helping
end-user developers enhance their problem-solving strategies
when they encounter difficulties.
A lifecycle view of the project would not capture everything here:
exploratory programming includes design, implementation,
testing, debugging, and reuse—but the human activity of
exploration is more holistic than these five different lifecycle
phases. For example, the lifecycle view would not capture the user
backing out of part of a solution, and would not capture a tool
helping the user find a useful problem-solving strategy. Thus, the
exploratory programming example suggests that first-class
consideration of developer intents (such as “explore”) may help
EUSE researchers devise future EUSE approaches that transcend
today’s lifecycle-phase silos.
4. CHALLENGE: WHY AND HOW
Devising new EUSE ideas for tools and techniques that actually
help end-user developers is difficult if we do not understand why
end-user developers might use them or how the new ideas could
work. Understanding these whys and hows has a de-siloing theme
in its ability to connect one type of knowledge with another.
207
One way to connect our tools and techniques with an
understanding of the developers they target is to ground our EUSE
research efforts in formative empirical studies, so as to understand
the target group of end-user developers. The research paradigm
of grounding research ideas with formative empirical work is
common in the EUSE area, and many of the research efforts we
have summarized here have proceeded in this way.
However, this research paradigm still leaves an important gap: a
missing or weak connection among similar EUSE research
projects. To close this gap, we argue for the importance of more
theoretical grounding in future EUSE work. Theory can help
yield a more holistic understanding of how EUSE results fit
together. Although a few EUSE projects have been human-theory-
based, such projects are still relatively rare.
As an example of the benefits such theories can bring to EUSE,
recall from Section 2 how the Model of Attention Investment [5]
provides a generalized way to gain insights into how and whether
an end-user developer might go about deciding whether to even
try a new EUSE tool or feature. Indeed, this model has already
been used to shape and predict acceptance of a variety of EUSE
approaches (e.g., [8, 10, 50, 59, 64]). Informing EUSE
approaches from the outset using a common model (here,
Attention Investment) connects seemingly disparate tools and
approaches through their shared set of fundamentals.
A second way theory can benefit EUSE research is to guide the
pre-study design and post-study interpretation of empirical
studies. For example, Ko and Myers developed a theoretical
framework and methodology for investigating causes of software
errors in programming systems [30]. This framework draws from
a cognitive theory of human errors and situates it with results
from many of the pioneering empirical studies of programming.
The resulting framework and related empirical methodology has
been used to iteratively evaluate and design the end-user
programming system Alice [13], and to then inform the work on
Whyline for Alice [31].
Theory can also connect tools after the fact—even tools that, on
the surface, seem to have little to do with one another. For
example, Information Foraging Theory, although a relative
newcomer to software engineering, has already been used to show
commonalities and design patterns among more than 20 different
software engineering tools spanning three domains: debugging,
refactoring, and reuse [22]. Information Foraging Theory [43],
which is rooted in earlier foraging theories from biology, is a
predator/prey theory that can both explain and predict how
software developers (information predators) go about gathering
the information they need (the prey) when performing
information-intensive software tasks like debugging. With the
exception of emerging work by Kuttal et al. [33], Information
Foraging Theory has not yet been applied to the design of EUSE
tools.
Another predator/prey model, which centers on notions of
ecosystem diversity, shows a fourth way theories can benefit
future EUSE work: using theory-derived measures to understand
why and how empirical measures of developer behaviors connect
with empirical measures of artifacts that might seem to be
surrogates for developer measures [44]. Posnett et al. used this
theory to derive complementary measures of “focus” for software
development (by professional developers): developer attention
focus and module activity focus. By using symmetric measures
and concepts for both human diversity of attention among models
and module diversity of exposure to humans, the theory provides
a new way to unify and to contrast results from data about humans
and data about artifacts. Demonstrating its usefulness, the results
of their empirical case study on the concept of focus captured
nuanced advantages and disadvantages of focusing developer
attention on a small number of modules that prior atheoretic
measures have not been able to expose.
5. DISCUSSION: WHAT EUSE HAS TO
SAY TO “CLASSIC” SOFTWARE
ENGINEERING RESEARCH
EUSE research began with the following overarching research
question: Is it possible to bring the benefits of rigorous
software engineering methodologies to end users?
To investigate this goal, EUSE researchers have built upon a
blend of foundations from software engineering and human-
computer interaction. The results may have implications not only
for future EUSE research, but also for “classic” software
engineering research into how to support professional software
developers.
The challenges of supporting end-user developers to do software
engineering activities lend a magnifying lens to problems that
may also affect professional software developers. A case in point
is the Whyline debugging technique, which was developed
originally for end-user developers in the Alice setting (Figure
2(b)) [31], and was later used as the basis for a Whyline for
professional developers working in Java [32]. Empirical
evaluations of the tool showed that novice Java developers with
the Whyline were twice as fast as expert Java developers without
it.
In addition to the Whyline case, the theory research on software
errors, born in the EUSE setting as discussed in Section 4, has
influenced several mainstream software engineering efforts (e.g.,
[28, 62, 65]). Similarly, research into how gender differences tie
with software tool usage, originally investigated in EUSE [4], was
later shown to be applicable to professional software development
tools as well [9].
At its core, the EUSE vision is of a different kind of partnership
between developers and tools than is sometimes the case in classic
software engineering. In the EUSE partnership vision, a
developer’s intents and work style take priority over what a tool
might “want” the developer to do—because if an end-user
developer does not appreciate the tool’s behavior, the developer is
likely to simply abandon it.
Because of this priority, and also because of its dual heritage from
software engineering and human-computer interaction, EUSE
research is arguably ahead of classic software engineering in
research that focuses squarely on the developers and their intents.
As the examples above demonstrate, bringing these focuses into
classic software engineering research can bring benefits to
professional developers, just as they do to end-user developers.
Indeed, going beyond the examples above, the use of more
advanced intent-oriented approaches such as building intent and
inferring intent (Section 2.2) could be used to entice professional
developers into more productive or effective software engineering
behaviors, or to deliver to them the information they need at the
very moment they need it. Implications for classic software
engineering research then could be to draw upon ideas we have
presented for end-user software engineering, so as to bring
benefits of end-user software engineering research back to
professional developers.
208
6. CONCLUSION
In this paper, we have argued for the benefits of de-siloing to
achieve the kinds of transformative changes that could come
about in EUSE’s future. Specifically, we advocate de-siloing
EUSE research from the perspectives of who, when, why and
how.
EUSE’s who: EUSE research is beginning to move beyond
assumptions of a fairly homogenous population of end-user
developers. Instead, we argue for the more modern notion that
end-user development is a role, or intent, at some particular time,
not a personal identity. An immediate benefit is that de-siloing
who EUSE targets in this way can open the door to new
populations of end-user developers, working alone or together.
More fundamentally, intent-focused approaches can promote
intent to a first-class concept in EUSE, allowing a rich set of
intent-oriented operations, such as attempting to manipulate it,
infer it, target it, or reward it.
EUSE’s when: EUSE already de-silos end-user programming by
moving research beyond the coding phase silo, but it does not go
far enough. We argue that it is not wise, once beyond the coding
silo, to then build a new silo for each phase of the software
lifecycle. Instead, we advocate moving toward research projects
that are not lifecycle-phase-centric, instead focusing on supporting
end-user developers as they actually work—opportunistically,
incrementally, and rarely one lifecycle phase at a time. One
possible approach is to start with the activity as the developer sees
it (e.g., “explore” my options in getting the functionality I want),
rather than starting from a lifecycle phase perspective (e.g.,
“reuse”).
EUSE’s why and how: We advocate a greater incorporation of
theory into EUSE research. Theories enable us to connect multiple
research efforts, even if on the surface they have little in common.
For example, creating multiple new EUSE approaches from a
common set of foundations connects all the approaches based on
those particular foundations. Finding ways to pattern-match
existing tools to a common theory can also connect those tools.
Theory can even be used to connect disparate data sets, if those
data sets revolve around a concept that can be intellectually
unified and symmetrically measured. In short, theories can enable
EUSE researchers to see through separate silos by generalizing
across them.
Realizing these goals will be challenging, and it would not be
realistic to expect these kinds of changes in the short term. Still,
we argue that end-user software engineering, with its dual heritage
in software engineering and human-computer interaction, is in a
good position to eventually achieve them. By doing so, we can
not only change the reliability that end users can expect from the
software they develop for themselves, but at the same time
contribute to their problem-solving skills, to their enjoyment of
their software development activities, and to our own ability as
researchers to have bigger visions, beyond just one silo at a time.
7. ACKNOWLEDGMENTS
We’d like to acknowledge the many researchers whose pioneering
works have helped establish the area of end-user software
engineering. We also thank the students and collaborators who
contributed to our own work in end-user software engineering.
Our work in developing this paper was supported in part by the
National Science Foundation under grants IIS-1116724, CNS-
1240957, IIS-1314356, IIS-1314384, and CCF-1302113. Any
opinions, findings and conclusions or recommendations expressed
in this material are those of the author(s) and do not necessarily
reflect those of the National Science Foundation.
8. REFERENCES
[1] Anonymous. 2013. User-unfriendly PR: Nearly 9 of 10 of
Americans have negative feelings about brands with poorly
performing websites and mobile apps, new SOASTA study
reveals, Bulldog Reporter Daily Dog (June 20, 2013),
https://www.bulldogreporter.com/dailydog/article/user-
unfriendly-pr-nearly-9-of-10-of-americans-have-negative-
feelings-about-brands- (Accessed Dec. 8, 2013).
[2] Abraham, R. and Erwig, M. 2009. Mutation operators for
spreadsheets, IEEE Transactions on Software Engineering,
35, 1, 94-108.
[3] Badame, S. and Dig, D. 2012. Refactoring meets spreadsheet
formulas, IEEE International Conference on Software
Maintenance, 399-409.
[4] Beckwith, L., Burnett, M., Wiedenbeck, S., Cook, C., Sorte,
S., Hastings, M. 2005. Effectiveness of end-user debugging
software features: Are there gender issues? ACM Conference
on Human Factors in Computing Systems (Portland, Oregon,
April 2005), 869-878.
[5] Blackwell, A. 2002. First steps in programming: A rationale
for attention investment models, IEEE Human-Centric
Computing Languages and Environments (Arlington, VA,
Sept. 2002), 2-10.
[6] Bogart, C., Burnett, M., Cypher, A., Scaffidi, C. 2008. End-
user programming in the wild: A field study of CoScripter
scripts, IEEE Symposium on Visual Languages and Human-
Centric Computing (Herrshing am Ammersee, Germany,
Sept. 2008), 39-46.
[7] Brandt, J., Guo, P., Lewenstein, J., and Klemmer, S. 2008.
Opportunistic programming: How rapid ideation and
prototyping occur in practice. ICSE’08 Workshop on End-
User Software Engineering. (Leipzig, Germany, May 2008).
[8] Burnett, M., Cook, C., Pendse, O., Rothermel, G., Summet,
J. and Wallace, C. 2003. End-user software engineering with
assertions in the spreadsheet paradigm, International
Conference on Software Engineering, (Portland, OR, May
2003), 93-103.
[9] Burnett, M., Fleming, S., Iqbal, S., Venolia, G., Rajaram, V.,
Farooq, U., Grigoreanu, V., Czerwinski, M. Gender
differences and programming environments: Across
programming populations, ACM-IEEE International
Symposium on Empirical Software Engineering and
Measurement, (Bolzano-Bozen, Italy, Sept. 2010), Article
28, 10 pages.
[10] Cao, J., Kwan, I., Bahmani, F., Burnett, M., Fleming, S.,
Jordahl, J., Horvath, A., Yang, S. 2013. End-user
programmers in trouble: Can the Idea Garden help them to
help themselves? IEEE Symposium on Visual Languages and
Human-Centric Computing (San Jose, CA, Sept. 2013), pp.
151-158.
[11] Cao, J., Riche, Y., Wiedenbeck, S., Burnett, M., Grigoreanu,
V. 2010. End-user mashup programming: Through the
design lens, ACM Conference on Human Factors in
Computing Systems, 1009-1018.
[12] Cao, X. and Iverson, L. 2006. Intentional access
management: Making access control usable for end-users,
209
Symposium on Usable Privacy and Security (SOUPS), (July
2006, Pittsburgh, PA), 20-31.
[13] Conway, M., Audia, S., Burnette, T., Cosgrove, D.,
Christiansen, K., Deline, R., Durbin, K., Gossweiler, R.,
Koga, S., Long, C., Mallory, B., Miale, S., Monkaitis, J.,
Patten, J., Pierce, J., Shochet, J., Staack, D., Stearns, B.,
Stoakley, R., Sturgill, S., Viega, J., White, J., Williams, G.,
Pausch, R. 2000. Alice: Lessons learned from building a 3D
system for novices, ACM Conference on Human Factors in
Computing Systems, 486–493.
[14] Costabile, M., Mussio, P., Provenza, L., Piccinno, A. 2008.
End users as unwitting software developers. ICSE’08
Workshop on End-User Software Engineering. (Leipzig,
Germany, May 2008). 6-10.
[15] Costabile, M., Mussio P., Provenza L., and Piccinno A. 2009.
Supporting end users to be co-designers of their tools. 2nd
International Symposium on End-User Development (LNCS
5435), (Siegen, Germany, Springer-Verlag, Berlin
Heidelberg, March 2009), 70-85.
[16] Cunha, J., Fernandes, J., Mendes, J., Pacheco, H., Saraiva, J.
2012. Bidirectional transformation of model-driven
spreadsheets, 5th International Conference on Model
Transformation (ICMT ‘12), 105-120.
[17] Eagan, J. and Stasko, J. 2008. The Buzz: Supporting user
tailorability in awareness applications. ACM Conference on
Human Factors in Computing Systems, (Florence, Italy,
April 2008), 1729-1738.
[18] Erwig M., Abraham R., Cooperstein I., Kollmansberger S.
2005. Automatic generation and maintenance of correct
spreadsheets. International Conference on Software
Engineering (St. Louis, Missouri, May 2005), 136-145.
[19] European Spreadsheet Risk Interest Group, EuSpRIG horror
stories: Spreadsheet mistakes news stories,
http://www.eusprig.org/horror-stories.htm (accessed Dec. 8,
2013).
[20] Fischer, G. and Giaccardi, E. 2006. Meta-design: A
framework for the future of end user development. In H.
Lieberman, F. Paternò, & V. Wulf (Eds.), End User
Development (Kluwer Academic Publishers, Dordrecht, The
Netherlands), 427-457.
[21] Fisher, M., Rothermel, G., Creelan T. and Burnett, M. 2006.
Scaling a dataflow testing methodology to the multiparadigm
world of commercial spreadsheets, IEEE International
Symposium on Software Reliability Engineering, (Raleigh,
NC, November 2006), 13-22.
[22] Fleming, S., Scaffidi, C., Piorkowski, D., Burnett, M.,
Bellamy, R., Lawrance, J., Kwan, I. 2013. An information
foraging theory perspective on tools for debugging,
refactoring, and reuse tasks, ACM Transactions on Software
Engineering and Methodology, 22, 2, (March 2013), Article
14, 41 pages.
[23] Groce, A., Kulesza, T., Zhang, C., Shamasunder, S., Burnett,
M., Wong, W., Stumpf, S., Das, S., Shinsel, A., Bice, F.,
McIntosh, K. (to appear). You are the only possible oracle:
Effective test selection for end users of interactive machine
learning systems, IEEE Transactions on Software
Engineering (to appear).
[24] Hermans, F., Sedee, B., Pinzger, M., Van Deursen, A. 2013.
Data clone detection and visualization in spreadsheets,
International Conference on Software Engineering, 292-301.
[25] Johnson, G. 1997. Labview Graphical Programming:
Practical Applications in Instrumentation and Control, 2nd
Edition. (McGraw-Hill, New York).
[26] Kahler, H. 2001. More than words - Collaborative tailoring
of a word processor. Journal of Universal Computer Science,
7, 9, 826-847.
[27] Kerr, J., Chou, M., Ellis, R., Kelleher, C. 2013. Setting the
scene: Scaffolding stories to benefit middle school
students learning to program, IEEE Symposium on Visual
Languages and Human-Centric Computing, (San Jose, CA,
Sept. 2013), 95-98.
[28] Kim, S., Zimmermann, T., Whitehead, J., and Zeller, A.
2007. Predicting faults from cached history, International
Conference on Software Engineering, 489-498
[29] Ko, A., Abraham, R., Beckwith, L., Blackwell, A., Burnett,
M., Erwig, M., Scaffidi, C., Lawrance, J., Lieberman, H.,
Myers, B., Rosson, M., Rothermel, G., Shaw, M.,
Wiedenbeck, S. 2011. The state of the art in end-user
software engineering, ACM Computing Surveys, 43, 3 (April
2011), Article 21, 44 pages.
[30] Ko, A. and Myers, B. 2005. A framework and methodology
for studying the causes of software errors in programming
systems, Journal of Visual Languages and Computing 16, 1
(Feb. 2005), 41-84.
[31] Ko, A. and Myers, B. 2004. Designing the Whyline: A
debugging interface for asking questions about program
behavior, ACM Conference on Human Factors in Computing
Systems (Vienna, Austria, April 2004), 151-158.
[32] Ko A. and Myers, B. 2008. Debugging reinvented: Asking
and answering why and why not questions about program
behavior, International Conference on Software Engineering
(Leipzig, Germany, May 2008), 301-310.
[33] Kuttal, S., Sarma, A., Rothermel, G. 2013. Predator behavior
in the wild web world of bugs: An information foraging
theory perspective, IEEE Symposium on Visual Languages
and Human-Centric Computing (San Jose, CA, Sept. 2013),
59-66.
[34] Lee, B., Kumar, S., Brafman, R., Klemmer, S. 2010.
Designing with interactive example galleries, ACM
Conference on Human Factors in Computing Systems
(Atlanta, GA, Apr. 2010), 2257-2268.
[35] Lee, M. and Ko, A. 2012. Investigating the role of purposeful
goals on novices’ engagement in a programming game. IEEE
Symposium on Visual Languages and Human-Centric
Computing, 163-166.
[36] Li, S., Xie, T., Tillmann, N. 2013. A comprehensive field
study of end-user programming on mobile devices, IEEE
Symposium on Visual Languages and Human-Centric
Computing (San Jose, CA, Sept. 2013), 43-50.
[37] Lieberman H., Paterno F. and Wulf V. (Eds). 2006. End-
User Development. (Kluwer / Springer. 2006).
[38] Lin, J., Wong, J., Nichols, J., Cypher, A., Lau, T. 2009. End-
user programming of mashups with Vegemite, ACM
International Conference on Intelligent User Interfaces
(Sanibel Island, Florida, Feb. 2009), 97-106.
210
[39] Nardi B. A Small Matter of Programming: Perspectives on
End User Computing. 1993. (Cambridge, MA: The MIT
Press).
[40] Oxford Dictionary,
http://www.oxforddictionaries.com/us/definition/american_e
nglish/silo (accessed Dec. 9, 2013).
[41] Ozenc, F., Kim, M., Zimmerman, J., Oney, S., Myers B.
2010. How to support designers in getting hold of the
immaterial material of software, ACM Conference on Human
Factors in Computing Systems (Atlanta, GA., April 2010),
2513-2522.
[42] Petre M. and Blackwell, A. 2007. Children as unwitting end-
user programmers, IEEE Symposium on Visual Languages
and Human-Centric Computing (Couer d’Alene, Idaho, Sept.
2007), 239-242.
[43] Pirolli, P. and Card, S. 1999. Information foraging. Psych.
Rev. 106, 4, 643–675.
[44] Posnett, D., D’Souza, R., Devanbu, P., Filkov, V. 2013. Dual
ecological measures of focus in software development,
International Conference on Software Engineering, 452-461.
[45] Raz O., Koopman P., Shaw M. 2002. Semantic anomaly
detection in online data sources. International Conference on
Software Engineering (Orlando, Florida, May 2002), 302-
312.
[46] Repenning, A. 2011. Programming goes back to school,
Communications of the ACM, 55, 5 (May 2012), 38-40.
[47] Rode, J., Rosson, M., Perez-Quinones, M. 2006. End user
development of web applications, End User Development
(Springer), 161-182.
[48] Rosson, M., Sinha H., Bhattacharya, M., Zhao, D. 2007.
Design planning in end-user web development. IEEE
Symposium on Visual Languages and Human-Centric
Computing, 189-196.
[49] Rothermel, G., Burnett, M., Li, L., Dupuis, C. and Sheretov,
A. 2001. A methodology for testing spreadsheets, ACM
Trans. Software Engineering and Methodology 10, 1 (Jan.
2001), 110-147.
[50] Ruthruff, J., Prabhakararao, S., Reichwein, J., Cook, C.,
Creswick, E. and Burnett, M. 2005. Interactive, visual fault
localization support for end-user programmers, Journal of
Visual Languages and Computing, 16, 1-2 (February/April
2005), 3-40.
[51] Scaffidi, C., Ko, A., Myers, B., and Shaw, M. 2006.
Dimensions characterizing programming feature usage by
information workers, IEEE Symposium on Visual Languages
and Human-Centric Computing, 59-64.
[52] Scaffidi, C., Myers, B., and Shaw, M. 2006. Challenges,
motivations, and success factors in the creation of Hurricane
Katrina “person locator” web sites, Psych. Programming
Interest Group Workshop (Sept. 2006), 9 pages.
[53] Scaffidi, C., Myers, B., and Shaw, M. 2008. Topes: Reusable
abstractions for validating data. International Conference on
Software Engineering, 1-10.
[54] Scaffidi, C., Shaw, M., and Myers, B. 2005. Estimating the
numbers of end users and end user programmers. IEEE
Symposium on Visual Languages and Human-Centric
Computing, 207-214.
[55] Schön, D. 1983. The Reflective Practitioner: How
Professionals Think In Action. (Basic Books, New York,
NY).
[56] Segal, J. 2007. Some problems of professional end user
developers, IEEE Symposium on Visual Languages and
Human-Centric Computing, (Couer d’Alene, Idaho, Sept.
2007), 111-118.
[57] Shaw, M. 2004. Avoiding costly errors in your spreadsheets.
Contractor’s Management Report 11, 2-4.
[58] Stolee, K. and Elbaum, S. 2013. Identification, impact, and
refactoring of smells in pipe-like web mashups, IEEE
Transactions on Software Engineering 39, 12 (Dec. 2013),
1654-1679.
[59] Subrahmaniyan, N., Burnett, M. and Bogart, C. 2008.
Software visualization for end-user programmers: Trial
period obstacles, ACM Symposium on Software Visualization
(Herrsching am Ammersee, Germany, September 2008),
135-144.
[60] Tillmann, N., de Halleux, J., Xie, T., Gulwani, S., Bishop, J.
2013. Teaching and learning programming and software
engineering via interactive gaming, International Conference
on Software Engineering (San Francisco, CA, May 2013),
1117-1126.
[61] Trigg, R. and Bødker, S. 1994. From implementation to
design: Tailoring and the emergence of systematization.
ACM Conference on Computer Supported Cooperative
Work, (Chapel Hill, North Carolina, October 1994), 45-54.
[62] Walia G. and Carver, J. 2009. A systematic literature review
to identify and classify software requirement errors,
Information and Software Technology 51. 1087–1109.
[63] Wiedenbeck, S. 2005. Facilitators and inhibitors of end-user
development by teachers in a school environment. IEEE
Symposium on Visual Languages and Human-Centric
Computing, 215–222.
[64] Wilson, A., Burnett, M., Beckwith, L., Granatir, O., Casburn,
L., Cook, C., Durham, M. and Rothermel, G. 2003.
Harnessing curiosity to increase correctness in end-user
programming, ACM Conference on Human Factors in
Computing Systems, (Ft. Lauderdale, FL, April 2003), 305-
312.
[65] Xie, J., Chu, V., Lipford, H. and Melton, J. 2011. ASIDE:
IDE support for web application security, ACM Annual
Computer Security Applications Conference (Dec. 2011).
267-276.
211
