Content uploaded by Johannah Sprinz
Author content
All content in this area was uploaded by Johannah Sprinz on Feb 14, 2022
Content may be subject to copyright.
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Johannah Sprinz
Crowdsourced User-Testing
Bachelor Thesis Presentation “Leveraging Human Computation for
Quality Assurance in Open Source Communities”
Submission Date: 29.01.2022
Presentation Date: 10.02.2022
Principal Advisor: Prof. Dr. François Bry
LMU Munich
Institute of Informatics
Teaching and Research Unit Programming and Modelling Languages
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Motivation: Crowdsourcing and Human Computation
Crowdsourcing: many people working towards a common goal (eg. Wikipedia)
Human Computation: delegating computation steps that are difficult to automate
from a machine to a human agent (eg. reCAPTCHA)
2
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Motivation: Open Source Software Development (OSSD)
3
publicly available source code
typically developed in volunteer communities
there may be paid staff, but a large part of the
work will be done by volunteers
volunteer developers work on things relevant
to their own use case
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Motivation: Software Quality Assurance (QA)
Quality Assurance: ensuring a software product’s fitness for use
more complex than expected, manual user-testing takes time
unit-, integration-, and system-tests can be automated
cave: automated tests need to be created and maintained
4
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
How does QA work for open-source communities?
automated tests may be neglected if developers are not interested in them
beta versions may be published before new releases for community feedback
users report bugs to allow developers to quickly address regressions
problem: little formalized testing, edge cases might not be thoroughly tested
idea: develop a human computation platform for crowdsourced user-testing
5
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Example Community: UBports Foundation
community of the Ubuntu Touch mobile OS
community structure resembles the onion model:
<10 paid developers
core team of ~30 developers
~200 contributing developers
~500 bug reporters
~15k users
6
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Use Case: Ubuntu Touch
operating system runs on 78 different devices
7
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Use Case: UBports Installer
graphical installation tool
78 devices
3 operating systems
different packages available
8
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
OPEN Crowdsourced User-Testing Suite
9
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Implementation: Graph Structure
10
system
test run
build
combination user
n
1
n1
1
n
1
n
n
1
n1
n
0..1
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Implementation: Query Resolution
11
Query.build()
db.builds.findOneById("42");
Build.tag()
Build.system()
Build.runs()
db.systems.findOneById(build.system);
System.name()
System.descr()
Run.comment()
Run.result()
Run.user()
db.users.findOneById(build.user);
db.runs.find({build: build.id});
User.email()
User.name()
context.user.id === user.id ?
user.email : "***masked***";
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Demo: Landing Page
12
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Demo: System Selection
13
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Demo: Build Selection
14
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Demo: Test Selection
15
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Demo: Test Submission
16
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Demo: System View
17
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Demo: Build View
18
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Demo: Combination Filters
19
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Demo: Property Combinations
20
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Demo: Build-Test-Property Tensor Slicing
21
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Evaluation
1. Is there any initial evidence of untapped potential for facilitating QA under the
OSSD by having inexperienced community members contribute formalized user
testing data to a human computation platform?
2. Can OPEN-CUTS be efficiently used by the inexperienced members of an open
source community to conduct and report formalized user tests?
3. Is OPEN-CUTS equipped to provide meaningful insights to open source
developers in their daily work?
22
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Usability Study
5 users and 4 developers, recruited from the UBports Community
all subjects receive an identical database and have to complete tasks
tasks represent how a subject would use OPEN-CUTS in the real world
task completion-times and concurrent think-aloud comments are recorded
retrospective probing questionnaire filled in after trial
23
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Results
24
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Results
25
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Future Work: Development
gamification, games with a purpose
automated reporting of real-world testing data (proof-of-concept already live!)
more powerful combination logic
improved data analysis and presentation, more caching
integration with CI/CD pipelines and bug trackers
interdependent tests
moderation features
support version-controlled (e.g., git) configuration files
26
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Future Work: Long-Term Study
evaluate long-term impact in the UBports Community (ubports.open-cuts.org)
What is the technical background of the users using OPEN-CUTS to run tests? Have
they contributed before?
How does the introduction of OPEN-CUTS impact overall contribution behavior? In
particular, are other low-skilled areas of contribution (e.g., translations, marketing, user
discussion, bug tracker activity) affected?
What is the contributor retention rate? I.e., how many contributors stay on as active
testers, how many merely occasionally contribute test results, how many lose interest?
Is the introduction of OPEN-CUTS considered helpful by a) the core development team,
b) the contributing developers, and c) the bug reporters?
27
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Future Work: Other Research
use OPEN-CUTS to support other research and development endeavours
Example: Expanding the UBports Installer to support other operating systems
it is not feasible for one developer to have access to hundreds of devices
crowdsourced testing might help
if installation configurations are crowdsourced, the testing should be as well
28
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022. 29
Thanks for listening!
Questions, comments, remarks?
J. Sprinz, “Leveraging Human Computation for Quality
Assurance in Open Source Communities,” LMU Munich,
Bachelor Thesis, Jan. 2022, doi: 10.5282/ubm/epub.91046
Future research will be published on spri.nz and open-cuts.org
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Implementation: Housekeeping and Build Discovery
30
discover new builds
delete builds older than a year if more than five
builds per channel,
delete oldest build
delete orphaned runs
GitHub releases
GitLab releases
Ubuntu Touch System-Image Server
UBports OpenStore
builds can be organized in channels,
e.g., devel, release candidate, stable
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
OCFL: OPEN-CUTS Combination Filtering Language
31
// First-order logic based on JSON
exp : <check> | <unaryexp> | <binaryexp> | <chainexp>
check : { CHECK: [<stringterminal>, [<stringterminal>, ...]] }
unaryexp : { <unaryop>: [<exp>] }
unaryop : NOT
binaryexp : { <binaryop>: [<exp>, <exp>] }
binaryop : IMPLIES
chainexp : { <chainop>: [<exp>, ...] }
chainop : AND | OR
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
OCFL Example
32
{"AND": [
{"IMPLIES": [
{ "CHECK": ["Package", ["exe"]] },
{ "CHECK": ["Environment", ["Windows"]] }] },
{"IMPLIES": [
{ "CHECK": ["Package", ["dmg"]] },
{ "CHECK": ["Environment", ["macOS"]] }] },
{"IMPLIES": [
{ "CHECK": ["Package", ["snap", "AppImage", "deb"]]},
{ "CHECK": ["Environment", ["Linux"]] }]
}] }
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022. 33
Implementation: Dockerized Microservices
Crowdsourced User-Testing, Bachelor Thesis Presentation by Johannah Sprinz. Presented at LMU Munich on 10.02.2022.
Literature
M. Aberdour, “Achieving Quality in Open-Source Software,” IEEE Software, vol. 24, no. 1, pp. 58–64, Jan. 2007, doi: 10.1109/MS.2007.2
S. S. Bahamdain, “Open Source Software (OSS) Quality Assurance: A Survey Paper,” Procedia Computer Science, vol. 56, pp. 459–464, Jan. 2015, doi:
10.1016/j.procs.2015.07.236
D. Kulesz and I. Bogicevic, “SystemTestPortal - A Web-Application for managing Manual System Tests,” Nov. 2017, doi: 10.5446/41642
E. Law and L. von Ahn, Human computation. Morgan & Claypool, 2011. doi: 10.2200/S00371ED1V01Y201107AIM013
R. Mühlhoff, “Human-aided artificial intelligence: Or, how to run large computations in human brains? Toward a media sociology of machine learning:” New
Media & Society, pp. 1868–1884, Nov. 2019, doi: 10.1177/1461444819885334
E. S. Raymond, “The cathedral and the bazaar,” First Monday, vol. 3, no. 2, Mar. 1998, doi: 10.5210/fm.v3i2.578
G. G. Schulmeyer, Ed., Handbook of software quality assurance, 4th ed. Boston: Artech House, 2008
J. Sprinz, “Leveraging Human Computation for Quality Assurance in Open Source Communities,” LMU Munich, Bachelor Thesis, Jan. 2022, doi:
10.5282/ubm/epub.91046
J. Sprinz, “A collaborative effort to create a user-friendly installer for different mobile operating systems,” submitted to the Journal of Brief Ideas, Feb. 2022,
Available: https://beta.briefideas.org/ideas/3f22fbc27fb6dcb3c4c33603ef597f9b
D. Wahyudin, A. Schatten, D. Winkler, and S. Biffl, “Aspects of Software Quality Assurance in Open Source Software Projects: Two Case Studies from
Apache Project,” Aug. 2007, pp. 229–236. doi: 10.1109/EUROMICRO.2007.19
34