Available via license: CC BY 4.0
Content may be subject to copyright.
CHARACTERISTICS
EFFECTIVENESS
TECHNICAL
PERFORMANCE
CITIZEN AUTONOMY
TRANSPARENCY
DATA PROTECTION
USABILITY
Captures non-judgemental criteria, e.g.,
app identifiers, organisation ids, functional
overview.
Measures the success of an app wrt its
accuracy/detecting/sharing of contacts/its
adoption by the population.
Focuses on system resource utilization and
execution speed including processor work-
ing, both front-end and back-end.
Measures the degree to which a user can
define their own comfort level in terms of
rights/access they grant an app.
Involves questions unrelated to functionality
and focuses on documentation of source
code, privacy policy, etc.
Indicates the capacity to protect the rights
of an individual during collection and pro-
cessing of personal data.
Assesses the user interface and its ability to
support satisfactory, accessible and consis-
tent user interaction.
Citizen-Focused Compare-and-Contrast
Evaluation Framework (C3EF) for
digital Contact Tracing Applications
(CTAs) for COVID-19
The Citizen-Focused Compare-and-Contrast Evaluation Framework (C3EF) is de-
signed to help improve existing digital Contact Tracing Applications (CTAs) de-
veloped for COVID-19. The framework includes 7 pillars, e.g., Characteristics,
Effectiveness, Technical Performance, Citizen Autonomy, Transparency, Data
Protection and Usability. Each pillar has attributes, sub-attributes and at least
one probing question.
Multimedia Appendix 3
Visualization of all pillars and questions.
4. App Content
C19 App Running
State
C20 Contact Tracing
Definition
C17 Processing
Overview
C18 Sensor
Employed
C21 Data Collection
C22 Data Storage
App Data
C24 Notification
Method
C25 Diagnosis
Status
C17
C18
C19
C20
C21
C22
C23
C24
C25
Is processing overview available to explain how the app works?
What sensors are employed (BT/GPS/Ultrasound)?
What is the running state (foreground/background)? Is the user aware of how long the app
uses the said functionalities?
What is the system's definition of close contact? (e.g., within 2m >15 minutes?)
What data does the app collect?
How long is the data stored?
What are the permissions being taken by the app?
How are notifications managed? (direct from app, or via contact by health professional)
-where notification means something to do with covid-related notifications. (not warnings or
alerts)
Is there a self-reporting function of a positive diagnosis?
C23 App Permissions
3. Organisational
Reputation
App Status
C11 Primary Developer
Development History
Development
C09 National App Status
C10 Official
Documentation
C12 Development
Partners to Include
Open Sources
C15 Developer
Homepage
C16 Help, Technical
Support Discussion
Forums
C13 Organisation
C14 Continual Maintenance
Is this the official national app?
Is there official documentation available relating to the app devel-
opment and design? Where can these official documents be
found?
What is the name of developer/software house?
What are the names of development partners?
Does the software development organisation have any past history
of developing data-sensitive apps? If so, please list those devel-
oped over the past 5 years and who was responsible?
Does development include evidence of continual maintenance of
the app?
What is the developer homepage (i.e., URL of official website)?
What form of technical support is available for the users to include
synchronous and asynchronous support?
C09
C10
C11
C12
C13
C14
C15
C16
2. Availability
C06 Internet
Connectivity:
App (other)
Platform
Dependency C08 App Download Size
C07 Platform/OS
Supported
Is internet access required to use the app?
What platform is supported? e.g., can app be applied to all types of smartphone?
What is the app download size for different platforms?
C06
C07
C08
What is the full name of the app?
Which country is the app developed for?
What are the current versions of the app for different platforms?
Which languages are supported by the App?
What age group is the app designed for as advertised on the Google play/App store?
C01
C02
C03
C04
C05
1. General Characteristics
C04 Language
Support
C05 Age of Users
C01 Name of
Application
C02 Country
C03 Current Versions
CHARACTERISTICS
How many downloads of the app?
How many uninstalls of the app?
Average frequency of user interaction
with the app/number of active users?
E09
E10
E11
E05
E06
E07
E08
What is the proportion of people alerted
who share their positive status per week?
What is the number of additional close
contacts found through the app per
week?
What is the number of these people who
subsequently tested positive?
How much time/effort is invested in
manual contact tracing versus app
contact tracing (maybe activities
based)?
1. Effective Reporting
E03 Reporting All
Close Contacts
E04 Reporting
Hotspots
E01 Detecting Close
Contacts
E02 Reporting
Positive Close
Contacts
E01
E02
E03
E04
How effective is the current version in
detecting the close contacts in terms
of accuracy and precision?
Are users correctly alerted to positive
past close contacts?
Are uses proactively and correctly
alerted to all close contacts?
How accurate is the app in alerting
users to hotspots?
EFFECTIVENESS
E07 No. of Those
Contacts Found
Positive
2. Effective Results
E08 Relative Effort
Per Contact Found
Versus Manual CT
E06 No. of (Аdditional)
Contacts/Week
Found
E05 Users Who
Share Their Data
E09 Population
Uptake
3. Effective Еngagement E10 Population
Retention
E11 Population
Engagement
TP06 Bandwidth
Usage
4. Resources and
Troubleshooting & Trust
TP07 Throughput
(Backend)
TP05 CPU/Memory
Usage
TP05
TP06
TP07
How much of a CPU/memory is used
by an app?
How much of a bandwidth is used by
an app?
How many user requests a backend
system can process per unit of time?
3. Consumption
TP03 Battery
TP04 Disk Space
TP03
TP04
How much of a battery is consumed
(%) by app per day/week/month?
How much of a disk space is used by
an app?
TECHNICAL PERFORMANCE
2. Efficiency TP02 Response Time TP02 How fast (in ms) the algorithm(s)
responds to user interaction?
1. Speed TP01 Response Time
(Frontend)
TP01 How fast (in ms) the frontend app
responds to user interaction?
3. Data Control
CA08 Data
Upload Authority
CA09 Uploaded
Data Location
Visibility
Does the user explicitly control
uploading of their medical-person-
al/GPS/Contact/Other data?
Is there user-available information to
show them where their uploaded
data resides?
CA08
CA09
2. Phone Functionality CA05 ENS Access
CA06 Notifications
CA03 GPS Access
CA04 Bluetooth
CA07 Microphone
Can the user back out of allowing the
app access to the GPS phone
functionality?
Can the user back out of allowing the
app access to the Bluetooth phone
functionality?
Can the user back out of allowing the
app access to the ENS phone
functionality?
Can the user back out of allowing the
app access to the phone notification
functionality?
Can the user back out of allowing the
app to record the microphone?
CA03
CA04
CA05
CA06
CA07
Is there an official discussion forum
where users can express their opin-
ions, app requirements and app
bugs?
If there is such a discussion forum, are
there moderators/viewers of that
forum empowered to adapt the app
accordingly?
CA01
CA02
CITIZEN AUTONOMY
1. App Discussion
Authority
CA01 Official
Discussion Forums
CA02 Empowered
Moderators
Is the purpose of the app made accurately and accessibly explicit
to the user?
Is the app being used only for the purpose it has been stated?
Is the app integrating any third-party software or using any
third-party apps? If yes, then what are the apps?
Are the app-developing companies/authorities made explicit to
the users? Is reputational information accessible?
Does the organisation have a clear and concise process for com-
munity feedback?
Is the source code publicly available?
Are the organisation’s internal policies, staff, functions and devel-
opment process visible?
Does the app explicitly identify the phone functionalities it accesses
(GPS, Bluetooth data) in a document format?
Is the app being transparent about the necessity of permissions
required for its functionality?
Does the app regulate a time period over the services being used
for its working?
T01
T02
T03
T04
T05
T06
T07
T08
T09
T10
1. App Transparency
App Purpose
App Permission
App Development
Knowlege
T06 Open Source
Repository
App-Purpose Knowledge
App Participation
Knowledge
Modus Operandi
Phone
T07 Policies and Process
T05 Community Feedback
T04 App-Developing Transparancy
T03 Third Party Involvement
T02 Accurate Use
T01 Explicit to User
T09 Permissions
T10 Service Regulations
T08 Sensor Knowledge
2. User Participation
T11 Does the app indicate and explain to the end-user about the voluntary nature of participation?
T11 Implications of
Participation
App Participation
Knowledge
3. Data Transparency
Minimisation,
Gathering, Store,
Accessibility, etc.
T13 Data-Storage
Knowledge
T14 Privacy-Policy
Knowledge
T15 Data Protection
T16 Data Anonymity
T17 CT Accuracy
Reporting
T18, T19 Data Capture
Knowledge
T12 Data Minimisation
GDPR Applicability
Life-Cycle T22 App Participation
Knowledge
EOL Knowledge
T25 User Control
on Data
T23 Data Storage Conditions
T24 Data Lifescycle
App Development
Knowledge
T21 Data-Access
Knowledge
Does the app only require minimal personal data of the end users (name, email address, location, etc.)?
Is the end user informed on the data being collected by the app(i) in DPIA and (ii) when the information
is being taken from the user?
Is the privacy policy documented?
Is it clear on how the data generated on the app is encrypted?
Is the data generated from the app anonymised so that the individuals are non-identifiable?
Is the app being transparent about the contact tracing accuracy that they are achieving?
Is the data captured by the app made accurately and accessibly explicit to the user?
Is the user made explicitly aware of where their data is stored (including jurisdiction) and for how long?
Is the user made accurately and accessibly aware of the data accessible to other bodies, both in terms
of the data, and the accessing bodies?
Are there limitations on how the data gets used?
Can users easily access all of their data?(A) via the app (B) by contacting the officials?
Is the user made explicitly aware of the conditions deemed necessary for app end-of-life, and what
happens to the recorded data at that point?
Is the end-user aware of the life-cycle of data collected?
Does the app allow the end user to delete their data?
T12
T13
T14
T15
T16
T17
T18
T19
T20
T21
T22
T23
T24
T25
T20 Data Sharing Transparency
TRANSPARENCY
Preliminaries
GDPR Principles
Rights
DP08 Basis for
Processing and
Withdrawal
DP05 Actors
DP06 Data Stored
DP07 Data Type
DP09 Minors
DP12 Lawfulness
DP13 Fairness
DP10 Purposeful
Limitation
DP11 Data
Minimisation
DP14 Storage
Limitation
DP17 Portability
DP18 Automated
Processing Rejection
DP15 Access
DP16 Object to
Reuse
DP19 Rectified,
Restricted or Erased
DATA PROTECTION
DP03 Software
Architecture
Security
DP04 SDLC and
Security
DP01 STRIDE
Taxonomy
/Vulnerabilities
DP02 CT-Specific
Threats
1. Security
What obvious security vulnerabilities are present?
What contact-tracing specific threats are posed to this
system?
How do these vulnerabilities relate to the software architec-
ture?
How do the vulnerabilities of the software architecture
affect its on going development?
DP01
DP02
DP03
DP04
2. GDPR
Who are the data controllers and
processors?
What personal data is collected?
Is the data identifiable, and is it special
category data?
What is the basis for processing the
data? (legal, consent etc.)
Does the data involve or identify
minors?
Is the data only being used for the
purpose that it was collected?
Can less data be collected for the
same purpose?
Is data collected legally? Does it
identify criminal offense?
Are there any clear issues in which
the data is not processed fairly?
Is data only stored for as long as
necessary?
Is there a clear way for the user to
access their data from the organisa-
tion in a specific time via subject
access request?
Is the user able to reject to instances
where their data is used for other
purposes?
Is there a method that allows the user
to retrieve their data in a portable
format?
Can the user reject automated pro-
cessing decisions?
Is there a clear way for the user to
change, restrict or erase their data
from the organisation? Does this
process work?
DP05
DP06
DP07
DP08
DP09
DP10
DP11
DP12
DP13
DP14
DP15
DP16
DP17
DP18
DP19
LEGEND
Main Theme
Attribute
Task Specific (TS)
Elements Specific (ES)
Some of the usability questions are Task
Specific (TS) in that they are applicable to
the execution of one specific task on the
interface from beginning to end. TS questions
are not applicable to the app as a whole.
An element can vary from a button to a
menu, a slider or a table that are used in the
design of the User Interface. The questions
assess the design of individual elements of an
interface (e.g., button, menu, etc.) and are
Element Specific (ES).
5. Ongoing App Evaluation U86 Frequency of
Upgrade
How often is the app
upgraded?
U86
U02 Motivations for
High Score
1. Subjective Satisfaction
U03 Motivations for
Low Score
U01 Rating
How would you rate your
expe- rience with this app
from 1 to 5?
What are the main motiva-
tions (reasons) for satisfied
users?
What are the main motiva-
tions (reasons) for unsatisfied
users?
U01
U02
U03
3. Design Effectiveness
User Interface
Helpfulness
Completeness
Configurability
U39 Core Functionality
U40 Optional
Functionalities
U41 Completeness of
Functionalities
Design
Aesthetic and Attractiveness
U53 Responsiveness
Clarity (Visibility, Simplicity)
Consistency
Suitability of Documentation
U66 Interactive Assistance
Technical
U51 Aesthetic
U52 Colour Scheme
U47 Connection Failure Feedback
U50 Haptic
U49 Acoustic
U63 Definitions
U64 Descriptions
U65 Examples
U46 Access Microphone
U45 Access Phone Notification
U44 Access ENS
U43 Access Bluetooth
U42 Access GPS
U48 Visual
What is the core functionality? (This question helps to identify the core functionality to which Task Specific (TS) questions should be applied.)
What optional functionalities are offered? (This question helps to identify all of the optional functionalities to which Task Specific (TS) question
could applied.)
Are all the necessary functionalities for CTA available? Is there evidence that users would expect more/other functionalities?
Does the app seek permission for access to the GPS phone functionality?
Does the app seek permission for access to the Bluetooth phone functionality?
Does the app seek permission for access to the ENS phone functionality?
Does the app seek permission for access to the phone notification functionality?
Does the app seek permission to record the microphone?
Is any feedback available when the app fails to connect to any of these technologies?
Does the app design accommodate the OS-level changes from the device settings (e.g., font, colour, contrast)?
Does the app design support user configuration on acoustics?
Does the app design support user configuration on haptics?
Is the app aesthetically pleasing?
Do the users like the colour scheme used in the user interface?
Does the user interface and readability of content look different on different platforms?
U39
U40
U41
U42
U43
U44
U45
U46
U47
U48
U49
U50
U51
U52
U53
(ES) What are the elements used in the design of the interface? (Тhis question helps to identify
the elements for the Element Specific (ES) questions.)
(ES) Are the interactive elements of the interface easy to identify?
(ES) How easy is it to understand the meaning of the elements (what they do/are)? Can I tell
what it is?
(ES) Are the structures identifiable/visible? (Structures are organisations of elements. This
question concerns the meaning, labelling, and structuring criteria used to group elements
together (e.g., Do items in one menu belong to the same category?)
(ES) Are the structuring/organising criteria understandable?(Structures are organisations of
elements. This question concerns the meaning, labeling, and structuring criteria used to group
elements together (e.g., Do items in one menu belong to the same category?)
(ES) Are the same elements designed in different ways?
(ES) Does the same element behave in different ways? Is the same element (e.g., button,
menu) coherently designed throughout the app (e.g., Are all buttons round and embossed?
Are all menus formatted the same)?
(ES) Are ways to visually organise elements used incoherently?
(ES) Are ways to organise elements used coherently?
U54
U55
U56
U57
U58
U59
U60
U61
U62
Are relevant definitions offered?
Are relevant descriptions offered?
Are examples offered?
Is interactive assistance available (i.e.,
technical support)?
U63
U64
U65
U66
Clarity of Structure
Consistency of Elements
Consistency of Structure
Clarity of Elements
U54 (ES), U55 (ES) Perceptual Clarity of Elements
U56 (ES) Conceptual Clarity of Elements
U57 (ES) Perceptual Clarity of Structure
U58 (ES) Conceptual Clarity of Structure
U59 (ES) Perceptual Consistency of Elements
U60 (ES) Conceptual Consistency of Elements
U61 (ES) Perceptual Consistency of Structure
U62 (ES) Conceptual Consistency of Structure
4. User Interaction
U70 Orientation
U69 Tied-up Resources
Efficiency
Robustness
Clarity of Interaction
with Elements
Consistency of
Interaction
with Elements
(TS) How many steps does it take to
complete the core task?
(TS) How much time does it take to
complete the main task?
Does completing the task require the
need of external resources (e.g., power,
etc.)?
Is portrait and/or landscape layout
available?
(TS)Can the same task ‘a’ be executed
while carrying out other tasks with the
same phone (e.g., using Bluetooth
earphones)?
(TS) Can the same task ‘a’ be executed
in different ways/in more than one place
(e.g., activate/deactivate the contact
tracing function, activate/deactivate
notifications, activate/deactivate
Bluetooth, GPS, micrphone)?
(TS) Are shortcuts if applicable available?
(TS) Can the same task ‘a’ be executed
in different contexts (e.g., choose as
pertinent: at dark, in a very noisy environ-
ment, not looking?
U67
U68
U69
U70
U71
U72
U73
(TS) Are error messages available? Are
these clear to the users? Are they
actionable?
(TS) Is error recovery (e.g., undo, redo)
available?
(TS) Are actions on elements reversible?
U74
U75
U76
(TS) I can tell what to do with it?
(TS) Is the feedback adequate when
interacting and using the app? Is
feedback clear, prominent and timely?
Is it actionable? Are there instances of
confusion or lack of feedback?
(TS) Are there any instances in which the
same interactive element should be
operated (interacted with) in different
ways (e.g., are all buttons just for push-
ing? If some are for holding, do these
look different?)?
(TS) Is the feedback consistent when
interacting with elements of the same
type? Are there instances of inconsistent
feedback when interacting with
elements?
(TS) Are constraints in place to prevent
errors? Are these effective?
(TS) Are there false or wrong constraints
when interacting with the interface (e.g.,
inactive buttons, greyed menus, missing
elements)?
(TS) Are different types of alert or error
messages available? Are they action-
able?
(TS) Does the app have built-in notifica-
tion settings? Is control on the notifica-
tions available in the app?
(TS) Are notification messages clear and
actionable?
(TS) Can the user easily access contact
tracing data for the last 14 days, can
these be visualised?
U77
U78
U79
U80
U81
U82
U83
U84
U85
U76 (TS) Action Reversibility
U74 (TS) Error Messages
U75 (TS) Error Recovery
U77 (TS) Interactional
Clarity with Elements
(Predictability)
U78 (TS) Clarity of
Feedback
U84 (TS) Notification
Messages
U79 (TS) Inconsistency
of Action on Elements
U81 (TS) Constraints
U82 (TS) Allert Messages
U71 (TS) Muliti-Tasking
U73 (TS) Adaptability
(TS) Errors
U80 (TS) Inconsistency of
Feedback
U83 (TS) Notification
Control
U85 (TS) Notifications
Access and Visualisation
U72 (TS) Flexibility/
Multimodality
U67 (TS) Human Effort
U68 (TS) Time
Alerts and
Notifications
Messages
2. Universality
U04, U05 Usage Without/or Limited Vision
U06, U07 Usage Without Perception of Colour
U08 Usage Without/or Limited Hearing
U09 Usage Without Vocal Capability
U11 Usage With Limited Cognition
U10 Usage With Limited Manipulation or Strength
Accessibility
U36 Language Activation
Cultural Universality
Cognitive
Visual
Auditory
Vocal
Motor
U18, U19, U20 Navigational
/Interactive Elements
U21, U22, U23 Input Field
U24, U25, U26, U27 Text and Graphics
U30, U31 Onboarding
U32, U33, U34 Low Physical Effort
U35 Gestures
Functional Performance
Accessible Interactions
UI Elements
U12, U13, U14, U15, U16, U17
Age/Parental Control
U28, U29 Content
U37 Age/Terms and
Conditions
Does the app design accommodate the assistive technologies activated in the accessibility
settings (e.g., screen readers)?
Does the app provide an appropriate content layout for screen readers available on iOS and
Android platforms (i.e., text for images, labels, icons and loading states)?
Is a colour-blind friendly design available (i.e., descriptive labels of colours filters and swatches,
underline links, the use of patterns and textures for graphs and charts, symbols to mark required
form fields)?
Are the used colours appropriate for colour blind users (e.g., colour combinations that should
be avoided: green-red, green-blue, green-brown, green-black, green-grey, blue-grey, light
green-yellow, blue-purple)?
Are alternative options to interact with the app that does not require/or with limited hearing
available?
Can a user fully operate the app without the use of voice?
Does the app accommodate a variety of input modalities, i.e., speech recognition, handwrit-
ing recognition, gestures or use of an external keyboard? If yes, do these support users to
validate inputs, i.e., confirmation screens or undo/redo options?
Are there features which make the technology simple and easy to use by individuals with
limited cognitive, language and learning abilities?
U04
U05
U06
U07
U08
U09
U10
U11
Is a notice available for the suitability of the content for specific age groups?
Is there a verification process available throughout the parental/legal guardian consent for children under the age of 13 (14 or 15) to minimise
the collection, use, or disclosure of personal information/data from children’s devices?
Are mechanisms used to prevent deactivation of contact tracing function on the child’s mobile device without the permission of the parents/-
legal guardians?
Are mechanisms used to prevent sending collected data on the child’s mobile device without the permission of the parents/legal guardians?
Does the app send an alert message to parents/legal guardians that the information (data) was sent to the health authorities/developers from
their child’s mobile device and they can request it to be deleted?
Is a simpler design available for kids?
Is labelling clear (i.e., tab bars, buttons)?
Are navigational components prominent enough?
What approaches are used for the navigation path, e.g., hamburger menu/hidden, tab bar, vertical navigation, buttons, plus button?
Are the input fields easy to identify and fill in? Are other options available for the input field i.e., voice control, QR code?
Are input fields 32-40px in height?
Is the touch target size between 7 to 10mm (i.e., for Android at least 48 × 48dp and for iOS at least 44 x 44pt)?
Is magnification available?
Is the font size between 14 and 16 pixels? Does the text incorporate appropriate spacing allowance between the lines (e.g., 1.2 times the font
height)?
Is the contrast ratio of small text (i.e., below 14pt bold/18pt regular) at least 4.5:1 against its background?
Is the contrast ratio of large-scale text (i.e., 14 pt bold/18 pt regular and up) and graphics at least 3:1?
Is the content appropriate for the user population (i.e., avoid the use of jargon or acronyms related to clinical knowledge)?
Does the content (e.g., functionality and readability) retain the same after the adjustments are made to accommodate accessibility?
Does the app offer onboarding for new users?
What approach was used for the design of onboarding, i.e., animations, video, text, images? If these are videos, animations and audios, is
there transcripts, captions and/or audio descriptions available?
Can a user complete a task without scrolling? If no, are the content elements critical to the app functionality positioned above the scroll line?
Does the app support interaction with one-hand use (i.e., left or right)?
Are select menus, radio buttons and checkboxes used?
U12
U13
U14
U15
U16
U17
U18
U19
U20
U21
U22
U23
U24
U25
U26
U27
U28
U29
U30
U31
U32
U33
U34
Is the name easy to recognise, to remem-
ber and to find on Google Play and
Apple Store? (e.g., appcovidhseie)
Does the name suggest any (false/mis-
leading) sense of protection/security?
(e.g., covid total protection app)
Does the name avoid any emphasis on
the disease/infection/virus, which might
cause social stigma? (e.g., personal
infection tracker)
What approach is used to activate the
available language(s)?
What age group is the app designed for
as stated in the ‘Terms and Conditions’?
(Are they the same as advertised in app
stores?)
Are cultural conventions used in the
design?
U35
U36
U37
U38
U38 Cultural Conventions
U35 Name Qualities
USABILITY
