Content uploaded by Maxim Kolomeets
Author content
All content in this area was uploaded by Maxim Kolomeets on Jun 22, 2022
Content may be subject to copyright.
Security Measuring System for IoT Devices
Elena Doynikova, Evgenia Novikova, Ivan Murenin, Maxim Kolomeec, Diana
Gaifulina, Olga Tushkanova, Dmitry Levshun, Alexey Meleshko, and
Igor Kotenko
St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC
RAS), St. Petersburg Institute for Informatics and Automation of the Russian
Academy of Sciences, 14-th Liniya, 39, St. Petersburg, 199178, Russia
{doynikova, novikova}@comsec.spb.ru, imurenin@gmail.com, {kolomeec,
gaifulina, tushkanova, levshun, meleshko, ivkote}@comsec.spb.ru
Abstract. Wide application of IoT devices together with the growth of
cyber attacks against them creates a need for a simple and clear system
of security metrics for the end users and producers that will allow them
to understand how secure their IoT devices are and to compare these
devices with each other, as well as to enhance the security of the devices.
The paper proposes a security measuring system that is based on the
hierarchy of metrics representing different security properties and inte-
grates these security metrics in one clear and reasonable score depending
on available data. The algorithms used for metrics calculation are briefly
described with the main focus on the algorithms for integral scores. To
demonstrate the operation of the proposed security measuring system,
the case study describing metrics calculation for the IoT device is given.
Keywords: Security measuring ·IoT devices ·Metrics ·Integral scores
·Data analysis ·Confidentiality ·Integrity ·Availability ·Privacy ·
Anomalies.
1 Introduction
The market of IoT devices nowadays is extremely heterogeneous. At the same
time the number of cyber attacks using such devices increases. Thus, it is impor-
tant to provide a simple and clear system of security metrics for the end users
and producers that will allow them to understand how secure their IoT devices
are and to compare these devices with each other, as well as to enhance the
security of the devices.
Currently, there are checklists of security requirements for the IoT devices
and corresponding metrics, for example, IoT Security Compliance Framework
2.0 (IoT Security Foundation) [9].
Besides, there are researches that propose the approaches and techniques for
calculation of different IoT security metrics, such as Confidentiality, Integrity,
Availability (CIA) [8, 10], Authenticity [15], Privacy [12–14], Transparency, Read-
ability [11], Trustworthiness [18], and others.
preprint
ADIOT-21 conf.
2 E. Doynikova et al.
At the same time there is no comprehensive framework that incorporates in-
terconnected security and privacy metrics for different security properties of the
IoT devices and algorithms for their calculation, and integrates these security
metrics in one clear and reasonable score, as well as there is no security mea-
suring system that implements calculation of such comprehensive set of metrics
including the integral security metric. Thus, there is a gap between a need for
comprehensive security measuring system for IoT devices and current solutions
that implement a limited set of security measurements.
Contribution. The main contribution of the research paper consists in the elim-
ination of the gap mentioned above via the development of the comprehensive
security measuring system for IoT devices. The proposed system automatically
calculates a set of security metrics representing different security and privacy
device characteristics considering available data and integrates them into the
common integral security & privacy score. Namely:
–The authors propose the hierarchy of the security and privacy metrics for the
IoT devices incorporating metrics that are calculated on the basis of static
and dynamic data describing device specification and behavior. The security
metrics determined based on the static data are as follows: static CIA score
(calculated based on the internal criticality of the device by confidentiality,
integrity and availability properties, exploitability, and confidentiality, in-
tegrity and availability impact considering device criticality); privacy policy
based score (defined based on the readability score and ontology-based repre-
sentation of privacy policy); APK (Android Package) based score (calculated
based on the application description-based score and APK permission-based
score), as well as integral static score. The security metrics determined based
on the dynamic data are as follows: dynamic privacy score (calculated based
on the APK based score and dynamic information on the privacy relating
anomalies detected in device and system logs); dynamic CIA score (com-
puted based on the dynamic exploitability, and static CIA score); as well as
integral dynamic score.
–The authors introduce the algorithms for the metrics calculation including
novel algorithms for static CIA score, ontology-based privacy score, APK
based score, dynamic privacy and CIA score.
–The authors introduce the set of novel algorithms for the calculation of the
integral security metrics.
–The authors develop an architecture of the security measuring system that
implements the proposed algorithms.
–The authors demonstrate the operation of the developed system on the case
study.
Novelty. The novelty of the proposed solution consists in the introduced hier-
archy of the security metrics for the IoT devices, novel algorithms for separate
metrics calculation, and novel algorithms for the integral metrics calculation
depending on the available input data.
Security Measuring System for IoT Devices 3
The paper is organized as follows. Section 2 analyzes the related research
in the area. Section 3 introduces the proposed security measuring system and
its components, the hierarchy of security metrics, the algorithms for their cal-
culation and the algorithms for the calculation of the integral scores. Section 4
describes the implementation of the proposed security measuring system and a
case study demonstrating its operation. Section 5 contains the discussion and
conclusion.
2 Related Research
A lot of research has been done in IoT security measuring. There are secu-
rity guidelines that specify main security principles that should be satisfied for
IoT devices, for example, Cyber Assessment Framework (CAF) [16] and IoT
Security Compliance framework [17]. Such guidelines represent what security
requirements should be satisfied and measured but do not describe how.
The study of these documents and of the security standards allowed us to
outline and specify the main security metrics that should be calculated for the
IoT devices, including Confidentiality, Integrity, Availability, Authenticity, Pri-
vacy, Trustworthiness, and others. Researchers proposed various approaches for
the calculation of these metrics. Thus, confidentiality, integrity, and availability
for a device can be calculated based on available information on device vulnera-
bilities. Known vulnerabilities and their Common Vulnerability Scoring System
(CVSS) scores [8] representing a likelihood of their exploitation and impact from
their exploitation for CIA can be found in publicly available databases such as
National Vulnerability Database (NVD). These scores can be used by themselves
or in the scope of more complicated approaches considering connections between
the devices and vulnerabilities that can lead to higher damage for device secu-
rity [10]. Another approach that can be used for confidentiality measuring as
well as for privacy measuring is the analysis of permissions granted to the de-
vice software and hardware. There are also approaches that calculate privacy
and transparency based on the policy text analysis. There are rule-based ap-
proaches [12], ontology-based approaches [20, 21, 13, 22, 23], and machine learn-
ing approach [14]. For readability calculation, different algorithms that assess
text complexity can be used [11]. Besides, to calculate CIA and authenticity
machine learning based approaches can be used [15].
Though a lot of research has been done in the field of IoT devices’ security
and privacy there is no comprehensive framework that incorporates intercon-
nected security and privacy metrics for different security properties of the IoT
devices and algorithms and algorithms for their calculation, and integrates these
security metrics in one clear and reasonable score, as well as there is no secu-
rity measuring system that implements calculation of such comprehensive set of
metrics including the integral security & privacy metric.
4 E. Doynikova et al.
3 Security Measuring System
We propose a security measuring system that aims to provide security tags (i.e.
grades or scores) for the IoT devices to compare them in security terms. It is
based on a hierarchy of security and privacy metrics that are calculated on the
basis of various attributes and characteristics of the device. The system takes
the following information as input: software installed on the device, description
of the software, corresponding privacy policies; names of the .apk installed on
the devices; NVD data on products (Common Platform Enumeration, CPE)
and known vulnerabilities (Common Vulnerabilities and Exposures, CVE), log
describing normal device’s behavior, their specification, and outputs a set of
metrics. The metrics could be divided into two groups - static and dynamic
depending on the type of input data. The hierarchy of proposed static metrics
is shown in Fig. 1. The static metrics serve as initial values for corresponding
dynamic scores recalculated on the basis of streaming logs of the device (Fig. 1).
Fig. 1. Hierarchy of static security and privacy metrics.
The calculation of the metrics is implemented by the corresponding compo-
nents of the security measuring system that are described in detail below:
1. Static CIA score calculation component implements CIA score calculation
on the basis of CVSS scores of known vulnerabilities of the devices.
2. APK based score calculation component implements privacy calculation for
the device considering requested and required permissions of the installed
applications based on its description.
3. Ontology based privacy score calculation component implements risk calcu-
lation for the device considering its privacy policy.
Security Measuring System for IoT Devices 5
Fig. 2. Hierarchy of dynamic security and privacy metrics.
4. Readability score calculation component calculates the readability score of
the device privacy policy to provide a comprehensive evaluation of the pri-
vacy risks associated with the device.
5. Integral scores calculation component implements calculation of integral
static and dynamic scores, including integral privacy policy based score.
6. CIA score calculation considering attack traces component implements dy-
namic CIA score calculation taking into account network configuration that
is used to determine traces of vulnerabilities that can be exploited to com-
promise the device.
7. Statistics based CIA and privacy scores assessment component and Machine
Learning (ML) based CIA and privacy score assessment component imple-
ment detection of anomalies in the device behavior and calculation of anoma-
lies weights for further recalculation of CIA and privacy scores in dynamics.
8. Log processing and integration component implements analysis and integra-
tion of various logs.
9. Measuring database.
The common scheme of the proposed system is given in Fig. 3.
Static CIA score calculation component. The static CIA score calculation
is based on the analysis of known vulnerabilities of the device software and
firmware. The risk assessment procedure includes the following basic steps: (1)
find vulnerabilities of the specific device; (2) get environmental CVSS scores for
vulnerabilities found; (3) calculate CIA score as maximum environmental CVSS
score across all found vulnerabilities.
The information about devices’ vulnerabilities could be found either in open
databases, such as NVD, or obtained from the penetration testing team. NVD
contains information about vulnerabilities in CVE format linked to CPE that
is a formal description of software and hardware. To find vulnerabilities, it is
necessary to form the list of hardware, applications or operational systems that
6 E. Doynikova et al.
Fig. 3. Common scheme of the proposed security measuring system.
needs to be analysed, to search the CPEs (for each element from the list), to
search the CVEs by CPE.
Each CVE entry has a CVSS score that captures the principal technical
characteristics of vulnerabilities. This score incorporates information both on the
impact on CIA in case of the vulnerability exploit and ease of its exploitation. In
the static CIA score calculation component the modified environmental CVSS
score is used. It allows considering the criticality of the device under analysis.
APK based score calculation component. This component implements pri-
vacy calculation for the Android device considering the requested and required
permissions of the installed applications. The idea is to assess permissions of the
application in context with its description available in mobile application store
(Google Play Store, Huawei App Gallery, etc.). The application description is
used to predict a set of permissions that the user expects from the app. For ex-
ample, the user does not expect the flashlight application to have access to the
contacts list but expects it for the social network application. The difference be-
tween predicted and actual permissions may serve as a basis for risk calculation.
Thus, the authors outline the following types of permission-based risks:
1. APK description-based privacy score with the range in [0,1.0];
2. APK permission-based privacy score with the range in [0,1.0];
3. APK based score that lies in the range [0,10] and could be transformed to
nominal risk value, i.e. High, Medium, Low.
Permissions could be grouped by the data type they may be associated with.
We outlined eight groups. Each group is assigned a weight wthat reflects crit-
icality of the permissions in the context of GDPR personal data types [25], for
example, the ”Health” permission group includes {BODY SENSORS}permis-
sion with w= 4 as it corresponds to ”Special” GDPR personal data type. The
predictions could be transformed into an 8-element binary vector, where each
element corresponds to the permission group. For actual permissions, the vector
is filled with 0, and the element is set to 1 if any permission from the permission
group is present in permissions of the APK application. For predicted permis-
sions, the vector is filled with 1 and the element is set to 0 if any permission
Security Measuring System for IoT Devices 7
from the permission group was predicted incorrectly based on the application
description. The APK permission-based privacy score P S is calculated as the
weighted sum of permissions. To calculate the description-based privacy score
DS, the authors first calculate the permission dissimilarity metric which is de-
fined as a difference in predicted and actual permissions. And then the weighted
sum of permissions from the obtained vector is calculated.
For example, for difference vector D= (0,0,1,0,0,0,0,0) and corresponding
weight vector W= (4,2,2,2,2,2,1,0), the description-based risk score DS is
calculated as follows:
DS =0∗4+0∗2+1∗2+0∗2+0∗2+0∗2+0∗1+0∗0
4+2+2+2+2+2+1+0 = 0.33.
The integral APK based score is calculated using the Algorithm 1.
Algorithm 1
1: log base =round(10 ∗P S)
2: if log base < e then
3: ipps =P S ∗ln(1 + ln(1 + DS))
4: else
5: ipps =P S ∗loglog base(1 + ln(1 + DS))
6: end if
7: if ipps > 1then
8: ipps = 1
9: end if
10: return ipps ∗10
In the provided algorithm P S – permission-based score in range [0.0; 1.0],
DS – description based score (risk) in range [0.0; 1.0]; ipps – the output integral
permissions based risk score in range [0.0; 10].
The underlying idea of the algorithm is as follows. The permission-based
score P S reflects risks associated with permissions requested by the application.
Description-based score DS reflects the risks that are calculated according to the
conformance between permissions requested by the application and its descrip-
tion, therefore it can be considered as conformance between requested data and
purposes they are collected. Thus if purposes are unclear and the corresponding
score is high, we need to increase the integral score, however, if the purposes are
clear, the integral score is defined by permission-based score, as risks associated
with usage of personal data are still present.
Ontology based privacy score calculation component. The basis of the
privacy risk assessment implemented by this component is an ontology that pro-
vides a formal representation of personal data processing scenarios. The proposed
system uses ontology and privacy risk calculation algorithm described in [6]. It
8 E. Doynikova et al.
provides a formal description of three basic personal usage scenarios - first party
collection and usage, third party sharing, and data retention.
Each personal data usage scenario is described by a set of linked concepts
that correspond to different attributes of a given usage scenario, for example,
type of personal data being collected or shared, the purpose of data processing,
retention time, etc.
However, the key concept is Data and its sub-classes such as Sensitive data,
User Account Info, Tracking Data, App & Dev Info, User Financial Data, etc,
that define the risk score base. Other concepts of usage scenario could either
increase it or decrease. Thus, the generic scheme for scenario risk calculation is
defined as follows:
P DDataU sageS cenarioRisk =P DRisk ScoreB ase ∗riskCoef f,
where P DDataU sageS cenarioRisk is a privacy risk score for the particular us-
age scenario, e.g. data retention, P DRiskS coreBase is a risk base calculated on
the basis of personal data types used in the usage scenario and their criticality.
riskC oef f is a risk coefficient that is defined on the basis of other usage scenario
concepts, i.e. purpose, and legal basis, opt-in/opt-out choices. To calculate risk
coefficient riskC oef f , it is necessary to determine concepts relating to a given
usage scenario except for the Data concept. Each concept has sub-classes or cat-
egories, for example, the Retention Time concept has 4 categories (sub-classes):
Not Defined, Stated, Indefinite and Other. For each category, it is possible to
determine their criticality level in a manner similar to the criticality defined
for categories of Data concept. Then riskC oef f is calculated on the basis of
categories of the related concepts with highest criticality [6].
Currently, the final ontology based score is calculated as a mean sum of
privacy risk scores calculated for each usage scenario detected in the privacy
policy.
Readability score calculation component. The readability metric relates
to the group of privacy aware metrics and reflects an indicator of the ease or dif-
ficulty of reading any text and, as a consequence, the difficulty of understanding
it [14]. If, for example, the text of a product privacy policy is difficult to read,
then there is a risk of misunderstanding how user data is used. Therefore, the
readability relating risks are also needed to be addressed.
The security measuring system uses the Flesch-Kincaid Grade Level read-
ability (FKGLR) to assess readability risks. This is a fairly well-known formula
that is used to test the difficulty of written texts.
The conclusion about the readability indicator of a specific text is made based
on the resulting FKGLR number, namely, based on the following intervals:
–F KGLR = [0,6] – low level of readability risk, the text is very easy to read;
–F KGLR = (6,10] – low level of readability risk, the text is simple for the
average reader;
–F KGLR = (10,12] – average level of readability risk, the text is somewhat
more complicated for the average reader;
Security Measuring System for IoT Devices 9
–F KGLR = (12,∞] – high level of readability risk, complex text, loyal to an
experienced reader who is ready to read scientific texts.
CIA score calculation considering attack traces component. This com-
ponent considers that device vulnerability depends on relations between the
vulnerabilities detected for the device and the network configuration, i.e. on the
attack traces. An attack trace is a sequence of atomic attack actions where each
action corresponds to the vulnerability exploitation. For example, some vulner-
ability v1 of the device has a low risk because it requires user privileges for
exploitation. But if there is another vulnerability v2 that allows obtaining the
required privileges we get the attack trace that increases the risks: v2−> v1.
The component takes logs from the device as input and implements the
following steps:
1. Extract networks, namely the devices that are connected to the same Gate-
way.
2. Extract devices within each network (their types and roles) to determine
their vulnerabilities.
3. Determine CVEs of vulnerabilities of the devices.
4. Classify CVEs. To generate attack traces it is necessary to determine pre and
post conditions of the vulnerabilities exploitation on the basis of the CVSS
of version 3 (CVSSv3) metrics: Attack Vector (AV), Required Privileges, and
Obtained Privileges. The authors outline 5 groups of vulnerabilities based
on their characteristics, they are shown in Table 1.
5. Generate CVE-based trees. In this step, CVE-based attack traces are gen-
erated considering relations between 5 outlined groups in Figure 4.
6. Calculate the CIA score considering attack traces for the device. The CIA
score is calculated on the basis of CVSSv3 scores as in the static case. Con-
fidentiality, integrity, and availability impact for the trace are calculated as
the maximum impact of its vulnerabilities. Total exploitability for the attack
trace is calculated as the product of the maximum AV score of the vulnera-
bilities in the trace, Access Complexity for all vulnerabilities in the trace, and
Privileges Required and User Interaction scores (depending on their values).
The CIA score for the trace is calculated based on the modified impacts and
exploitability. Finally, the CIA score considering attack traces for the device
is calculated as the maximum of CIA scores from all traces of the device.
The component outputs the attack traces based CIA score in the range
[0; 10] that could be further transformed to the qualitative CIA score in range
{low, medium, high}.
Dynamic CIA and privacy score calculation components. This com-
ponent updates security and privacy scores in dynamics based on the detected
anomalies. The confidentiality, integrity, availability, and privacy anomalies are
outlined. To detect anomalies statistics based and ML based methods are used,
which results are integrated. Depending on the number of the detected anomalies
10 E. Doynikova et al.
Table 1. Vulnerability groups
Group
\Characteristic AV Required
Privileges
Obtained
Privileges
V0 Network OR Local None None
V1 Network OR Local None !None
V2 Any Equal to Obtained
Privileges of V1 Any
V3 Adjacent Network None Any
V4 Any Equal to Obtained
Privileges of V3 Any
Fig. 4. The connections between the vulnerabilities of different groups
the weight coefficients are calculated for privacy score and exploitability score.
Exploitability score is a dynamic part of the CIA score and should be changed
to recalculate dynamic CIA.
For a given device or user the component builds feature vectors based on
timescale log aggregation first. Then it computes the normal range for different
feature values for all devices or users and checks user or device activity searching
for significant deviations known as anomalies.
The selected features are constructed based on a time-aggregated count of
messages considering values of specific attributes of source logs (e.g. errors). For
assessment of device activity based on computed features, the time interval of
60 seconds was selected. The experiments showed that such interval is enough to
track some minor changes in device activity and at the same time gives the ability
to generalize the device activity patterns. An example of overall device activity
on a certain time period and activity pattern for the device for 60 seconds is
shown in Figure 5.
Security Measuring System for IoT Devices 11
To determine the device’s normal behavior the range of normal values for each
feature is calculated. We used sequentially the following methods: Interquartile
range (iqr), Grubbs test, ESD-test, and Exponential smoothing.
Normal feature values and device features are used as input for anomaly de-
tection. To detect time intervals with anomalies the component: (1) compares the
feature values for each device with ranges of normal values to detect anomalous
activity for devices and calculates an anomaly intensity defined as the relation
of the distance between feature value and closest bound of the range to feature
value, (2) uses Local Outlier Factor ML method [2] to detect anomalous activity
for devices, (3) integrates the results taking into account the anomaly intensity.
Besides, the following ML methods for detecting anomalies were investigated
before selecting Local Outlier Factor: one class support vector machines [3]; Iso-
lation Forest algorithm [4]; ellipsoidal data approximation [5]; artificial neural
networks of different structures (autoencoders, LTSM, recurrent networks).
Finally, the number of time intervals with anomalies of different types for each
device is calculated. It is used to calculate anomaly weights for exploitability and
privacy recalculation. These weights are calculated as a relation of anomaly time
intervals to all activity intervals.
Fig. 5. Device activity plot time interval = 60s.
Integral security and privacy scores calculation component. Integral
security and privacy scores calculation component implements calculation of
integral scores. There are several approaches for calculating integral security
metrics: expert (or table-based) approach; min-max approach; weighted sum
function.
The most common approach is a table-based approach that is used mostly
for nominal parameters. The first row and column of such table contain possible
values of input metrics, while the inner cells of the table contain values of an
integral score. For example, this approach is used facilitated risk analysis and
assessment process (FRAAP) proposed in [7]. An obvious benefit of such an ap-
proach is the transparency of the calculation procedure, however, creating tables
for more than three metrics is a quite complicated process. The min-max ap-
proach is usually used in the context of security measures selection and supposes
12 E. Doynikova et al.
minimization of such parameters as attack probability, attack impact, response
costs, while maximization of such parameters as benefit from security measures
implementation [19]. The approach based on weighted sum is also widely used,
for example, it is adopted for calculating CVSS scores [8]. Application of the
weighted sum requires setting ranks or weights for the metrics. In some cases,
the definition of metrics weights is a quite natural process. For example, when
calculating integral static privacy score based on privacy policy, it is necessary
to consider two metrics - readability score and ontology-based score. The read-
ability score characterizes the transparency of privacy policy, while information
about the usage of personal data is incorporated in the ontology-based score.
Thus, its priority is higher than the priority of readability score, and this differ-
ence in metrics priority could be easily reflected by weight coefficients. Currently,
we suggest using the following values of weight coefficients: weight coefficient for
ontology-based score wo= 0.9; weight coefficient for readability score wr= 0.1.
It should be also noted that the readability score lies in range [0,∞], and it
needs to be re-scaled to the range [0,∞], this could be done as follows:
1. If (rs > rescaling threshold) then rs =rescaling threshold.
2. Rescaled rs =rs ∗10/rescaling threshold.
The rescaling threshold artificially defines the possible maximum value of read-
ability score, it is set to 16 because the average range of readability score intervals
that define the different level of text difficulty is 4, and the lower border of the
interval that corresponds to the highest difficulty level of text is 12. Let os be
ontology based score in range [0,10], and rs – readability score in range [0,+∞]
then the integral policy–based privacy score calculation algorithm includes the
following steps:
1. Re-scale readability score rs to range [0; 10].
2. Calculate integral static privacy policy based score as pps =wo∗os +wr∗rs.
The algorithm outputs the integral score pps in the range [0,10]. The following
small example illustrates the calculation procedure of integral privacy score based
on analysis of privacy policy. Let ontology-based score os = 5.6, readability score
rs = 12, then privacy policy based score pps =wo∗os +wr∗RESCALE(rs) =
0.9∗5.6+0.1∗7.5=5.8
The weighted sum function could be used also for calculating integral privacy
and security static and dynamic scores, but in many cases it is not possible
to define what metric has a higher priority. To solve this problem, the authors
suggest the following algorithm for the case when all input metrics are considered
equally meaningful. The metric with the highest risk score serves as a basis, then
the values of other metrics are added, but firstly the logarithm dependent on their
values and maximum values is calculated to scale the value nonlinear. Authors
introduce non-linearity to avoid the fast growth of integral metric value. Let
SC ORES be a list of metrics with values in the range [0, 10], then the generic
algorithm for integral privacy and security calculation consists of the following
steps:
Security Measuring System for IoT Devices 13
1. If all metrics in SC ORES are not defined (or null), return not defined (or
null)
2. Set max score as a maximum element of SC ORES
3. Remove max score from SC ORES
4. Calculate integral score =max score +log(1 + sum(SCORES),
10 ∗length(SCORES))
5. If integral score > 10 set integral score as 10.
6. Return integral score.
The algorithm outputs the integral score integral score in the range [0,10]. The
following small example illustrates the calculation procedure of integral static
privacy & security score. Let static CIA score be 4.6, APK-based score be 3.0,
and integral privacy policy based score is 4.5. The maximum score is static CIA
score, and it serves as the basis of the integral score. Then the final static privacy
& security score is calculated as follows: integral score = 4.6 + log(1 + (3.0 +
4.5),10 ∗2) = 5.2.
To analyze the difference between the proposed algorithm and the weighted
sum, the authors implemented the following experiment. We considered the case
when three metrics are used to calculate integral score - static CIA score, APK-
based score, and privacy-policy based score. All these metrics are equally mean-
ingful and that is why corresponding weights were set equal to each other. Then
we evenly changed the values of two metrics (static CIA score, APK-based score)
from 0.0 till 10.0 while the value of privacy policy based score was fixed to 5.0,
and analyzed how the values of integral score changed. Figure 6 and Figure 7
show the difference in values of integral security & privacy metric when it is
calculated using the weighted sum and proposed algorithm. When the weighted
sum was used, the integral score changed linearly in the range from 1.7 to 8.3.
The score of 1.7 corresponds to the case when static CIA score and APK-based
score were set to 0, and the privacy policy based score was equal to 5. The score
of 8.3 corresponds to the case when two metrics have the highest scores. Thus,
the algorithm based on weighted sum reduces the values of the integral score
when one metric has either high or small score relatively other metrics, because
for the case when all metrics are equally important it simply averages the values.
The proposed algorithm does not reduce the highest value of the metric, as it is
selected as a base for the integral score, and this base is increased proportion-
ally to the values of the rest metrics. Figure 7 shows that integral score grows
slowly when the Static CIA and APK-based score are either small or compara-
ble with privacy policy based risk, but when these two metrics became greater
than privacy policy based risk (more than 6), the integral score starts growing
faster reaching the highest score when Static CIA and APK-based score equal to
8.6. So it could be concluded that the proposed algorithm produces cumulative
scores.
4 Implementation and Test Case
Common architecture of the developed system is provided in Figure 8. The
proposed system is implemented using Python. We use the PostgreSQL database
14 E. Doynikova et al.
Fig. 6. The values of integral security
& privacy metric when they are calcu-
lated with the weighted sum algorithm, all
weights are equal to each other.
Fig. 7. The values of integral security &
privacy metric values when they are calcu-
lated with the proposed algorithm.
to store information about the devices under analysis, values of metrics, normal
profiles of the devices, and intermediate data about detected anomalies in device
functioning. This database also contains data about known vulnerabilities in IoT
devices. This information is updated every 6 hours automatically or each time
the NVD update script is launched manually.
Fig. 8. Common architecture of the proposed security measuring system.
The suggested workflow with the security measuring system is as follows.
Initially, the system updates information about known vulnerabilities and takes
a description of privacy policy ontology as input data. Then user fills in a de-
vice specification, that may include device type, model, manufacturer, etc. This
information can be also automatically filled in using system and device logs.
Security Measuring System for IoT Devices 15
Based on this information the system automatically searches for and downloads
APKs, their description available on application marketplaces, privacy policy for
the given device or the device manufacture. These data could be also specified
manually by the system user. Afterward the system outputs the static integral
security & privacy score. This is done by implementation of the following steps:
1. Calculate readability score for the device/manufacture privacy policy.
2. Construct P2Onto ontology based on given template and calculate ontology-
based privacy score.
3. Calculate integral privacy policy based score.
4. Calculate static CIA score based on vulnerabilities associated with down-
loaded APKs.
5. Calculate description-based scores and permission-based scores for the APKs.
6. Calculate integral APK based score.
7. Calculate and output integral static security and privacy score.
To produce dynamic scores it is required to provide security measuring sys-
tem access to device and system logs. The system extracts from logs the following
information: information about the connections between the devices; information
about the devices, such as device internal characteristics, device state informa-
tion, time of login/logout; and information about the errors.
Afterward the system outputs the dynamic integral security & privacy score.
This is done by implementation of the following steps:
1. Process logs to generate attack traces based on the connections between the
devices and their known vulnerabilities.
2. Recalculate CIA score based on the generated traces.
3. Process logs to calculate features that describe device behavior.
4. Construct normal behavior device profiles based on normal values of features.
5. Use new portions of logs to detect anomalies. New logs are processed once a
day.
6. Calculate dynamic CIA score based on the detected anomalies.
7. Calculate dynamic privacy score based on the detected anomalies.
8. Calculate and output integral dynamic security and privacy score.
All calculated metrics are stored in the database and available to the user.
Let us consider the following example of security measuring system opera-
tion. The analyzed device is a smart lock produced by August company [24]. This
company produces devices for the smart home environment, such as smart locks,
doorbell cameras, and other accessories. Their smart lock allows implementing
a variety of convenient functions such as remote locking and unlocking the door,
logging exit/entrance activity of smart lock owners as well as their guests, sup-
porting identification, and voice assistant. To obtain the static integral security
and privacy score, the system uploaded the following data:
–the application August Home 11.5.1 for Android that manages the activity
of smart devices including smart lock and its description from APKPure
market place;
16 E. Doynikova et al.
–privacy policy for website and products from manufacture’s site [24].
The obtained readability score was defined as 13.0, that corresponds to the
text with high difficulty.
The analysis of privacy policy using its ontology representation revealed some
interesting scenarios such as a collection of personal data from guests who visit
the owner of a smart lock. Figure 9 shows the collection of financial data by the
first party. This type of data is collected in order to provide Internet payment
services and legal authorization. The calculated ontology based score is equal to
8.37, that is quite high, but could be explained by a variety of data types - user
account data, application and device data, tracking and financial information -
being collected and shared.
Thus, the integral privacy policy based score is 8.34.
Fig. 9. First party collection of financial data presenting using P2Onto ontology.
The analysis of APK permissions and description showed that the actual per-
missions set for the ”August Home” app is {READ CONTACTS, ACCESS FINE LOCATION,
ACCESS COARSE LOCATION, READ EXTERNA- L STORAGE, READ PHONE STATE,
CAMERA, RECORD AUDIO}, while a set of predicted permissions - {ACCESS FINE LOCATION,
READ EXTERN- AL STORAGE, READ PHONE STATE, RECORD AUDIO}.
Thus, permissions READ CONTACTS, ACCESS COARSE LOCATION’, CAM-
ERA were not predicted. This resulted in the following values for permission and
description based scores:
–permission-based risk score DS = 0.6;
–description-based risk score P S = 0.4.
Thus, APK based score equals 7.1.
For the given APK security measuring system detected the following CPE
entry in NVD database: cpe:2.3:a:august:august home:-:*:*:*:*:android:*:*, and
the corresponding CVE: CVE-2019-17098 (Figure 10). Thus, the static CIA score
is 6.5 considering the CVSS score of CVE-2019-17098 and device criticality.
Integrating scores of static CIA score, APK based score, and integral privacy
policy based score we obtain integral static security & privacy score equal to 9.2
Security Measuring System for IoT Devices 17
Fig. 10. Detected CVEs for smart lock APK.
when calculating using proposed non-linear algorithm and 7.4 when calculating
using the weighted sum function.
The next part of the case study is related to dynamic assessments. The log
processing and integration component of the security measuring system pro-
cesses the logs first. Input logs are represented as series of csv files, partitioned
by different days of activity. The component aggregates and normalizes the het-
erogeneous data in logs of various types. The resulting integrated log contains
attributes representing the time of message registration in the log, the connec-
tions between the devices, information about the devices (such as device ID,
its model, IP-address, etc.), device state information, time of its login/logout,
and information about the errors. After defining the set of attributes for the
integrated log, messages are combined and sorted by time, and an additional
attribute is introduced indicating the type of log that contains the message.
The resulting integrated log is used to generate attack traces based on the
connections between the devices and their known vulnerabilities first. Based on
this log the security measuring system detects a connection between the smart
lock and August connect (bridge). The trace is generated based on the smart lock
APK vulnerability (CVE-2019-17098) and August connect vulnerability (CVE-
2018-20100). The dynamic CIA score based on the attack traces calculation com-
ponent classified CVE-2019-17098 as V3 and CVE-2018-20100 as V1 according
to Figure 4. As soon as CVE-2018-20100 helps to obtain admin privileges, there
is a following trace between the devices: CVE-2018-20100 ->CVE-2019-17098.
It increases the exploitability of CVE-2019-17098 considering traces from 2.84
to 2.99. CIA score changes from 6.5 to 6.59, that doesn’t influence on integral
static security & privacy score. High integral static security & privacy score val-
ues grows rather slowly with growth of the CIA score (the CIA score should
increase on 1.9 to affect the integral score).
On the next step the security measuring system processes the logs to calculate
features that describe device behavior and constructs normal behavior device
profiles based on normal values of features. Further new logs are processed once
a day to detect anomalies in the device behavior. If anomalies are detected then
dynamic CIA score or dynamic privacy score is recalculated depending on the
type of the detected anomaly that, in its turn, changes the integral dynamic
security and privacy score. The authors used the generated logs with anomalies
to test the dynamic scores calculation. Detected anomalies affect the CIA score
and privacy score (depending on anomaly type). Thus, 10% of CIA anomalies
detected in log resulted in changing the CIA score from 6.59 to 6.7. But, as it
is said above the CIA score should increase on 1.9 to affect the integral static
security & privacy score and it stays 9.2.
18 E. Doynikova et al.
5 Discussion and Conclusion
The paper described the developed security measuring system for IoT devices.
The introduced system is based on the novel hierarchy of the security and pri-
vacy metrics, algorithms for their calculation, and integral scores calculation
algorithms. The developed system automates all stages of data gathering, pro-
cessing and analysis for calculation of the selected metrics and their recalculation
when new data arrives. The difference of the proposed measuring system from
the other analogical frameworks consists in consideration of different security
and privacy aspects for comparison of the security level of different IoT devices
and integration of these aspects in one integral security and privacy score. The
underlying approach supposes calculating the base static score for the device
using its internal characteristics and its further recalculation in dynamics con-
sidering new obtained data, such as data on device connections and behavior.
It is implemented via the following steps: calculate readability score for the
device/manufacture privacy policy; construct P2Onto ontology based on given
template and calculate ontology-based privacy score; calculate integral privacy
policy based score; calculate static CIA score based on vulnerabilities associ-
ated with downloaded APKs; calculate description-based scores and permission-
based scores for the APKs; calculate integral APK based score; calculate and
output integral static security and privacy score; get and process new portion of
logs; construct normal device behavior profile; get and process new portion of
logs every specified time interval; recalculate CIA score considering connections
between the devices in logs and constructing possible attack traces; check for
anomalies in normal device behavior; recalculate scores if there are anomalies.
The whole process is demonstrated on the case study for the IoT device.
While novel algorithms were proposed for separate metrics the main con-
tribution consists in the algorithms for the integral security and privacy scores
calculation. There are different approaches to integral scores calculation, includ-
ing the expert (or table-based) approach [7], the min-max approach [19] and the
approach based on weighted sum [8]. We selected the weighted sum approach
but modified it as soon as all input metrics are considered equally meaningful. If
we would set corresponding weights equal to each other the algorithm would re-
duce the values of integral security and privacy score because it simply averages
the values. The authors consider it unacceptable in the case of the security and
privacy scores. Thus, in the proposed algorithm the metric with the highest score
serves as a basis, while the values of other metrics are added to it, but firstly
the logarithm dependent on their values and maximum values is calculated to
scale the value nonlinear. The authors introduced non-linearity to avoid the fast
growth of integral metric value. The experiments showed that the proposed al-
gorithm does not reduce the highest value of the metric, as it is selected as a
base for the integral score, and this base is increased proportionally to the values
of the rest metrics.
There are some features of the proposed system that can and should be im-
proved in future work. New metrics, such as authenticity and transparency can
be added. Currently, device criticalities that are used to calculate static CIA
Security Measuring System for IoT Devices 19
score are set depending on the device type. In the future calculation of the criti-
calities can be automated considering the device’s role in the system. Automated
searching for CPEs and CVEs should be improved as soon as because of lack of
unification and errors sometimes they are missed. The anomaly detection pro-
cess can be enhanced by introducing new complex features. Besides, anomaly
detection by device type profile should be added. Integration with intrusion de-
tection systems can enhance the solution as well. And in the future work the
authors plan to add the security recommendations that will allow improving the
calculated metrics.
And finally, the main challenge consists in the verification of the proposed
security measuring system. While separate algorithms were tested on the exper-
iments and applicability of the proposed system was shown on the use case, the
only usage of the device and statistics on real successful incidents can demon-
strate if the calculated security and privacy scores were correct. In future work
the authors plan to research and overcome this challenge by developing the test
stand for scoring and compromising the IoT devices.
References
1. Ahmed, M., Mahmood, A.N., Hu, J., 2016. A survey of network anomaly detection
techniques. Journal of Network and Computer Applications 60, 19–31.
2. Local Outlier Factor. URL: https://en.wikipedia.org/wiki/Local outlier factor.
3. Sch¨olkopf, B., Platt, J. C., Shawe-Taylor, J., Smola, A. J., Williamson, R. C. (2001).
Estimating the Support of a High-Dimensional Distribution. Neural Computation,
13(7), pp. 1443-1471. doi:10.1162/089976601750264965.
4. Liu, F. T., Ting, K. M., Zhou, Z.-H. (2012). Isolation-Based Anomaly Detec-
tion. ACM Transactions on Knowledge Discovery from Data, 6 (1), pp. 1–39.
doi:10.1145/2133360.2133363
5. Rousseeuw, P.J., Van Driessen, K. “A fast algorithm for the minimum covariance
determinant estimator” Technometrics 41(3), 212 (1999).
6. Novikova E., Doynikova E., Kotenko I. (2020) P2Onto: Making Privacy Policies
Transparent. In: Katsikas S. et al. (eds) Computer Security. CyberICPS 2020,
SECPRE 2020, ADIoT 2020. Lecture Notes in Computer Science, vol 12501.
Springer, Cham. https://doi.org/https://doi.org/10.1007/978-3-030-64330-0 15
7. Peltier T.R.: Information security risk analysis, 3d edition, CRC Press, 2010, 456 p.
8. Common Vulnerability Scoring System v3.1: Specification Document, URL:
https://www.first.org/cvss/specification-document (access date: 12/29/2019)
9. IoT Security Foundation official web-site, URL:
https://www.iotsecurityfoundation.org/best-practice-guidelines (access date:
30/07/2021)
10. Doynikova, E., Chechulin, A., Kotenko, I.: Analytical attack modeling and secu-
rity assessment based on the common vulnerability scoring system. In: Proceed-
ings of the XXth Conference of Open Innovations Association FRUCT, pp. 53–61.
10.23919/FRUCT.2017.8071292 (2017)
11. Kincaid, J.P., Fishburne, R.P., Rogers, R.L., Chissom, B.S.: Derivation of new
readability formulas (automated readability index, fog count, and flesch reading
ease formula) for Navy enlisted personnel. Research Branch Report 8–75. Chief of
Naval Technical Training: Naval Air Station Memphis (1975)
20 E. Doynikova et al.
12. Ardagna C.A., De Capitani di Vimercati S., Samarati P.: Enhancing User Privacy
Through Data Handling Policies. In: Damiani E., Liu P. (eds.) Data and Applica-
tions Security 2006, LNCS, vol. 4127. Springer, Berlin, Heidelberg (2006)
13. Pardo R., Le M´etayer D.: Analysis of Privacy Policies to Enhance Informed Con-
sent. In: Foley S. (eds.) Data and Applications Security and Privacy XXXIII. DBSec
2019, LNCS, vol. 11559. Springer, Cham (2019)
14. Welderufael B. Tesfay, Hofmann P., Nakamura T., Kiyomoto S., Serna
J.: PrivacyGuide: Towards an Implementation of the EU GDPR on In-
ternet Privacy Policy Evaluation. In: Proceedings of the Fourth ACM In-
ternational Workshop on Security and Privacy Analytics (IWSPA ’18), pp.
15–21. Association for Computing Machinery, New York, NY, USA (2018).
https://doi.org/https://doi.org/10.1145/3180445.3180447
15. Wei, R., Cai, L., Yu, A., Meng, D.: AGE: Authentication Graph Embedding for
Detecting Anomalous Login Activities (2020). https://doi.org/10.1007/978-3-030-
41579-2 20
16. National Cyber Security Center official web-site. NCSC CAF guidance,
https://www.ncsc.gov.uk/collection/caf/cyber-assessment-framework. Last ac-
cessed 30 Jul 2021
17. IoT Security Foundation, IoT Security Compliance Framework, Re-
lease 2, December 2018, https://www.iotsecurityfoundation.org/wp-
content/uploads/2018/12/IoTSF-IoT-Security-Compliance-Framework-Release-
2.0-December-2018.pdf. Last accessed 30 Jul 2021
18. Najib, Warsun, Sulistyo, Selo, Widyawan, Widyawan: Survey on Trust Calculation
Methods in Internet of Things. In: Procedia Computer Science, 161, pp. 1300–1307
(2019). https://doi.org/10.1016/j.procs.2019.11.245
19. MHR Khouzani, Zhengliang Liu, Pasquale Malacaria, “Scalable min-max multi-
objective optimization over probabilistic attack graphs”, European Journal of Op-
erational Research, Vol. 278, Issue 3, 2019, P. 894-903
20. De, S. J., Metayer, D. Le.: Privacy Risk Analysis to Enable Informed Privacy
Settings. In: 2018 IEEE European Symposium on Security and Privacy Workshops
(EuroS&PW), pp. 95–102. London (2018)
21. Bar-Sinai, M., Sweeney, L., Crosas, M.: DataTags, Data Handling Policy Spaces
and the Tags Language. In: 2016 IEEE Security and Privacy Workshops (SPW),
pp. 1–8. San Jose, CA (2016)
22. Le M´etayer D.: A Formal Privacy Management Framework. In: Degano P.,
Guttman J., Martinelli F. (eds.) Formal Aspects in Security and Trust. FAST 2008,
LNCS, vol. 5491. Springer, Berlin, Heidelberg (2009)
23. Pandit H.J., Fatema K., O’Sullivan D., Lewis D.: GDPRtEXT - GDPR as a Linked
Data Resource. In: Gangemi A. et al. (eds.) The Semantic Web. ESWC 2018. LNCS,
vol. 10843. Springer, Cham (2018)
24. August Device and Service Privacy Policy Homepage. Available online:
https://august.com/pages/privacy-policy (accessed on 30 March 2021)
25. General Data Protection Regulation (GDPR), https://gdpr-info.eu/. Last accessed
31 Jul 2021
