Article

Maturity Level Assessments of Information Security Controls: An Empirical Analysis of Practitioners Assessment Capabilities

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Maturity models are a widely used concept for measuring information security. The idea is to systematically evaluate the maturity of security-relevant processes in an organisation. This enables decision makers to get an overview of the implementation status of relevant processes to identify neuralgic points. Maturity models thus play a central role in the conception of information security management systems (ISMS). Some industries, for instance, the German automotive industry, have even established security maturity levels as the de facto standard for measuring information security. However, the quality of security maturity level assessments has not been sufficiently investigated yet. We have analysed to what extent security managers can accurately assess the maturity levels of security controls. To verify the quality of maturity level assessments a case study was conducted where security experts assessed a subset of the ISO/IEC 27002 security controls for a hypothetical scenario using the COBIT maturity levels. Additionally, ex-post interviews have been conducted with several study participants to verify some of the hypotheses developed during the previous analyses. Our results show that many security experts struggled with the task and did not perform well. However, we discovered professional characteristics that have a strong significant effect on the assessment capabilities. We also identified various types of additional support that can help practitioners to make more reliable assessments in practice. Moreover, the experts self-perception was overly optimistic when asked to assess their performance. We even found a weak inverted correlation for more experienced experts, also known as Dunning-Kruger effect. Our results have a strong impact on practise since they indicate that practitioners need support to carry out high-quality assessments and they also show what kind of support addresses the identified challenges.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... In this study there are also suggestions, first suggestions it is hoped that Communication and Information Department Mojokerto can improve information system security management, rules, and information system security procedures so that external and internal threats related to information security can be controlled [17]. ...
Article
Full-text available
This study aims to provide an overview to the Communication and Information Department Mojokerto regarding the maturity level of physical and environmental security management at the agency, as well as to provide future recommendations. The results of research related to physical and environmental safety using the ISO 27002 standard, indicate that the level of physical and environmental security at the Communication and Information Department Mojokerto is still relatively low. Things that are still lacking include the lack of protection from external threats such as natural disasters, as well as the lack of care and maintenance of infrastructure. The maturity level of physical and environmental security control is 0.85 which is still at level 1 or Initial Ad Hoc from a maximum value of 5, which is at the Optimized level. It can be concluded that Communication and Information Department Mojokerto only knows that there are things that need attention but there is no standardization of the process. With this research, it is hoped that the Communication and Information Department Mojokerto can make improvements to improve physical and environmental security. In addition, it is also a consideration to obtain ISMS Certification with the ISO 27002 standard in the future.
Article
Businesses use sentiment analysis for market insights to improve performance. Sentiment analysis application in building a smart society is immense, and there is a need to delineate the trend holistically. The present study addresses the research gap by a comprehensive bibliometric review of 353 research articles published between 2010 and 2021 to discern the performance, content, and thematic analysis. Finding points at the use of sentiment analysis for innovation, transparency, citizen participation, and improved efficiency in public services. The content analysis points to the applicability of sentiment analysis for citizens' engagement to solve social problems like traffic congestion, crime prediction, disaster management, etc. According to the strategic map, information retrieval and supervised learning are the motor themes; sentiment analysis and ubiquitous computing are the basic themes, while semantics and recommender systems are the niche themes. Thematic analysis of intellectual structure indicates sentiment analysis applications for smart governance, smart mobility, smart infrastructure, and smart living as the building blocks for smart societies. Based on a comprehensive review, future research directions and managerial implications are provided.
Article
Full-text available
Information security risk assessment frameworks support decision-makers in assessing and understanding the risks their organisation is exposed to. However, there is a lack of lightweight approaches. Most existing frameworks require security-related information that are not available and that are very challenging to gather. So they are not suitable in practice, especially for small and medium-sized enterprises (SMEs) who often lack in data and in security knowledge. On the other hand, other explicit SME approaches have far less informative value than the proposed framework. Moreover, many approaches only provide extensive process descriptions that are challenging for SMEs. In order to overcome this challenge, we propose LiSRA, a lightweight, domain-specific framework to support information security decision-making. It is designed with a two-sided input where domain experts initially provide domain-specific information (e.g. attack scenarios for a specific domain), whereupon users can focus on specifying their security practices and organisational characteristics by entering information that many organisations have already collected. This information is then linked to attack paths and to the corresponding adverse impacts in order to finally assess the total risk. Moreover, LiSRA can be used to get transparent recommendations for future security activities and presents detailed insights on the mitigating effects of each recommendation. The security activities are being evaluated taking into account the security activities already in place, and also considering the dependencies between multiple overlapping activities that can be of complementary, substitutive or dependent nature. Both aspects are ignored by most existing evaluation approaches which can lead to an over-investment in security. A prototype has been implemented, and the applicability of the framework has been evaluated with performance and robustness analyses and with initial qualitative evaluations.
Conference Paper
Full-text available
Generally, measuring the information security maturity is the first step to build a knowledge information security management system in an organization. Unfortunately, it is not possible to measure information security directly. Thus, in order to get an estimate, one has to find reliable measurements. One way to assess information security is by applying a maturity model and assess the level of controls. This does not need to be equivalent to the level of security. Nevertheless, evaluating the level of information security maturity in companies has been a major challenge for years. Although many studies have been conducted to address these challenges, there is still a lack of research to properly analyze these assessments. The primary objective of this study is to show how to use the analytic hierarchy process (AHP) to compare the information security controls’ level of maturity within an industry in order to rank different companies. To validate the approach of this study, we used real information security data from a large international media and technology company.
Conference Paper
Full-text available
A web-based platform was developed to support the inter-organisational collaboration between small and medium-sized energy providers. Since critical infrastructures are subject to new security regulations in Germany, the platform particularly serves for the exchange of experience and for mutual support in information security. The focus of this work is the security self-assessment component. In order to ease the burden of going through a long questionnaire we have implemented small, motivating modules that are spread across the platform. The data entered is used for an individual risk assessment but also for a fine granular inter-organisational security benchmarking which builds a common added value for the entire community on the platform and strengthens the community building process. We implemented a prototype of the platform and evaluated the it in a focus group.
Article
Full-text available
A Maturity Model is a widely used technique that is proved to be valuable to assess business processes or certain aspects of organizations, as it represents a path towards an increasingly organized and systematic way of doing business. A maturity assessment can be used to measure the current maturity level of a certain aspect of an organization in a meaningful way, enabling stakeholders to clearly identify strengths and improvement points, and accordingly prioritize what to do in order to reach higher maturity levels. This paper collects and analyzes the current practice on maturity models, by analyzing a collection of maturity models from literature.
Article
Full-text available
Information is a fundamental asset within any organization and the protection of this asset, through a process of information security is of equal importance. COBIT and ISO27001 are as reference frameworks for information security management to help organizations assess their security risks and implement appropriate security controls. One of the most important sections of IT within the COBIT framework is information security management that cover confidentiality, integrity and availability of resources. Since the issues raised in the information security management of COBIT, are the area covered by the ISO/IEC27001 standard, the best option to meet the information security management in COBIT infrastructure, is using of ISO/IEC27001 standard. For coexistence of and complementary use of COBIT and ISO27001, mapping of COBIT processes to ISO/IEC 27001 controls is beneficial. This paper explores the role of information security within COBIT and describes mapping approach of COBIT processes to ISO/IEC27001 controls for information security management.
Technical Report
Full-text available
Measurement is one of the foundations of sound engineering practices, because—as Tom DeMarco put it—you cannot control what you can’t measure. This principle should also apply to software security engineering. However, providing useful metrics or at least indicators for characterizing the security properties of a software system is surprisingly challenging. The research community is well aware of the urgent need for security metrics, and it has put significant research effort into this field. Numerous qualitative and quantitative security measures have been proposed in the scientific literature, but few of them found wide-spread adoption by practitioners. Due to the significant body of work, it has become increasingly difficult to overlook the state of the art in specifying, determining, comparing, or predicting security qualities. This report surveys the published work on security indicators. In the context of this survey, a security indicator is understood as an observable characteristic that correlates with a desired security property. Our survey covers current research into qualitative and quantitative security indicators as well as applied key performance indicators and security standards. We developed a uniform classification scheme for categorizing and comparing the indicators that we elicited. Based on this classification, our survey reveals trends and deficiencies in security research and security practice. It also suggests explanations for the apparent difficulties in providing meaningful security indicators. Moreover, our classification can guide practitioners to adequate methods for the specification of security requirements and for the measurement of relevant security attributes of their products and processes.
Article
Full-text available
For many years, we've been trying to measure "security" so that we can increase accountability, demonstrate compliance, and determine whether and by how much our investments in products and processes are making our systems more secure. This article investigates why security measurement is difficult and what strategies might help address our needs.
Article
Full-text available
To ensure security, it is important to build-in security in both the planning and the design phases and adapt a security architecture which makes sure that regular and security related tasks, are deployed correctly. Security requirements must be linked to the business goals. We identified four domains that affect security at an organization namely, organization governance, organizational culture, the architecture of the systems, and service management. In order to identify and explore the strength and weaknesses of particular organization's security, a wide range model has been developed. This model is proposed as an information security maturity model (ISMM) and it is intended as a tool to evaluate the ability of organizations to meet the objectives of security.
Article
Full-text available
People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities.
Article
Management of information without regard to risk the achievement of enterprise goals can have an impact on organizational performance, financial loss or organization's credibility. The risk control for the negative effects and utilization of chance in achieving enterprise goals is called information security. Information security are generally solved by partial and limited. It also happens to INTRAC that apply only management area of information security by adopting ISO/IEC 27001:2009 and ISO/IEC 27002:2005. This study aims to develop process assessment model that support the implementation of information security governance on an organization. The method used in this study is qualitative method. Based on the validation by expert judgment, information security governance model has been prepared in accordance with the requirements of information security, particularly in the INTRAC.
Article
People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of the participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Control Objectives for Information and Related Technology (COBIT) becomes very popular in recent years and is regarded as the most comprehensive IT governance framework. However, its actual utilization and effectiveness are not clear due to the lack of academic studies. Also, the proliferation of other IT standards and best practices, such as ISO27000 series and ITIL, creates great challenges for organizations to understand their relations and to take advantage of them. The main objective of this research is to explore the practicability of COBIT framework and its actual usage. A pilot COBIT program within an IT department was carried out to collect primary data. The actual usage of COBIT tools is analyzed and compared to their theoretical design. Practical problems of COBIT framework are identified. A COBIT-BSC model is proposed to illustrate a simple way of structuring COBIT control objectives. This study will contribute some practical insights to COBIT framework and help organizations take advantage of COBIT as well as other IT control frameworks.
Conference Paper
Twelve years ago a development program for security managers was started in Finland. The first program was designed to fulfil the needs of a security manager in an organization. However, the content was an educated guess. During the second program, in 1993, we made a study on how security managers themselves feel the requirements of their work and which part of their work they felt difficult or easy. We were interested in what kind of education security managers would need. These results were used when the following programs were planned and some extra courses were introduced. However, these results were never really published. Now, we try to improve the development program for security managers again. We took the results of the old study and found out the current situation. We noticed that the requirements for a security manager have changed. The security managers have become a manager of a department instead of a single specialist. They do not need as deep specialized knowledge as nine years ago. Instead they need understanding of business processes and managerial skills. Another finding is that security manager is a long term career. Few security managers have proceeded to higher vacancies. Instead many of those security managers who participated the old study as a security manager are now retiring from that very same position.
Conference Paper
Planning information security investment is somewhere between art and science. This paper reviews and compares existing scientific approaches and discusses the relation between security investment models and security metrics. To structure the exposition, the high-level security production function is decomposed into two steps: cost of security is mapped to a security level, which is then mapped to benefits. This allows to structure data sources and metrics, to rethink the notion of security productivity, and to distinguish sources of indeterminacy as measurement error and attacker behavior. It is further argued that recently proposed investment models, which try to capture more features specific to information security, should be used for all strategic security investment decisions beneath defining the overall security budget.
Conference Paper
The emerging ISO/IEC 15504 standard provides a framework and a model for software process assessment and improvement. There are two requirements for reliable process assessment: internal reliability and external reliability. The objective of the study is to provide an empirical case of external reliability, i.e. the interrater agreement in ISO/IEC 15504-based software process assessment. Interrater agreement implies the extent to which independent assessors agree in their ratings of software process attributes. Our dataset was from two assessments conducted using the ISO/IEC 15504 standard. The results showed "substantial" to "excellent" agreement. This implies that the two assessments acquired external reliability
Cost of a data breach report
  • Ibm Security
Tisax participant handbook
  • F Gleich
F. Gleich, Tisax participant handbook, 2019.
Cybersecurity capability maturity model (c2m2)
  • Christopher
Assessor agreement in rating spice processes
  • Emam
NIST Releases Version 1.1 of its Popular Cybersecurity Framework
  • NIST
Information security assessment
  • Automobilindustrie Verband Der
Verband der Automobilindustrie (VDA), Information security assessment, https://www.vda.de/de/services/Publikationen/informationsecurity-assessment.html, 2020.
Information Security Handbook: A Guide for Managers
  • P Bowen
  • J Hash
  • M Wilson
  • C M Gutierrez
  • W Jeffrey
P. Bowen, J. Hash, M. Wilson, C. M. Gutierrez, W. Jeffrey, Information Security Handbook: A Guide for Managers, NIST Special Publication 800-100 (2006).
International Electrotechnical Commission (IEC), ISO/IEC 15504-5:2012, information technology -process assessment -part 5: An exemplar software life cycle process assessment model
International Organization for Standardization (ISO), International Electrotechnical Commission (IEC), ISO/IEC 15504-5:2012, information technology -process assessment -part 5: An exemplar software life cycle process assessment model, 2012.
CobiT 5: A Business Framework for the Governance and Management of Enterprise IT
Information Systems Audit and Control Association (ISACA), CobiT 5: A Business Framework for the Governance and Management of Enterprise IT, Rolling Meadows, 2012.
Bundesamt für Sicherheit in der Informationstechnik
Bundesamt für Sicherheit in der Informationstechnik, Zuordnungstabelle ISO zum modernisierten IT-Grundschutz, 2018.