Conference PaperPDF Available

Developing a Framework for Trustworthy AI-Supported Knowledge Management in the Governance of Risk and Change


Abstract and Figures

This paper proposes a framework for developing a trustworthy artificial intelligence (AI) supported knowledge management system (KMS) by integrating existing approaches to trustworthy AI, trust in data, and trust in organisations. We argue that improvement in three core dimensions (data governance, validation of evidence, and reciprocal obligation to act) will lead to the development of trust in the three domains of the data, the AI technology, and the organisation. The framework was informed by a case study implementing the Access-Risk-Knowledge (ARK) platform for mindful risk governance across three collaborating healthcare organisations. Subsequently, the framework was applied within each organisation with the aim of measuring trust to this point and generating objectives for future ARK platform development. The resulting discussion of ARK and the framework has implications for the development of KMSs, the development of trustworthy AI, and the management of risk and change in complex socio-technical systems.
Content may be subject to copyright.
Developing a Framework for Trustworthy AI-Supported
Knowledge Management in the Governance of Risk and
Rebecca Vining1,*[0000-0002-2715-8129], Nick McDonald1, Lucy McKenna2[0000-0002-6035-7656],
Marie E Ward1,3[0000-0002-6638-8461], Brian Doyle1,4[0000-0002-9106-9526], Junli Liang2[0000-1111-
2222-3333], Julio Hernandez2[0000-0003-1347-9631], John Guilfoyle4, Arwa Shuhaiber5, Una
Geary3, Mary Fogarty3, Rob Brennan2[0000-0001-8236-362X]
1 Centre for Innovative Human Systems, School of Psychology, Trinity College, The University
of Dublin, D02 PN40 Dublin, Ireland
2 ADAPT Centre, School of Computing, Dublin City University, D09 PX21 Dublin, Ireland
3 Quality and Safety Improvement Directorate, St James’s Hospital Dublin, D08 NHY1 Dublin,
4 Health and Safety Unit, Dublin Fire Brigade, D02 RY99 Dublin, Ireland
5 Beacon Renal, Sandyford Business Park, D18 TH56 Dublin, Ireland
*Corresponding author
Abstract. This paper proposes a framework for developing a trustworthy
artificial intelligence (AI) supported knowledge management system (KMS) by
integrating existing approaches to trustworthy AI, trust in data, and trust in
organisations. We argue that improvement in three core dimensions (data
governance, validation of evidence, and reciprocal obligation to act) will lead to
the development of trust in the three domains of the data, the AI technology, and
the organisation. The framework was informed by a case study implementing the
Access-Risk-Knowledge (ARK) platform for mindful risk governance across
three collaborating healthcare organisations. Subsequently, the framework was
applied within each organisation with the aim of measuring trust to this point and
generating objectives for future ARK platform development. The resulting
discussion of ARK and the framework has implications for the development of
KMSs, the development of trustworthy AI, and the management of risk and
change in complex socio-technical systems.
Keywords: Access-Risk-Knowledge (ARK), socio-technical systems analysis,
risk governance, artificial intelligence, trust.
1 Introduction
Safety regulation increasingly calls for a strategy that goes beyond compliance to being
proactive, predictive, and preventive [1]. Under such a strategy, effective organisational
risk governance relies on evidence-based knowledge, which can be leveraged in support
of actions to mitigate risk. A sophisticated knowledge management system (KMS) is
needed to oversee this mechanism. While many organisations, particularly in high-risk
domains, are generating large amounts of data from diverse sources, the challenge for
risk and safety management is to base operational and strategic decision-making on a
coherent, integrated body of data and evidence (knowledge). Our work develops such
a system through the case study of deploying an artificial-intelligence (AI)-based
software platform that manages risk among three healthcare organisations. There is an
ethical obligation to build trustworthiness into AI technology [2], but this obligation
must be extended to incorporate issues of trustworthiness in complex socio-technical
systems (STS). This paper explores how this extension can be achieved, integrating
strategies for building trust in AI and in organisations in order to develop a framework
for trustworthy AI-supported knowledge management. This suggests two research
(1) What are the components of a trustworthy AI-supported KMS?
(2) How can these components be achieved in the development and deployment
of a software platform for mindful risk governance?
The Access-Risk-Knowledge (ARK) Platform [3] is a software platform that
supports the management of risk and change in complex operational systems. The
platform deploys the Cube framework for socio-technical systems analysis (STSA) [4-
8] along with a risk register, an evidence service, risk mitigation project management
tools, analytics, and reports. Risk assessments can be imported from an existing risk
register or completed within the platform and are then linked to safety projects. These
features enable what we define as mindful governance of risk by leveraging human-
and machine-based knowledge to analyse causal relationships. The result of a
completed ARK project is an evidence-based analysis of a risk mitigation project
throughout the full project management cycle. Projects can also be interlinked in order
to synthesise results or to compare results, evidence, or domains. Results can be
disseminated to the organisation using the customisable report generation feature.
ARK-Virus is a collaborative project between an academic research team from both
the computer science and organisational psychology disciplines, as well as a
Community of Practice (CoP) involving quality and safety staff from a 1000-bed urban
academic teaching hospital, medical staff from a private renal dialysis service, and
management staff from a large urban fire and emergency medical services (EMS)
provider. The aim of the project is to develop the ARK platform via a use case relating
to infection prevention and control (IPC) in each of the three participating
organisations. There are four ARK platform development trials planned; at the time of
writing, the project is between the third and fourth of these. A fuller description of each
trial and the research activities involved is outlined in a previous paper [7]; the focus of
this paper is to develop a framework for trustworthy AI-supported knowledge
management, which spans all four of the trials.
In earlier stages of ARK-Virus, our research focused on issues relating to usability,
but trust has become increasingly important. Discussions with users centred around a
key set of issues: how to make sense of the data, how to do something useful with it,
and how to generate a sound basis for engaging others within the organisation. Trust in
the platform’s ability to deliver this may be a key mechanism for understanding the
relationships between the ARK platform, knowledge, users, and the organisation. In
this paper, we draw upon the literature on trust in organisations, AI technology, and
data, and upon several decades of research on risk in aviation and healthcare, to outline
a framework for the development of a trustworthy KMS that is supported by AI
technologies. As the ARK-virus project continues, we aim to apply this framework so
that trust can be built into future platform development stages.
Our work is situated at the intersection of technology and people, and there is a clear
link between trust in these two domains. Building trustworthy AI involves the full
organisational context of implementation, while building trust in the organisation
similarly requires taking into account the role of technology supported knowledge as
evidence as a rational basis for action. The convergence of knowledge between
technology and people inevitably means that technology-based knowledge is a critical
resource for human decision-making, as it can generate leverage to address complex
problems. Risk and safety management must be based on data and evidence that is
integrated into operational decision making. As trust is core to the management of
safety and the implementation of change, a unified view of trust that bridges risk
management and trust in AI is needed.
Our model of trust incorporates existing theories of organisational trust [9,10],
governance of risk [11], and data governance [12]. Drawing upon several decades of
research, dialogue with collaborators, and the literature, three core dimensions of trust
were identified: data governance, validation of evidence, and reciprocal obligation to
act. By supporting improvements in these three dimensions, trust is improved at the
level of trust in the organisation [10,13], trust in the AI technology [2], and trust in the
data [14]. The framework is outlined in Figure 1.
Fig. 1. Framework for developing a trustworthy AI-supported KMS for risk governance.
The ARK platform instantiates this model to support human-directed decision-making
and implementation as part of an accountable governance framework. Data
governance is at the core of ARK’s services. Validation of evidence is the core activity
of STSA Cube analysis, deploying the flexible schemata of Knowledge Graphs to bring
together diverse data sources to support analysis, decision-making and project
management by quality and safety experts. The reciprocal obligation to act is
engendered by the mindful governance of a risk project from problem state to verified
In this paper, five stages are outlined in the development of trust in such a system.
The trust model is used to analyse and assess the ARK platform’s deployment within
each collaborating organisation. Over the course of the previous ARK-Virus trials, trust
has been developed through a variety of strategies in each organisation. Using the
model of trust as an explanatory concept in this way provides a set of objectives for
future development of the project. This suggests the possibility of a capability maturity
model (CMM) to provide guidance in development of trustworthy governance of
system risk based on verifiable outcomes to demonstrate the effective mitigation of
system risk.
2 A Framework for Trustworthy AI-Supported Knowledge
Trust has been defined in the literature on trustworthy AI as “(1) a set of specific beliefs
dealing with benevolence, competence, integrity, and predictability (trusting beliefs);
(2) the willingness of one party to depend on another in a risky situation (trusting
intention); or (3) the combination of these elements" [15]. The European Union Ethical
Principles for Trustworthy AI [2] outline a set of seven requirements for
trustworthiness; our work supports and extends these principles by integrating
trustworthy AI, trust in data, and trust in organisations.
Mollering offers a model that helps us build an understanding of the problems with
trust relating to our work [9]. Trust is defined as a strategy to cope with the complexity,
uncertainty, and risk in the world at large; the necessity to assume a level of certainty
projected to the future is based on a combination of reason, routine, and reflexivity.
Keymolen applies this model to analyse the relation between trust in other individuals,
trust in an organisation and trust in technology [16]. Ward, through a series of case
studies in an aviation organisation, illustrates the dynamic nature of the factors that
combine to develop trust in an organisational context: understanding and sharing
common goals; open communication of information and knowledge; building
relationships in resolving conflicts in the process of work; together reviewing and
adjusting work-as-imagined based on how work actually happens; it also implies a
belief in the future and establishes the basis for future action [10].
The three components of Mollering’s model provide a powerful framework to
analyse the nature of trust in a data-rich organisational system that is dedicated to
managing risk (achieving certain outcomes) through the deployment of diverse
dedicated roles and relationships. Figure 2 illustrates the connections between that
model and the trust dimensions identified in our work.
Fig. 2. Mollering’s triad and the core trust dimensions.
At a basic level there are three objects of a trusting relationship (trust domains):
Trust in the data itself.
Trust in the processing or transformation of data into usable information and
knowledge (Trustworthy AI).
Trust in the sharing of knowledge with colleagues and building trusting
relationships, leading in turn to trust in the organisational processes that
deploy and use that information and knowledge.
Trustworthy data governance ensures high-quality data and efficient, effective use
of the data, thus leading to more meaningful and trustworthy evidence. Validation of
that evidence in turn links data governance to reciprocal obligation to act by linking
cause to effect. In turn, the obligation to act drives a need for continued collection of
high-quality, trustworthy data. What results is a cyclical pathway driving continuous
improvement of trust in the KMS.
Table 1 illustrates from a theoretical perspective how each dimension (data
governance, validation of evidence and obligation to act) builds trust in each domain
(data, AI technology, organisation), explaining the key mechanism by which
improvements in the core dimensions will result in the development of trust in each
domain. Each column in Table 1 represents the impacts of improvements in that
dimension on each of the three trust domains (i.e., the column labelled ‘Data
Governance’ describes how good data governance improves trust in data, trust in AI
technology, and trust in the organisation). The row labelled ‘AI Technology’ draws
directly on the requirements set forward in the European Union Ethical Principles [2].
Table 1. Impacts of Core Dimensions on Trust Domains
Data Governance
Validation of Evidence
Reciprocal Obligation
Ensures data quality
Generates trust-related
metadata and
trustworthy data as an
outcome of cause and
Action based on data
validates the data based
on action outcomes - if
the outcome works, it
increases the confidence
in the data.
AI Tech.
Supports human
agency and oversight,
privacy and data
diversity and
fairness, and
technical robustness
and safety.
Ensures human agency
and oversight,
transparency, diversity
and fairness, societal
and environmental
wellbeing, and technical
robustness and safety.
Sustains human agency
and oversight,
Leads organisational
decisions to be data-
driven and ensures
data decisions are
aligned with
organisational goals.
Ensures that data-driven
decisions are grounded
in causal relations.
Sustains coherent
response throughout the
project cycle, including
stakeholder feedback.
3 The ARK (Access-Risk-Knowledge) Platform and Trust
ARK (Figure 3) is a software platform that builds and maintains a Resource Description
Framework (RDF)-based unified knowledge graph [17] of risks and projects to link
available datasets on practices, risks, and evidence. This bridges traditional qualitative
risk evidence and quantitative operational or analytics data, which in turn makes large-
scale evidence collection and risk analysis more tractable. Through ARK, human-
oriented quantitative risk information is transformed into structured, machine-readable
data suitable for automated analysis, querying, and reasoning. A privacy by design
approach is taken and data governance principles are followed to ensure support for
evidence linkage, classification, and search. The ARK platform is designed to support
human-directed decision-making and implementation as part of an accountable
governance framework. Data governance, data protection and confidentiality are key
features of the design. Knowledge graphs are a natural way to bring together such
diverse data sources due to their flexible schemata and through use of uplift to common
ontologies, ontology alignment techniques, Natural Language Processing (NLP)-based
knowledge extraction and metadata-based integration, e.g., data catalogues.
Fig. 3. The ARK Platform risk governance services, risk knowledge graph, data governance
services, and foundation services.
ARK supports the development of trust via the key pathway of leveraging data to create
knowledge in support of action by embedding the trust dimensions as described below.
Data governance is at the core of ARK’s services since it supports Khatri and
Brown’s data governance decision domains of data principles, data quality, lifecycle,
metadata, and access [18] to manage projects, evidence, and risk. The Comprehensive
Knowledge Archive Network (CKAN) data catalogue is used to build the ARK
Evidence Service. This enables collecting and tracking of extensive metadata on all
evidence, relating to provenance, verifiability, reputation, and licensing. Within the
Cube knowledge graph, World Wide Web Consortium (W3C) standards for
provenance, classification, identity and access control [19] have been used to capture
this metadata on all data entities within the graph and a flexible policy-driven, General
Data Protection Regulation (GDPR) enabled, context-aware access control system has
been implemented to enable federated data sharing within and between organisations
Validation of evidence is the core activity of STSA, where quality and safety
experts use ARK to perform a structured analysis of risks and safety projects linking
them to a wider range of data sources to support synthesis (with operational data) to
give evidence-based assessment of risk and create new knowledge via that synthesis.
The structured user interface of ARK exposes multiple views of an underlying ontology
that unifies the analysis and enables the combination of traditional qualitative textual
analysis fields with structured data in the form of evidence datasets, risk, and domain
classification taxonomies. A natural language processing component based on the
BERT language model [21] suggests appropriate taxonomy terms and these are
approved by the human expert. Uploading of new evidence (as opposed to evidence
linking) is an access-controlled activity and only users with sufficient permissions can
do this, to facilitate manual validation of evidence prior to upload.
The reciprocal obligation to act is made explicit in numerous parts of the ARK
platform. Firstly, the platform is arranged around the sequence of project stages through
verification of the outcome, which gives information about the outcome as well as how
the entire sequence works. The Cube summary, project analysis, reporting and synthesis
interfaces all contribute to exposing the importance of the problem, the effectiveness of
the solution and the viability of the pathway that underlie the obligation to act. Finally,
the use of knowledge graphs and feature for linking multiple projects in hierarchies or
more general graphs enable a development of a new level of organisational knowledge,
facilitating innovative meta-projects rather than reinforcing what’s already known. This
understanding is leveraged for effective action, responsibility for which can be
distributed explicitly to individuals within the organisation.
4 Stages in the Acquisition of Trust
Analysing progress in the three core dimensions provides an enriched understanding of
the evolution of trust in ARK-Virus. Understanding the dimensions and the interactions
between them develops trust into an explanatory concept, which can be used to inform
a set of development objectives. In Table 2 we have outlined five stages in the
acquisition of trust, from neophyte to multiple organisations. In the upcoming phase of
the ARK-Virus project, the goal is for each organisation to progress up a stage:
Organisation 1 from single projects to multiple; Organisation 2 from neophyte to
intermediate; and Organisation 3 from intermediate to single projects. This table offers
a way of measuring where each organisation is in the trust development process, which
will be useful as a point of comparison in the future and support us in determining the
key issues to be addressed at that point in time.
Table 2. Stages in the Acquisition of Trust
Data governance
Evidence validation
1. Neophyte
Resolve issues of access
and privacy.
interpretation and
evidence gathering.
Initial individual
use. Potential for
2. Intermediate
Assemble and begin
integrating relevant data
Gathers evidence and
performs effective
Engages people in
real projects that
3. Single
Develop knowledge
graphs to generate project-
level knowledge.
Catalogue data source
Diverse evidence
synthesised &
validated as
representing process
& outcome.
Embedded in
processes that
accountable action
and outcome.
4. Multiple
Link data at the level of
multiple projects to
further develop
knowledge graphs and
generate organisation-
level knowledge. Assure
data quality.
Synthesis of
evidence provides a
basis for policy.
Engage strategic &
operational loops
of knowledge
lifecycle across &
5. Multiple
sector level
Fully developed private &
public knowledge space,
routine transformation of
private into public.
Evidence provides a
basis for guidance,
regulation or
Guidance feeds
back into the
evidence base.
5 Application of the Trust Model to a Community of Practice
In this exercise, we applied our model of trust to the ARK-Virus project within each of
the three participating organisations, asking users to reflect on the ways in which trust
had been developed to this point and the next steps for further development. The results
of this exercise in each organisation are outlined in the subsections below.
Several commonalities emerged in terms of needs moving forward. Firstly, it was
noted that many of the more salient issues for the CoP were related to data governance.
For Organisation 1, this was the acquisition of data from different stakeholders within
the organisation; for Organisation 2, data privacy and obtaining formal permissions to
enter information into the platform; for Organisation 3, the resolution of data
complexity and organising data from a large number of different sources. Secondly,
there is a clear need across all three organisations to extend the user base to encompass
the full range of relevant decision-makers. This expansion improves capacity in all
three dimensions, but in particular the reciprocal obligation to act. Thirdly, there is a
pragmatic need to gather and disseminate evidence showing that actions from ARK
projects lead to good outcomes at the organisational level, thus increasing trust in all
three dimensions.
5.1 Organisation 1
Organisation 1 developed a project examining personnel compliance with COVID-19
IPC risk management and control measures. At the onset, the organisational
representatives hoped to collect data measuring personnel compliance in rest areas, as
these were suspected to be a key source of staff-to-staff COVID-19 transmission.
However, this was deemed unfeasible as there was a need to develop trust in the project
among personnel before such data could be collected. Instead, data was drawn from
what was available in terms of occupational health data, guidelines and control
measures over time, impact of limited personnel availability on service provision, and
implicit/explicit knowledge about the linkages between the evidence sources from the
organisation’s ARK-Virus project team. The ARK platform then enabled the project
team to analyse a complex and intractable problem for a full project cycle (from
problem to embedment). The structured approach to STSA helped frame the problem
and identify possible solutions, which were transposed into an implementable
operational solution. The platform was also utilised to effectively communicate and
implement the solution and verify the efficacy of the solution. Further projects utilising
the ARK platform within the organisation have been initiated, indicating the
organisation's trust in the platform.
Data governance: Data Protection (DPA) and Non-Disclosure (NDA) Agreements
guaranteed a level of data protection that was acceptable to the organisation. However,
access to more granular data remained restricted due to concerns about anonymity of
personnel. While there were difficulties in acquiring granular data and evidence, the
process of seeking this evidence for use on the platform resulted in the acquisition of
knowledge from within the organisation which verified the efficacy of the implemented
Validation of evidence: Gathering of evidence was somewhat restricted due to
privacy issues, the organisation's work practises, and the organisation’s clinical
environment. The evidence gathered was done so utilising a top-down/bottom-up
approach, with stakeholders from various departments, including operations, health and
safety, and logistics, gathering, interpreting, and validating the uploaded evidence.
Reciprocal obligation to act: Organisation 1 has a fairly strict hierarchical rank
structure, with a promotional process that means senior managers have fulfilled
operational roles, sometimes alongside personnel they now manage. This structure was
felt to enhance the level of social trust across ranks in the organisation, contributing to
a peer-driven environment where personnel are amenable to the idea of change based
on that trust. Initially, there were three personnel from the organisation who engaged
directly with the platform, from middle and senior management and health and safety.
However, input was also sought from other areas of the organisation, including
operations, resources allocation, health and safety, and senior management. To
strengthen reciprocal obligation to act, there is a need to involve these stakeholders
more formally, in particular by training more personnel as ARK users.
5.2 Organisation 2
Organisation 2 aimed to assess patient compliance with PPE measures upon arrival. Six
months of data on patient compliance were collected by front desk staff, a timeframe
which covered two different sets of PPE requirements. There were, however, significant
issues with obtaining access to the data, with the DPA and NPA taking nearly a full
year to complete. In the meantime, users from the organisation were able to fill out
sample projects on the platform and participate fully in the other aspects of the project
such as the CoP meetings and workshops. In Trial 3, the goal for Organisation 2 is to
move from the first stage of trust to the second. The risk is currently being actively
managed at the local level (clinical frontline), but having overcome data governance
barriers, a thorough analysis of evidence will enable the organisation to strengthen its
management of that risk.
Data governance: Access to data was granted just one week prior to writing of this
paper (the datasets remain within the organisation only, while analysis of the data is
accessible to others within the ARK-Virus project). Trust in data governance as it
relates purely to data has been heightened through the formalisation of data governance
procedures via the DPA and NPA, but there is still much progress to be made in terms
of data governance and trust in the AI technology and the organisation.
Validation of evidence: The organisation is at the stage of moving from data
collection to analysis and use of the data. Moving forward, the organisation is working
to identify variables in the data and complete the STSA component of an ARK project,
which will allow for further exploration and validation of the predictors and/or
outcomes of PPE compliance.
Reciprocal obligation to act: At this stage, operational staff are the primary user
group; an important development will be the engagement of a wider variety of users,
particularly in more strategic or risk management roles.
5.3 Organisation 3
Early on in the project, it became apparent that a key issue for Organisation 3 was the
vast amount of data being produced and reviewed, with no unified structure for tracking
all of the data. Over 100 discrete performance indicators are currently monitored in
relation to the actions taken for the prevention and control of healthcare-associated
infections (PCHCAI), and the processes for capturing, reporting on, collating, and
presenting the data can be fragmented and time consuming. As a result of the
organisation’s experiences completing an ARK project related to environment hygiene
and the wider PCHCAI programme, the organisation conducted a data governance
mapping exercise. PCHCAI metrics were mapped along dimensions of data governance
including the purpose of the metric; type of metric; basis of metric (numerator and
denominator); owner; reporting; tools or platforms used for gathering, analysing and
reporting the data; whether it could be considered an outcome, process, structure or
balancing measure; and the national and international benchmarks and regulatory basis
of the data.
Data governance: Progress was made in terms of data governance processes,
addressing the issue of the large amount of data and how to turn it into a more
manageable data catalogue that provides a clear rationale for management and use.
What remains to be done is to expand and embed the data governance processes so that
subsequent actions and outcomes can be obtained and measured. The fact that the
platform created a strong rationale for compiling and auditing data is an argument in
favour of understanding the entire data system prior to initiating a real-world project;
in other words, to avoid prematurely structuring an evidence trail without first having
agreement on the purpose of each of the metrics.
Validation of evidence: The organisation is moving from the validation and use of
individual data sets to the validation and use of knowledge, which will be undertaken
by quality and safety improvement staff and PCHCAI programme contributors using
the STSA components of the ARK platform.
Reciprocal obligation to act: To this point, work on this ARK trial has been situated
in the core quality and safety improvement team, with some level of engagement via
production of the stakeholder report in the previous trial. The results of this trial will
strengthen this engagement, forming the basis for drawing additional stakeholders into
a collaborative programme and widening the ARK platform user base. Building
interpersonal trust within the local team is the first step to engaging a wider stakeholder
group and building an organisational basis for trust (and subsequently organisational
obligation to act).
5.4 Capability Maturity in Trust in AI and the Organisation
The idea that there are phases in the development of a trustworthy AI-supported KMS
suggests the possibility of a CMM that would provide a framework for verifying
progress through these phases and provide guidance in development and application.
De Bruin, et al. discuss the development of CMMs and provide a relevant example of
a Knowledge Management Capability Assessment metric with progressive stages in the
sharing, managing, and improving of knowledge assets [22]. An example from safety
management in aviation is the Civil Air Navigation Services Organisation (CANSO)
model of excellence in safety management for Air Traffic Control Organisations [23].
For the development of the ARK platform, we need a hybrid combination that spans
between the technology, the AI, and the organisation.
Table 3 outlines two phases in the development of the platform: Trials 1 and 2, and
Trial 3. Trial 1 and 2 measurements were collected in the earlier phase of the project.
Trial 3 trust measurements will be collected in the upcoming phase of the project, as
will measurements on platform usability and effectiveness. The strategic requirements
for achieving advancements in trust are outlined in the middle column, Trial 3 Strategy.
Table 3 represents a synthesis of the first two tables and an initial attempt to define and
measure progress at this point in ARK-Virus towards the development of a trustworthy
AI-supported KMS.
Table 3. Trust-Related Measurements and Development Strategy
Trial 3 Strategy
Trial 3 Measurements
Security (advanced access
control policies to enable
federated sharing)
GDPR compliance
(privacy by design,
compliance reporting,
Privacy-aware data
interlinking mechanisms
Trustworthy data metrics
to measure provenance,
verifiability, reputation,
and licensing [14]
of evidence
Analyse and better
illustrate quality of causal
Validate sequence of
activity and outcome
Meta-analysis of multiple
projects to support
proposal of new guidance
Develop guidance
material based on
Initiate new projects
based on expectation of
outcomes of value
obligation to
Represent different user
roles in platform
Represent relationships
between reports and their
owners in platform
Engage stakeholders
within and outside of CoP
Build set of expert users
and widen user base
Engagement with
implementation of
guidance material
6 Discussion
In order to move the ARK platform along the pathway from development to
implementation to embedment, it is crucial that the technology and the system it
engenders are trusted by the participating user organisations. Operationally, the ARK
platform is for management of risk and change, which involves analysing the issues to
do with causal relationships, outcomes, and changing the outcomes. The key
mechanism for changing outcomes is the leveraging of knowledge as evidence. A better
understanding of this process can help explain the differential success of change
projects, impacting at the level of the organisation, sector, and society.
The ARK-Virus project has been a strong stimulus to organise evidence in the
participating organisations. Although so far that collection has not been highly
sophisticated in terms of AI, and while there has not been the opportunity for in-depth
AI supported analysis, there is confidence that the platform will deliver this in the
future. The organisation of evidence is a necessary first step. In addition, this exercise
showed that the first step is to build trust at the local level; trust is developed in stages,
and overestimating the level of trust already achieved within an organisation should be
avoided. Trust was built locally by enhancing relationships with working colleagues at
the level of the research team, the CoP, and the user groups from each organisation.
Access to data presented key challenges in terms of project progress across the
participating organisations. This highlights a need for updated data governance models
that enable effective action, rather than solely protecting privacy, aligning with the
work of Janssen, et al. [24]. Inter-organisational trust in data governance practices, in
particular with regards to protecting anonymity of personnel, appears to play a role in
securing access to data, though legal agreements are also necessary.
The ARK-Virus project is a work in progress. This exercise enabled us to develop a
structured framework for examining the stages in development of the project and the
ARK platform. Analysing trust has helped us to outline a plan for moving forward in
the project in a way that supports the embedment of the platform in existing risk
management processes within the participating organisations and led to the selection of
key outcome measures relating to the development of trust, constituting the first step in
developing a CMM.
7 Conclusions
In this exercise, we outlined a framework for developing a trustworthy AI-supported
KMS. In the proposed model, three key dimensions (data governance, validation of
evidence, and reciprocal obligation to act) contribute to improved trust in three domains
(organisation, AI technology, and data). There are five stages in the development of
trust, against which organisations can measure their progress. We then applied the
framework to the ARK-Virus project, which deploys a risk management platform in
three participating healthcare organisations. This application resulted in a set of
objectives that, when achieved, will improve trust in each organisation, as well as a
measurement strategy that can be used to track the development of trust. This suggests
the possibility of a CMM to provide guidance in development of trustworthy
governance of system risk based on verifiable outcomes to demonstrate the effective
mitigation of system risk.
Over the course of the previous ARK-Virus trials, trust has been developed through
a variety of strategies in each organisation, including participation in the CoP, active
feedback loops, engagement of key stakeholders, comprehensive data protection
agreements, and building a better understanding of the data. We aim to continue
focusing on trust moving forward by measuring the level of trust and developing trial
objectives that specifically support its development. There is currently a high level of
trust in the platform and its future deployment, particularly in Organisation 1 as
evidenced by their selection of the ARK to support additional projects in the coming
months. However, there is room for improvement as well. The most salient issues
identified were related to data governance, meaning a focus on this area in the coming
months will be key. Core needs also included the expansion of the ARK platform user
base and the production of a follow-up stakeholder report which consolidates the
evidence for beneficial organisational outcomes as a result of ARK projects. These
needs will be addressed in subsequent development trials.
Integration of a technology-based knowledge system has social implications,
meaning that beyond trust in data or technology, the organisational dimensions of trust
must be considered. At the same time, the role of knowledge and evidence is critical
for developing trust in the organisation; it is not merely a question of social
relationships or expectations. There is a need for frameworks guiding the development
of trust in this holistic way. There is also a need to develop guiding principles for AI
implementation that support and extend the European Union principles for ethical AI,
in particular focusing on the organisational dimension having to do with
implementation, action, and outcome. In this exercise, we have contributed to the
resolution of this gap by operationalising Mollering’s triad [9] to outline a framework
for the development of trust in an AI-supported KMS. While our focus has been on a
system that has formal structures for looking at risk and change, any complex STS
would benefit from practical examination of a technology-based KMS in terms of trust.
8 Acknowledgments
This research was conducted with the financial support of Science Foundation Ireland
under Grant Agreement No. 20/COV/8463 at the ADAPT SFI Research Centre at
Dublin City University and Trinity College Dublin. This research was conducted with
the financial support of Science Foundation Ireland at ADAPT, the SFI Research
Centre for AI-Driven Digital Content Technology at DCU [13/RC/2106_P2]. For the
purpose of Open Access, the author has applied CC BY public copyright licence to any
Author Accepted Manuscript version arising from this submission.
1. Hollnagel E., Wears, R.L., and Braithwaite, J. From Safety-I to Safety-II: A
White Paper. The Resilient Health Care Net. University of Southern Denmark,
University of Florida, USA, and Macquarie University, Australia (2015).
2. European Commission High-Level Expert Group on Artificial Intelligence. Ethics
Guidelines for Trustworthy Artificial Intelligence. Brussels, Belgium (2019).
3. Crotti Junior, A., Basereh, M., Abgaz, Y., Liang, J., Duda, N., McDonald, N., and
Brennan, R. The ARK platform: enabling risk management through semantic web
technologies. In Proceedings of the 11th International Conference on Biomedical
Ontologies. Bolzano, Italy. (2020, 17 Sept).
4. Ward, M., McDonald, N., Morrison, R., Gaynor, D., and Nugent, T. A
performance improvement case study in aircraft maintenance and its implications
for hazard identification. Ergonomics, 53(2), pp. 247-267 (2010).
5. Ulfvengren, P. and Corrigan, S. Development and Implementation of a Safety
Management System in a Lean Airline. Cognition, Technology, and Work, 17,
pp. 219236 (2015).
6. Corrigan, S., Kay, A., O'Byrne, K., Slattery, D., Sheehan, S., McDonald, N.,
Smyth, D., Mealy, K., and Cromie, S.A. Socio-Technical Exploration for
Reducing & Mitigating the Risk of Retained Foreign Objects. International
Journal of Environmental Research and Public Health, 15(4), 714 (2018).
7. McDonald, N., McKenna, L., Vining, R., Doyle, B., Liang, J., Ward, M.E.,
Ulfvengren, P., Geary, U., Guilfoyle, J., Shuhaiber, A., Hernandez, J., Fogarty,
M., Healy, U., Tallon, C., and Brennan, R. Evaluation of an Access-Risk-
Knowledge (ARK) Platform for Governance of Risk and Change in Complex
Socio-Technical Systems. International Journal of Environmental Research and
Public Health, 18(23), 12572 (2021).
8. Geary, U., Ward, M.E., Callan, V., McDonald, N., and Corrigan, S. A socio-
technical systems analysis of the application of RFID-enabled technology to the
transport of precious laboratory samples in a large acute teaching hospital.
Applied Ergonomics, 102, 103759 (2022).
9. Möllering, G. Trust: Reason, Routine, Reflexivity. Elsevier, Amsterdam,
Netherlands (2006).
10. Ward, M. Contributions to human factors from three case studies in aircraft
maintenance [Doctoral thesis]. Trinity College Dublin (2006).
11. McDonald, N., Callari, T.C., Baranzini, D., and Mattei, F. A Mindful Governance
model for ultra-safe organisations. Safety Science, 120, pp. 753-763 (2019).
12. Brous, P. and Janssen, M. Trusted Decision-Making: Data Governance for
Creating Trust in Data Science Decision Outcomes. Administrative Sciences,
10(4), 81 (2020).
13. Blomqvist, K. and Stahle, P. Building Organizational Trust. In: 16th ANNUAL
IMP-CONFERENCE, Bath, U.K (2000).
14. Zaveri, A., Rula, A., Maurino, A., Pietrobon, R., Lehman, J., and Auer, S. Quality
Assessment for Linked Data: A Survey. Semantic Web, 7(1), 63-93 (2016).
15. Siau, K., Wang, W. Building Trust in Artificial Intelligence, Machine Learning,
and Robotics. Cutter Business Technology Journal (31), pp. 4753 (2018).
16. Keymolen, E. Trust on the line: a philosophical exploration of trust in the
networked era [Doctoral thesis]. Erasmus University Rotterdam (2016).
17. World Wide Web Consortium. RDF 1.1 Concepts and Abstract Syntax. In
Cyganiak, R., Woods, D., Lanthaler, M. (Eds.) W3C Recommendation,, last accessed 2022/4/3.
18. Khatri, V., and Brown, C.V. Designing Data Governance. Communications of the
ACM, 53(1), pp. 148-152 (2010).
19. World Wide Web Consortium (W3C) Standards,,
last accessed 2022/4/27.
20. Hernandez, J., McKenna, L., and Brennan, R. TKID: A Trusted Integrated
Knowledge Dataspace for Sensitive Healthcare Data Sharing. In: IEEE 45th
CONFERENCE, 1855-1860. (2021).
21. Jacon Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. Bert:
Pretraining of deep bidirectional transformers for language understanding,, last accessed 2022/4/27.
22. de Bruin, T., Rosemann, M., Freeze, R., Kulkarni, U., and Carey, W.
Understanding the Main Phases of Developing a Maturity Assessment Model.
Sydney, Australia. (2005, 29 Nov-2 Dec).
23. Civil Air Navigation Services Organisation. CANSO Standard of Excellence in
Safety Management Systems,
excellence-in-safety-management-systems/, last accessed 2022/4/3.
24. Janssen, M., Brous, P., Estevez, E., Barbosa, L.D., and Janowski, T. Data
governance: Organizing data for trustworthy Artificial Intelligence. Government
Information Quarterly, 37(3), 101493, (2020).
ResearchGate has not been able to resolve any citations for this publication.
Full-text available
Three key challenges to a whole-system approach to process improvement in health systems are the complexity of socio-technical activity, the capacity to change purposefully, and the consequent capacity to proactively manage and govern the system. The literature on healthcare improvement demonstrates the persistence of these problems. In this project, the Access-Risk-Knowledge (ARK) Platform, which supports the implementation of improvement projects, was deployed across three healthcare organisations to address risk management for the prevention and control of healthcare-associated infections (HCAIs). In each organisation, quality and safety experts initiated an ARK project and participated in a follow-up survey and focus group. The platform was then evaluated against a set of fifteen needs related to complex system transformation. While the results highlighted concerns about the platform’s usability, feedback was generally positive regarding its effectiveness and potential value in supporting HCAI risk management. The ARK Platform addresses the majority of identified needs for system transformation; other needs were validated in the trial or are undergoing development. This trial provided a starting point for a knowledge-based solution to enhance organisational governance and develop shared knowledge through a Community of Practice that will contribute to sustaining and generalising that change.
Full-text available
Organizations are increasingly introducing data science initiatives to support decision-making. However, the decision outcomes of data science initiatives are not always used or adopted by decision-makers, often due to uncertainty about the quality of data input. It is, therefore, not surprising that organizations are increasingly turning to data governance as a means to improve the acceptance of data science decision outcomes. In this paper, propositions will be developed to understand the role of data governance in creating trust in data science decision outcomes. Two explanatory case studies in the asset management domain are analyzed to derive boundary conditions. The first case study is a data science project designed to improve the efficiency of road management through predictive maintenance, and the second case study is a data science project designed to detect fraudulent usage of electricity in medium and low voltage electrical grids without infringing privacy regulations. The duality of technology is used as our theoretical lens to understand the interactions between the organization, decision-makers, and technology. The results show that data science decision outcomes are more likely to be accepted if the organization has an established data governance capability. Data governance is also needed to ensure that organizational conditions of data science are met, and that incurred organizational changes are managed efficiently. These results imply that a mature data governance capability is required before sufficient trust can be placed in data science decision outcomes for decision-making.
Full-text available
A Retained Foreign Object (RFO) is a fairly infrequent but serious adverse event. An accurate rate of RFOs is difficult to establish due to underreporting but it has been estimated that incidences range between 1/1000 and 1/19,000 procedures. The cost of a RFO incident may be substantial and three-fold: (i) the cost to the patient of physical and/or psychological harm; (ii) the reputational cost to an institution and/or healthcare provider; and (iii) the financial cost to the taxpayer in the event of a legal claim. This Health Research Board-funded project aims to analyse and understand the problem of RFOs in surgical and maternity settings in Ireland and develop hospital-specific foreign object management processes and implementation roadmaps. This project will deploy an integrated evidence-based assessment methodology for social-technical modelling (Supply, Context, Organising, Process & Effects/ SCOPE Analysis Cube) and bow tie methodologies that focuses on managing the risks in effectively implementing and sustaining change. It comprises a multi-phase research approach that involves active and ongoing collaboration with clinical and other healthcare staff through each phase of the research. The specific objective of this paper is to present the methodological approach and outline the potential to produce generalisable results which could be applied to other health-related issues.
Full-text available
In this article, we look at trust in artificial intelligence, machine learning (ML), and robotics. We first review the concept of trust in AI and examine how trust in AI may be different from trust in other technologies. We then discuss the differences between interpersonal trust and trust in technology and suggest factors that are crucial in building initial trust and developing continuous trust in artificial intelligence.
The scale and pace of improvement in patient safety in healthcare has been unacceptably slow. A paucity of research into the application of systems-thinking concepts and a failure to appreciate health systems complexity are cited as barriers to sustainable health systems improvement. This study reports on a socio-technical systems analysis, called the CUBE, of the characteristics of a large acute teaching hospital's system for the transport of precious specimens, a system enabled by radio-frequency identifier device tracking technology. The CUBE proved itself to be an effective analytic tool. The analysis provided a constructive framework to link diverse data and documentation; explicitly inviting consideration of the roles and understandings of different stakeholders; as well as broader cultural factors that could influence current or future activity. The analysis also supported recommendations to improve and extend operations. This study supports the argument for systems understanding and systems thinking being at the core of new approaches to patient safety.
The rise of Big, Open and Linked Data (BOLD) enables Big Data Algorithmic Systems (BDAS) which are often based on machine learning, neural networks and other forms of Artificial Intelligence (AI). As such systems are increasingly requested to make decisions that are consequential to individuals, communities and society at large, their failures cannot be tolerated, and they are subject to stringent regulatory and ethical requirements. However, they all rely on data which is not only big, open and linked but varied, dynamic and streamed at high speeds in real-time. Managing such data is challenging. To overcome such challenges and utilize opportunities for BDAS, organizations are increasingly developing advanced data governance capabilities. This paper reviews challenges and approaches to data governance for such systems, and proposes a framework for data governance for trustworthy BDAS. The framework promotes the stewardship of data, processes and algorithms, the controlled opening of data and algorithms to enable external scrutiny, trusted information sharing within and between organizations, risk-based governance, system-level controls, and data control through shared ownership and self-sovereign identities. The framework is based on 13 design principles and is proposed incrementally, for a single organization and multiple networked organizations.
Mindful organising is a key integrating concept in resolving the organisational accident. Mindful organising is both the unique source of critical information about the normal operation, as well as the key recipient of intelligence about the operation, ensuring that operational actions are always informed by the most current, relevant information about potential risks no matter how remote. However, the mindful organising construct has never been operationalised as a practical and effective approach for complex ultra-safe systems. Within the Future Sky Safety programme the construct has been reworked to reinforce the idea that mindful organising is more than just a state of mind. It is about the gathering and flow of information to ensure awareness and appropriate action, both at the operational level and amongst middle management in ensuring improvements are effectively implemented. A novel model has been advanced which provides an organisational context for its implementation, based on the behavioural-economic principle that being well informed about an issue, having an effective and practical solution and being accountable, creates a compelling obligation to act in an appropriate manner. It is suggested how the operationalisation of this model could be supported through a set of generic prototype software applications. The potential applicability of this approach covers not only all sectors of aviation, but also all industries that carry a significant operational risk, including other transport modes, health and social care, emergency services and financial services.
All stakeholders within the aviation sector are currently facing immense system changes due to implications from the future Single European Sky concept and the new requirement for a Safety Management System (SMS). At the same time, the airline industry is under great financial pressure. So, there are no margins for failure to adapt and comply with these system changes. Yet, the reported success rate of organisational change in industry is low. The MAnaging System Change in Aviation (MASCA—EU FP7 2010-2013) project addressed these industrial needs. MASCA was driven by industrial-based case studies. The change case reported on in this paper is an ongoing development and implementation of an SMS in a major European Airline. The overall objective in this industry case was to develop a SMS that will demonstrate safety performance to comply with new regulations. A new approach for human factors and safety was developed in earlier research. As part of this a system and process analysis, a concept called the System Change and Operations Evaluation (SCOPE) model was developed. The approach taken in this case was to apply the SCOPE to enhance core functionalities and further develop the airline SMS. Taking an action research approach, researchers worked closely with the airline’s safety department to support their development of the SMS and to study that system change. The MASCA research objective was to develop a theory for change which included a methodology to evaluate change to support implementation. This case contributed to the multi-case study in MASCA in which a framework for evaluating change emerged, called the Structured Enquiry (SE). The SE was applied to provide complementary recommendations to the proposed enhanced SMS. The SE provided support to the fact that, even if all the pieces are in place required for a compliant SMS, many other essential areas need to be addressed to make all the pieces work together, such as information and knowledge cycles and social relations building teams and trust. Results show benefits of combining SCOPE and SE in system change in aviation in order to encompass identified essential components for safety performance and increasing the chances for a successful change.
Introduction Organizations are becoming increasingly serious about the notion of "data as an asset" as they face increasing pressure for reporting a "single version of the truth." In a 2006 survey of 359 North American organizations that had deployed business intelligence and analytic systems, a program for the governance of data was reported to be one of the five success "practices" for deriving business value from data assets. In light of the opportunities to leverage data assets as well ensure legislative compliance to mandates such as the Sarbanes-Oxley (SOX) Act and Basel II, data governance has also recently been given significant prominence in practitioners' conferences, such as TDWI (The Data Warehousing Institute) World Conference and DAMA (Data Management Association) International Symposium. The objective of this article is to provide an overall framework for data governance that can be used by researchers to focus on important data governance issues, and by practitioners to develop an effective data governance approach, strategy and design. Designing data governance requires stepping back from day-to-day decision making and focusing on identifying the fundamental decisions that need to be made and who should be making them. Based on Weill and Ross, we also differentiate between governance and management as follows: • Governance refers to what decisions must be made to ensure effective management and use of IT ( decision domains ) and who makes the decisions ( locus of accountability for decision-making ). • Management involves making and implementing decisions. For example, governance includes establishing who in the organization holds decision rights for determining standards for data quality. Management involves determining the actual metrics employed for data quality. Here, we focus on the former. Corporate governance has been defined as a set of relationships between a company's management, its board, its shareholders and other stakeholders that provide a structure for determining organizational objectives and monitoring performance, thereby ensuring that corporate objectives are attained. Considering the synergy between macroeconomic and structural policies, corporate governance is a key element in not only improving economic efficiency and growth, but also enhancing corporate confidence. A framework for linking corporate and IT governance (see Figure 1) has been proposed by Weill and Ross. Unlike these authors, however, we differentiate between IT assets and information assets: IT assets refers to technologies (computers, communication and databases) that help support the automation of well-defined tasks, while information assets (or data) are defined as facts having value or potential value that are documented. Note that in the context of this article, we do not differentiate between data and information. Next, we use the Weill and Ross framework for IT governance as a starting point for our own framework for data governance. We then propose a set of five data decision domains, why they are important, and guidelines for what governance is needed for each decision domain. By operationalizing the locus of accountability of decision making (the "who") for each decision domain, we create a data governance matrix, which can be used by practitioners to design their data governance. The insights presented here have been informed by field research, and address an area that is of growing interest to the information systems (IS) research and practice community.