Content uploaded by Rob Brennan
Author content
All content in this area was uploaded by Rob Brennan on Jan 22, 2025
Content may be subject to copyright.
Accountable Risk Management in Healthcare during the COVID-19 Pandemic;
the Role of STSA and AI
Nick McDonalda, Marie E. Wardb,c, Lucy McKennad, Rebecca Vininga, Julio Hernandezd, Brian Doylea, Una
Gearyb, John Guilfoylee, Arwa Shuhaiberf, Rob Brennang
a – School of Psychology Trinity College Dublin Dublin, Ireland, nmcdonld@tcd.ie
b – Quality and Safety Improvement Directorate, St James’s Hospital
c – Centre for Innovative Human Systems, Trinity College, The University of Dublin
d – ADAPT Centre Dublin City University Dublin, Ireland
e – Health and Safety Unit, Dublin Fire Brigade, D02 RY99 Dublin, Ireland
f – Beacon Renal, Sandyford Business Park, D18 TH56 Dublin, Ireland
g – ADAPT Centre, University College Dublin, Ireland
ABSTRACT
Effective governance necessitates going beyond compliance with rules, regulations and procedures;
particularly as adverse events are generally the result of a combination of human, organisational,
technological, and economic factors. This study explores the use of STS analysis (STSA) in an Artificial
Intelligence platform called Access-Risk-Knowledge to go beyond established accountability frameworks
by facilitating the linking of evidence, outcomes and accountability. The aim of this study is to extend and
deploy the ARK Platform to support mindful risk governance of infection prevention and control (IPC) for
healthcare organisations during the COVID-19 pandemic. The platform was deployed across three
healthcare organisations; fire and emergency medical services, outpatient dialysis unit and a large acute
hospital. Three organisationally-based projects were compiled into a synthesis project. A set of guidance
principles for a pandemic preparedness strategy were proposed. A Community of Practice enabled the
successful deployment of ARK, including intense interdisciplinary collaboration and facilitating
practitioner-researchers in the implementing organisations. Data governance methods and tools
supported a whole organisation and multi-organisation approach. This first full implementation trial of
the ARK platform deploying a dedicated STSA within a semantically structured AI framework
demonstrates accountable risk management that addresses the complex antecedents of risk, links to
evidence, and has the potential for managing the full cycle of risk mitigation and improvement, in the
context of a multi-project strategic risk profile.
KEYWORDS: STSA, AI, Infection Prevention Control, COVID-19, Accountable Risk Management
BACKGROUND
Organisations want to be able to manage the risks that they face in a manner that is coherent,
comprehensive and effective. Excellence in operational risk management means going beyond reacting
to specific incidents or regulatory compliance; it implies being able to muster the evidence to support the
transformation of system performance to meet strategic goals in a proactive and generative way. This
involves meeting a range of challenges:
• Harnessing data to generate evidence
• Generating system understanding
• Building joined-up governance
• Developing the capability to change
• Developing networks of support, learning and the exchange of knowledge.
The ARK-Virus Project
The ARK-Virus project was concerned with developing a socio-technically informed methodology for the
management of the risks associated with infection prevention and control (IPC) in the context of the Covid
pandemic. The Access-Risk-Knowledge (ARK) software platform was developed around a socio-technical
methodology for analysing complex risks and mitigating these through a managed process of change. The
basic theoretical concept behind the ARK platform is the Mindful Risk Governance model of McDonald et
al. (2019). The platform was deployed within three collaborating healthcare organisations. The platform
links a structured socio-technical system analysis (STSA) to relevant data and documentary evidence to
support the assessment of risk and the management of that risk through a sequence of project stages.
Users and stakeholders are supported by report generation, visualisations and flexible navigation of the
linked risks, projects, analyses and evidence.
The deployment of the ARK platform is based on best practice in data governance through developing
extensive metadata and an evidence catalogue that includes datasets and publications. The platform uses
privacy by design (data protection in data processing incorporated in the design of the technology) for
controlled data sharing between organisations. A Community of Practice (CoP) has been formed between
developers and users of the platform. The CoP facilitates learning and collaboration by connecting
platform users, primarily risk experts, from different healthcare organisations. The ARK platform leverages
Semantic Web technologies (Gutierrez & Sequeda, 2021), with WWWConsortium (W3C) standards, and
security and privacy information standards - ISO 27001 (ISO, 2013), to model, classify, integrate, and
secure risk and operational data.
Since May 2021, ARK has supported IPC risk management projects in a large acute hospital (O1), a fire and
emergency medical services organisation (O2), and an outpatient dialysis unit (O3). Each organisation
used ARK to carry out an IPC-related case study. O1 focussed on improvement of data governance related
to the prevention and control of healthcare-associated infections (PCHCAI). O2 and O3 focussed on issues
relating to prevention of COVID-19 infection among staff. Throughout the three trials, participants used
the platform and gave feedback to direct its improvement (see McDonald et al., 2021). This paper focuses
on Trial 3 (January-April 2022). The project generated a Stakeholder Report (ARK-virus, 2022) which
supported a wider consultation with a diverse set of stakeholders, including both regulatory and
technology focussed agencies.
METHODOLOGY
ARK provides a risk management platform that guides users in carrying out a risk mitigation project from
identification of the risk (typically via risk assessment) to verification of the mitigation project’s outcomes.
Each ARK project focuses on one specific risk and is organised in five stages: Problem, Solution, Plan &
Prepare, Implement, and Verify & Embed. At each project stage, analysis is done using these features
(Figure 1):
• An assessment of the risk and the value (potential loss and gain) of the project.
• STSA using the Cube framework, which guides users in managing organisational change by asking
them a series of questions. The Cube links goals, actions, and outcomes to operational processes,
developing awareness of multiple dimensions of the system through engaging with it (see
McDonald et al., 2021; Ward et al., 2022; Geary et al., 2022 for examples of the Cube in use).
• A Context-Mechanism-Outcome analysis (Pawson and Tilley, 1997), which synthesises the
findings of the Cube under three headings: Context (the general conditions which define the
relevance of this particular intervention), Mechanism (the modes of leverage that can act to
sustain or transform the situation) and Outcome (the generated products of that activity or
intervention).
• Evidence catalogue containing infection rates, IPC protocols, PPE compliance rates, etc., which is
linked to specific parts of the analysis.
ARK projects assist risk experts in evaluating risks and managing mitigation projects through this high-
level evidence-based analysis. By integrating all available evidence and making it available at the point of
decision-making, it is possible to prioritise key projects and adopt a more proactive approach to risk
management.
Semantic features of the ARK platform, including a core unified knowledge graph, ontologies with
suggestion capability, and similarity comparisons between projects, enable the high-level synthesis of
multiple ARK projects. Learnings from three organisational-level projects outlined in this chapter were
integrated into a synthesis project, the results of which are outlined below. The project was used to create
a shared understanding of IPC practice in emergencies and to develop a set of evidence-based
recommendations for practice.
Figure 1: ARK Risk Mitigation Project Phases with Linked Risks, Evidence and Analysis
RESULTS
This section describes the synthesis project in stages (Problem, Solution, Plan & Prepare, Implement, and
Verify & Embed) and summarises the resulting IPC guidance.
Problem
Trial 3 dealt with issues of platform usability and aimed to apply the IPC learnings from previous trials. Key
challenges were:
• Having too much data or a complete lack thereof; for example, O1 tracks over 110 PCHCAI metrics,
while in O2, occupational health data is scarce and difficult to access.
• The lack of a clear and consistent relationship between data linked to IPC risk and follow up
measures aimed at risk mitigation.
• There is a need for better documentation of organisational roles and structure and incorporation
of that documentation into the ARK platform.
• While there was good communication within the Community of Practice, communication with
non-expert staff within the organisations was limited, the ARK platform was perceived to be
complex and lacked interfaces for effective engagement with non-experts..
• Users had a limited amount of protected time for engaging with the ARK platform and because
the STSA approach to risk management is time-intensive, the level of engagement in some
organisations was limited, particularly in O3.
Solution Stage
The CoP identified a three-part solution to the problem outlined above: (1) Further develop the ARK
platform and deploy the next version, (2) continue developing the working relationships in the CoP, and
(3) apply ARK to an IPC case study in each organisation. The three case studies selected were:
• O1: Improve monitoring, oversight, and continuous improvement of PCHCAI data through
improved data governance.
• O2: Improve staff PPE compliance in order to reduce COVID-19 infection risk for staff.
• O3: Track PPE compliance and identify key areas of COVID-19 IPC risk for staff and patient.
Plan and Prepare
A new version of the ARK platform was developed (version 1.5) (McKenna et al., 2021) and the CoP worked
together to develop a plan for deployment. Feedback channels and individualised support were made
available to user organisations in order to address emerging issues and bugs. In the plan stage, the
evidence linking feature was highly emphasised and organisations focussed heavily on increasing the
amount of evidence uploaded. At this stage, O1 and O2 identified key IPC risk mechanisms:
• O1: Lack of unified data governance means that although a large amount of data is collected, it is
generally not deployed in a way that can improve staff decision-making.
• O2: Staff need access to social support (e.g., break rooms) which can conflict with their
responsibility to comply with PPE protocols. When infections occurred, they had a high impact on
the organisation because numbers of staff-staff exposures were high and because there were pre-
existing staffing issues.
Implement
Version 1.5 of ARK was deployed in each of the three case studies. It was deployed fully in O1 and O2 and
deployed partially in O3.
• O1: Created a new global view of PCHCAI monitoring through system-wide data lineage mapping
of PCHCAI measures. This map will be very useful in directing future analysis of PCHCAI clincal
effectiveness.
• O2: Combined explicit knowledge of staff COVID-19 infection and exposure rates with implicit
knowledge of impacts on service delivery. This was used to generate a better understanding
of the relationship between the two areas and identify geographic and temporal areas of
high COVID-19 infection risk.
• O3: Gathered six months of data on PPE compliance and used this to better understand the
problem/solution space with regards to COVID-19 prevention.
The three case studies were then combined to generate a set of recommendations for IPC practice in
emergencies (see Discussion).
Verify and Embed
Users of the ARK platform and internal stakeholders from collaborating organisations were asked for their
feedback on the project and the IPC practice document in order to assess the quality of implementation
against the initial plan. At the same time, the general usability of the platform was assessed using the
Simple Usability Scale (SUS) (Brooke, 1996) The results of this exercise are as follows:
• For the ARK platform to be adopted, it has to become part of the methods and processes an
organisation uses to manage its risks. O1 has been developing an Enterprise Risk Management
framework; the organisations risk management structure and data streams for PCHCAI have been
addressed using ARK and a new data-led PCHCAI project has been developed. This has enabled
the wide range of existing data sources to be catalogued, better understood, and selected data
streams integrated and analysed to support a proactive assessment of system risk. This improved
data management is enabled by the evidence features on the ARK platform and in turn facilitates
a more effective data governance framework.
• O2 is using ARK to support the development and implementation of an entirely new
communications infrastructure.
• Progress has been made towards a data management framework that allows for proactive risk
management by linking different data sets in order to support decision-making. ARK’s evidence
linking feature was especially helpful for this. However, a higher level of data governance maturity
is still needed.
• There is a training need for advanced systemic risk management in general and for use of the ARK
platform in particular. A new short course in Systemic Risk Management has been developed and
delivered, complementing an existing Masters program in managing risk and system change.
• ARK’s SUS score has progressively improved but remains below average. Even domain experts
find it difficult to use. While, hitherto, development has prioritised core functionality over user
experience, these results support further interface developments combined with increased
project data on the platform that will enable usability to be progressively addressed.
DISCUSSION
The findings of the ARK-Virus project were integrated with the academic literature and input from the CoP
and used to iteratively develop a set of recommendations for practice (Table 1) (ARK-Virus, 2022). The
recommendations provide a good overview of the three organisations’ experiences with IPC risk
management during the COVID-19 pandemic, as well as their hopes for the future. It is hoped that as more
case studies are completed, the recommendations will feed into more substantial guidelines on effective
IPC risk management in emergencies.
Table 1: Key Findings - Recommendations for IPC Implementation in Emergencies (source: ARK-
virus Stakeholder Report)
1. Core operational processes: adequate personnel and physical resources at all times are critical
to maintaining service delivery and allowing for resilience during a crisis.
2. Performance standards: evidence-driven standards should be in place ahead of a crisis so that
organisations can monitor their IPC performance.
3. Quality and flow of information: mature data governance programmes should be in place so that
in an emergency, data can be rapidly found and shared; communication channels need to be in
place to transfer information and knowledge to the point of decision-making.
4. Situational awareness and informed decision-making: implicit and explicit knowledge must be
leveraged to create a collective understanding of the situation; access to data is important, but
equally important is knowledge about that data and its provenance, quality, and intended use.
5. Responsive risk governance infrastructure: a risk governance framework (such as ERM) that links
clinical and operational data should be in place. The infrastructure must be flexible and
responsive so that measures can be escalated or de-escalated.
6. Quality and consistency of task performance: in order to monitor and validate implementation
of control measures, data are needed about the outcomes of interest (i.e., transmission rates).
Feedback loops must be in place that generate information back into the system for continuous
performance improvement.
7. Embedment within behavioural norms: maintaining strict control measures may be difficult,
particularly in stages when the emergency is perceived as less severe; transparent and open
communication foster trust that the measures are necessary, and enhanced supervision may be
needed to reinforce them. A stronger culture of vigilance contributes to maintenance of control
measures.
8. Quality of collaboration, training, and leadership: more intensive collaboration and increased
social support within the organisation is needed during an emergency, particularly as staff roles
may change. Personnel need to be trained in their new roles ahead of a crisis.
9. Trust and transparency: a high level of trust across the organisational hierarchy is needed,
engendered by a trust that the data is of high quality and that organisational decisions are made
based on an understanding of that high-quality data.
10. Shared understanding: a comprehensive system of communication and reciprocal feedback is
critical to generate a shared understanding of a collective response to risk.
CONCLUSIONS
This paper described the first ARK Platform operational trials in healthcare settings as part of the ARK-
Virus project. This novel embedding of STSA in clinical risk management is a new departure for healthcare
organisations. These trials have resulted in several significant outputs: (1) a new version of the ARK
platform with enhanced features, and (2) a new machine-readable and reusable ARK knowledge base that
describes the healthcare setting (IPC), ontologies, and taxonomies for risk mitigation projects. All aspects
of the knowledge base are linked to evidence. We observe that the ARK Platform is becoming embedded
in the risk management systems of two of the ARK-Virus CoP members. We have agreed and initiated
plans for deeper adoption. The potential value of ARK has been illustrated but further progress of the trial
risk projects is required to establish the value of ARK in governing core strategic risks in the organisations.
ARK was received favourably by each of the participating organisations, who had overwhelmingly positive
feedback on its potential for improving IPC risk governance. Participants recognised the benefits and
further potential of intra-organisational collaboration experienced by applying the mindful governance
methodology. In addition, the CoP was highly valued for providing a venue for the transfer of knowledge
relating to IPC, AI, knowledge management, and risk governance; in fact, it was considered to be critical
for successful ARK deployment. There was an intense interdisciplinary collaboration between ICT/AI and
STSA academics, as well as between the academic team and risk experts working in healthcare;
researchers who also worked within the organisation were instrumental in realising the project. There are
still some issues with usability of ARK, based upon feedback from users and the SUS score, which will be
improved through further development of the user interface and of specialised interfaces for users in
different operational roles (i.e., non-expert users).
Data governance tools, such as data catalogues, data dictionaries, metadata, and data lineage models,
were used to improve data integration, sharing, trust, and quality both within and between organisations
(Vining et al., 2022). Mature data governance systems are important for developing intra- and inter-
organisational patient safety systems built on a diverse range of data and evidence sources.
ARK-Virus is being extended beyond the original project lifecycle:
• Higher Education Authority (HEA)-funded extension on cross-sectoral analysis of ARK in aviation
as well as healthcare domains
• O1 will perform a behavioural analysis of clinical time series data on PCHCAI with Science
Foundation Ireland (SFI) and ADAPT Centre
• O2 will use ARK to conduct a risk project on a national-level emergency communications
infrastructure change
The project members are interested in engaging with new partners for further deployments.
Set within the context of the COVID-19 pandemic, this study represents the first full implementation trial
of the ARK platform deploying a dedicated STSA within a semantically rich AI framework. It demonstrates
accountable risk management in healthcare that addresses the complex antecedents of risk, links to
evidence and has the potential for managing the full cycle of risk mitigation and improvement, in the
context of a multi-project strategic risk profile.
ACKNOWLEDGEMENTS
This research was conducted with the financial support of SFI under Grant Agreement No. 20/COV/8463
at the ADAPT SFI Research Centre at Dublin City University and Trinity College Dublin. The ADAPT SFI
Centre for Digital Content Technology is funded by Science Foundation Ireland through the SFI Research
Centres Programme and is co-funded under the European Regional Development Fund (ERDF) through
Grant \#13/RC/2106\_P2. The authors would like to acknowledge and thank all of the staff who
participated in this research across the three organisations in the Community of Practice.
REFERENCES
ARK Virus: Access Risk Knowledge Platform for Mindful Governance of Infection Prevention and Control Risk.
(2022). Final Stakeholder Report. Available at
https://openark.adaptcentre.ie/ARK_Websites/Stakeholder_Report_Trial3.pdf
Geary, U., Ward, M. E., Callan, V., McDonald, N., & Corrigan, S. (2022). A socio-technical systems analysis of the
application of RFID-enabled technology to the transport of precious laboratory samples in a large acute
teaching hospital. Applied ergonomics, 102, 103759. https://doi.org/10.1016/j.apergo.2022.103759
Gutierrez, C & Sequeda J.F. (2021). Knowledge graphs. Communications of the ACM, 64 (3), 96-104.
https://dl.acm.org/doi/10.1145/3418294
ISO (2013) Information technology — Security techniques — Information security management systems —
Requirements ISO/IEC 27001:2013
McDonald,N., Callari,T.C., Baranzini,D., and Mattei,F. (2019). A Mindful Governance model for ultra-safe
organisations. Safety Science, 120, pp. 753-763.
McDonald, N., McKenna, L., Vining, R., Doyle, B., Liang, J., Ward, M.E., Ulfvengren, P., Geary, U., Guilfoyle, J.,
Shuhaiber, A., Hernandez, J., Fogarty, M., Healy, U., Tallon, C., Brennan, R. (2021). Evaluation of an
Access-Risk Knowledge (ARK) platform for governance of risk and change in complex socio-technical
systems. International Journal of Environmental Research and Public Health, 18(23), 12572.
https://doi.org/10.3390/ijerph182312572
McKenna, L., Liang, J., Duda, N., McDonald, N., and Brennan, R. (2021). ARK-Virus: An ARK Platform Extension for
Mindful Risk Governance of Personal Protective Equipment Use in Healthcare. In Companion
Proceedings of the Web Conference 2021 (WWW '21). Association for Computing Machinery, New York,
NY, USA, 698–700. DOI: https://doi.org/10.1145/3442442.3458609
Pawson, R., & Tilley, N. (1997). An introduction to scientific realist evaluation. In E. Chelimsky & W. R. Shadish
(Eds.), Evaluation for the 21st century: A handbook (pp. 405–418). London, UK: SAGE Publishing.
Brooke, J. (1996). SUS: A ‘quick and dirty’ usability scale. In Jordan, P.W., Thomas, B., McClelland, I.L.,
Weerdmeester, B., (Eds) Usability Evaluation in Industry, 1st ed (pp. 189-194). Taylor and Francis:
London, UK.
Vining, R., McDonald, N., McKenna, L., Ward, M.E., Doyle, B., Liang, J., Hernandez, J., Guilfoyle. J., Shuhaiber, A.,
Geary U., Fogarty, M., Brennan, R. (2022). Developing a Framework for Trustworthy AI-Supported
Knowledge Management in the Governance of Risk and Change. In: 24th International Conference on
Human-Computer Interaction (HCI'22), 26 Jun - 1 Jul 2022, Virtual. (In Press)
Ward, M. E., Daly, A., McNamara, M., Garvey, S., & Teeling, S. P. (2022). A Case Study of a Whole System
Approach to Improvement in an Acute Hospital Setting. International Journal of Environmental Research
and Public Health, 19(3), 1246. https://doi.org/10.3390/ijerph19031246
WorldWideWebConsortium(W3C)Standards,https://www.w3.org/standards/, last accessed 2022/4/27.
