Available via license: CC BY 4.0
Content may be subject to copyright.
Evaluation of wait-time-saving effectiveness of triage algorithms
Evaluation of wait-time-saving effectiveness of triage algorithms
Yee Lam Elim Thompson1Gary M Levine1Weijie Chen1Berkman Sahiner1Qin Li1Nicholas
Petrick1Jana G Delfino1Miguel A Lago1Qian Cao1Qin Li1*Frank W Samuelson1
1The U.S. Food and Drug Administration YeeLamElim.Thompson@fda.hhs.gov
Abstract
In the past decade, Artificial Intelligence (AI) algorithms
have made promising impacts to transform healthcare in all
aspects. One application is to triage patients’ radiological
medical images based on the algorithm’s binary outputs. Such
AI-based prioritization software is known as computer-aided
triage and notification (CADt). Their main benefit is to speed
up radiological review of images with time-sensitive findings.
However, as CADt devices become more common in clin-
ical workflows, there is still a lack of quantitative methods
to evaluate a device’s effectiveness in saving patients’ wait-
ing times. In this paper, we present a mathematical frame-
work based on queueing theory to calculate the average wait-
ing time per patient image before and after a CADt device
is used. We study four workflow models with multiple radi-
ologists (servers) and priority classes for a range of AI di-
agnostic performance, radiologist’s reading rates, and patient
image (customer) arrival rates. Due to model complexity, an
approximation method known as the Recursive Dimensional-
ity Reduction technique is applied. We define a performance
metric to measure the device’s time-saving effectiveness. A
software tool is developed to simulate clinical workflow of
image review/interpretation, to verify theoretical results, and
to provide confidence intervals of the performance metric we
defined. It is shown quantitatively that a triage device is more
effective in a busy, short-staffed setting, which is consistent
with our clinical intuition and simulation results. Although
this work is motivated by the need for evaluating CADt de-
vices, the framework we present in this paper can be applied
to any algorithm that prioritizes customers based on its binary
outputs.
Introduction
The fast-growing development of artificial intelligence (AI)
and machine learning (ML) technologies bring a potential
to transform healthcare in many ways. One emerging area is
the use of AI/ML as Software as a Medical Device (SaMD)
in radiological imaging to triage patient images with time-
sensitive findings for image interpretation (van Leeuwen
et al. 2022). These devices are known as computer-aided
triage and notification (CADt) devices, by which medical
*Dr. Qin Li has left the FDA and is currently the director of
Translational Medicine at Astrazeneca. Her contribution to this
work was made when she was at the FDA.
images labeled as positive by an AI algorithm are priori-
tized in the radiologist’s reading queue. The major benefit
of a CADt device is to increase the likelihood of timely di-
agnosis and treatment of severe and time-critical diseases
such as large vessel occlusion (LVO), intracranial hemor-
rhage (ICH), pneumothorax, etc. In 2018, the U.S. Food and
Drug Administration (FDA) granted marketing authoriza-
tion to the first CADt device for potential LVO stroke pa-
tients via the de novo pathway (The US Food and Drug Ad-
ministration 2018). Since then, multiple studies have shown
improvements in patient treatment and clinical outcomes
due to the use of CADt devices (Hassan et al. 2020; Yahav-
Dovrat et al. 2021; Barreira et al. 2018; Hassan et al. 2021).
Most of these analyses focus on the diagnostic performance
when evaluating these CADt devices, but a quantitative esti-
mate of time savings for truly diseased (signal-present) pa-
tient images in a clinical environment remain unclear. There-
fore, the goal of this work is to fill this gap by developing a
queueing-theory based tool to characterize the time-saving
effectiveness of a CADt device in a given clinical setting.
Figure 1 illustrates the radiologist workflows without and
with a CADt device being used. In the standard of care with-
out a CADt device, patient images are reviewed by a radiol-
ogist on a first-in, first-out (FIFO) basis. In the context of
queuing theory, our servers are radiologists, and our cus-
tomers are patient images. Occasionally, the radiologist may
be interrupted by an emergent case, for example, when a
physician requests an immediate review of a specific patient
image. To distinguish these emergent cases from those in the
reading queue, we call the images in the reading list “non-
emergent.” If a CADt device is included in the workflow, the
device only analyzes non-emergent patient images. Cases la-
beled as AI-positive are either flagged or moved up in a ra-
diologist’s reading list, giving them higher priority, and the
radiologist will review those cases before all AI-negative pa-
tient images. Just like the without-CADt scenario, the radi-
ologist may be interrupted by emergent cases, which always
have the highest priority over other images. Overall, with-
out a CADt, we have a queue with two priority classes, and
we have a queue with three priority classes in a with-CADt
scenario.
It is noted that, though applied to radiology clinics, the
mathematical frameworks presented here could be used to
evaluate discrimination algorithms in other queueing con-
1
arXiv:2303.07050v1 [stat.AP] 13 Mar 2023
Figure 1: Radiologist workflows without and with a CADt device. Top: the without-CADt scenario in which patient images are
reviewed in the order of their arrival. Bottom: the with-CADt workflow in which AI-positive patient images are reviewed first
before the AI-negative images. In both scenarios, the radiologist may be interrupted by emergent cases. All cartoon icons are
adopted from Microsoft PowerPoint application.
texts. For example, algorithms may attempt to identify cus-
tomers or jobs who may require less service time and place
them into a higher priority class, thereby reducing overall
wait time for customers on average.
Parameters
Before applying queueing theory, a few parameters are de-
fined to describe the clinical setting.
•fem is the fraction of emergent patient images with re-
spect to all patient images.
•λis the Poisson arrival rate of all patient images. Patient
images can be divided into subgroups, and each subgroup
ihas a Poisson arrival rate λi=piλ, where piis the
fraction of image subgroup iwith respect to all patient
images.
• The disease prevalence πis defined within the non-
emergent patient population, i.e.
π=Number of diseased, non-emergent cases
Number of non-emergent cases .(1)
• CADt diagnostic performance is defined by its sensitivity
(Se) and specificity (Sp), which are also defined within
the non-emergent patient images i.e.
Se =Number of AI-positive, diseased, non-emergent cases
Number of diseased, non-emergent cases ,
and
Sp =Number of AI-negative, non-diseased, non-emergent cases
Number of non-diseased, non-emergent cases .
•Nrad is the number of radiologists on-site. Typically, a
clinic has at least one radiologist at all times. For a larger
hospital, multiple radiologists may be available during
the day.
• The radiologist’s reading rates are denoted by µ’s. For
emergent (highest-priority) cases, the reading time Tem
is assumed to be exponentially distributed with an av-
erage reading rate µem = 1/T em. For a non-emergent
image, the average reading rate depends on the radi-
ologist’s diagnosis i.e. µDif diseased image or µND
if non-diseased image. Therefore, in the without-CADt
scenario, the reading time of the non-emergent (lower-
priority) cases follows a hyperexponential distribution
where the mean (1/µnonEm)is determined by the mean
reading rates of the two subgroups and the probability of
disease prevalence π.
1
µnonEm
=π
µD
+1−π
µND
.(2)
In the with-CADt scenario, the average reading rates for
AI-positive (middle-priority) and AI-negative (lowest-
priority) classes are denoted by µ+and µ−respectively.
The AI-positive group consists of true-positive (TP) and
false-positive (FP) patients, and the probability that an
AI-positive case is a TP is defined by the positive predic-
tive value (PPV). Hence,
1
µ+
=PPV
µD
+1−PPV
µND
.(3)
Similarly, the average AI-negative reading rate is given
2
by
1
µ−
=1−NPV
µD
+NPV
µND
,(4)
where NPV is the probability that an AI-negative case is
a true-negative (TN).
•ρis the traffic intensity defined as ρ=λ/µeff, where µeff
is effective reading rate considering all priority classes
and Nrad in the queueing system. ρranges from 0 with
no patient images arriving to 1 implying a very congested
hospital.
• With regard to the queueing discipline, when no CADt
device is used, patient images are read in the order of
their arrival time i.e. first-in first-out (FIFO). In the with-
CADt scenario, we consider a preemptive-resume prior-
ity scheduling: whether or not a CADt device is used,
whenever a higher-priority patient image enters the sys-
tem, the reading of a lower-priority patient image will
be interrupted and later resumed. Although in reality
some radiologists may prefer finishing up the current
lower-priority image when a CADt device flags a higher-
priority case (which would be a non-preemptive-resume
priority), many CADt devices are designed assuming a
radiologist reads the flagged cases immediately. There-
fore, a preemptive-resume priority is assumed in this
work.
To assess the time-saving effectiveness of a given CADt
device in a clinical setting defined by the above parameters,
we first define four radiologist workflow models in Section
. For each of the models, we provide the Markov chain ma-
trices to compute the mean waiting time for each priority
class in both with- and without-CADt scenarios. Section
discusses an in-house simulation software developed to ver-
ify theoretical results and to provide confidence intervals
around the theoretical mean time savings. Section defines
a metric that quantifies the time-saving effectiveness of a
CADt device, and Section discusses the results obtained
from theory and simulation.
Radiologist workflow models
We consider four radiologist workflow models:
• Model A: The baseline model (Nrad = 1,fem = 0, and
µD=µND )
• Model B: Model A but with emergent patient images
(Nrad = 1,fem >0, and µD=µND )
• Model C: Model B but with two radiologists (Nrad = 2,
fem >0, and µD=µND )
• Model D: Model B but with different reading rates for
diseased and non-diseased images (Nrad = 1,fem >0,
and µD6=µND )
For each model, two calculations are performed: one as-
sumes a without-CADt scenario, and the other assumes the
use of a CADt device. Each scenario has a set of states that
keeps track of the numbers of patient images in different
priority classes. The transition rates among states form a
stochastic Markov chain matrix, from which the matrix ge-
ometric method is applied to calculate the set of state proba-
bilities (Stewart 2009) . For models involving multiple radi-
ologists and priority classes, we apply the Recursive Dimen-
sionality Reduction (RDR) method proposed by (Harchol-
Balter et al. 2005) to facilitate the calculation. Little’s Law
(Stewart 2009) is then applied to calculate the mean waiting
time per patient image for each priority class involved.
Model A: Baseline model
We start with a simple model with the absence of emergent
patient images (fem = 0), one radiologist on-site (Nrad =
1), and identical reading rates for diseased and non-diseased
subgroups (µD=µND ).
Model A in without-CADt scenario First, we consider
the without-CADt scenario. Given that fem = 0, only one
priority class (the non-emergent subgroup) exists, and the
arrival rate λis the arrival rate of non-emergent patient im-
ages λnonEm. When µD=µN D and with only 1 radiologist
on-site, the effective reading rate for the non-emergent sub-
group is µnonEm =µD=µND. Hence, Model A turns into
a classic M/M/1/FIFO queueing model (Stewart 2009). Its
transition diagram is shown in Figure 2, from which the state
probability pnis given by
pnnonEm =ρn
nonEm(1 −ρnonEm ),(5)
where nnonEm denotes the number of non-emergent patient
image in the system. From the state probability pnnonEm , the
average waiting time per non-emergent patient image can be
calculated by the following steps.
1. Calculate the average number of non-emergent patient
images in the system, L, from the state probability
pnnonEm . That is, L=hpnnonEm i, where hi is the expecta-
tion operator.
2. Calculate the average response time per non-emergent
patient image, W, via Little’s Law i.e. W=L/λnonEm.
3. Calculate the average waiting time in the queue per non-
emergent patient, WqnonEm . Because Wis the sum of
WqnonEm and the mean radiologist’s reading time T=
1/µnonEm, we have WqnonEm =W−1/µnonEm.
In summary, the average waiting time per non-emergent pa-
tient image WqnonEm in a without-CADt scenario is given by
WqnonEm =hpnnonEm i/λnonEm −1/µnonEm.(6)
Model A in with-CADt scenario When a CADt-device is
used with no emergent patient images (fem = 0), two pri-
ority classes exist: an AI-positive, higher-priority class and
an AI-negative, lower-priority class. The arrival rates of AI-
positive and AI-negative classes depend on the CADt diag-
nostic performance.
λ+=πSe + (1 −π)(1 −Sp)λ, (7)
λ−=π(1 −Se) + (1 −π)Spλ. (8)
The state of a two-priority class system is defined by the
number of AI-positive cases n+and that of AI-negative n−.
As shown in Figure 3, the exact transition diagram is infinite
3
Figure 2: The Markov chain transition diagram for non-emergent patient images in Model A without a CADt device. Gray
bubbles represent the state nnonEm, the total number of non-emergent patient images in the system. Top orange arrows represent
the transition rate λto increase nnonEm one at a time, and bottom green arrows represent the transition rate µto decrease nnonEm
one at a time.
in both horizontal (n−) and vertical (n+) directions. With
an assumed preemptive-resume priority scheduling, this 2D-
infinity problem can be resolved using the Recursive Dimen-
sionality Reduction (RDR) method (Harchol-Balter et al.
2005), in which the tangled two-priority-class system is bro-
ken down into two independent calculations, one for each
priority class.
First, we focus on the AI-positive, higher-priority system.
Because of the preemptive-resume queueing discipline, the
AI-positive subgroup is not affected by the AI-negative im-
ages at all and is, by itself, a classic M/M/1/FIFO queue-
ing model. Therefore, to solve for the average waiting time
per AI-positive patient image, one can reuse Figure 2 and
replace nnonEm by n+. The state probability for AI-positive
patient images is modified based on Equation 5;
pn+=ρn+
+(1 −ρ+),(9)
where ρ+≡λ+/µ+is the traffic intensity for the AI-
positive subgroup only. Following the steps in Equation 6,
the average waiting time per AI-positive patient image Wq+
is given by
Wq+=hpn+i/λ+−1/µ+.(10)
For the calculation of the AI-negative, lower-priority
class, we cannot ignore the presence of AI-positive cases.
However, with only one radiologist, no AI-negative patient
image can exit the system when n+≥1. As noted by
(Harchol-Balter et al. 2005), there is no need to keep track
of every state beyond n+≥1. Hence, every column in Fig-
ure 3 can be truncated such that all states beyond n+≥1
are represented by (1+, n−). The RDR-truncated transition
diagram is shown in Figure 4.
Because of the truncation, the transition rate Bfrom
(1+, n−)to (0, n−)no longer represents a simple exponen-
tial transition time distribution. In fact, the shape of this tran-
sition time distribution is often unknown but can be approx-
imated to an Erlang-Coxian (EC) distribution. As shown in
Figure 5, a general EC distribution consists of exactly two
Coxian phases and NEC −2Erlang phases. For a given dis-
tribution of unknown shape, (Osogami and Harchol-Balter
2006) provided closed-form solutions to calculate the first
three moments of the unknown distribution and the six pa-
rameters in the EC distribution that best matches the first
three moments.
When applying the EC-approximation method to the
RDR-truncated transition diagram in Figure 4, only the two-
phase Coxian distribution is sufficient. No Erlang phases are
needed; hence, pEC ,NEC , and λYEC in Figure 5 are 1, 2, and
0 respectively. The non-exponential transition Bcan then be
explicitly expressed in terms of the approximated exponen-
tial transition rates t’s as shown in Figure 6, where
t1= (1 −pXEC )λX1EC ;t12 =pXEC λX1EC ;t2=λX2EC .
(11)
Figure 6 is a typical Markov chain transition diagram,
and its transition rate matrix MAcan be formed (see Sec-
tion in Electronic Companions). Using the matrix geomet-
ric method, an analysis method for quasi-birth–death pro-
cesses where the Markov chain matrix has a repetitive block
structure (Stewart 2009), the state probability pn−is com-
puted. Hence, the average waiting time per AI-negative, low-
priority patient image, Wq−, can be calculated;
Wq−=hpn−i/λ−−1/µ−.(12)
Model B: Model A with emergent patient images
Model B is similar to Model A but with the presence of
emergent patient images (fem >0). These emergent im-
ages are prioritized to the highest priority regardless of the
presence of CADt devices. Although the waiting time of the
emergent subgroup can be studied, this work only focuses on
the non-emergent, AI-positive, and AI-negative subgroups
which are impacted by the CADt device.
Model B in without-CADt scenario In the standard of
care without a CADt device, the presence of emergent class
results in a two-priority-class queueing system: emergent
and non-emergent classes. For the emergent subgroup, µem
denotes its radiologist’s reading rate, and its arrival rate is
given by
λem =femλ. (13)
The arrival rate for the non-emergent class is
λnonEm = (1 −fem )λ. (14)
Similar to Model A, because µD=µND and Nrad = 1,
the effective reading rate for the non-emergent subgroup is
µnonEm =µD=µND .
With only one radiologist on-site, the analysis of non-
emergent, lower-priority class is exactly the same as that of
the AI-negative class in Model A in the with-CADt scenario.
Figure 6 (and Equation 21 in Electronic Companions) can be
reused by replacing λ+with λem,λ−with λnonEm ,µ+with
µem, and µ−with µnonEm . After solving for the state prob-
ability pnnonEm , the average waiting time per non-emergent
patient image is given by Equation 6.
4
Figure 3: The exact transition diagram for Model A in a with-CADt scenario. Gray bubbles represent the state (n+, n−)defined
by the numbers of AI-positive patient images, n+, and AI-negative patient images, n−, in the system. Each row represents the
transition of increasing or decreasing n−, and each column represents the transition of n+. Note that AI-negative patient images
can leave the system only when n+= 0, and hence the µ−arrows only show up in the first row of transition diagram.
Figure 4: The RDR-truncated transition diagram for AI-negative, low-priority patient images in Model A in the with-CADt
scenario. For every column in Figure 3, all states beyond n+≥1are truncated as (1+, n−)with an approximated transition
rate B.
Model B in with-CADt scenario When a CADt is in-
cluded in the workflow, three priority classes exist: emer-
gent (highest priority), AI-positive (middle priority), and
AI-negative (lowest priority) classes. With the presence of
emergent patients, the arrival rates of AI-positive and AI-
negative classes are now
λ+=πSe + (1 −π)(1 −Sp)(1 −fem)λ, and (15)
λ−=π(1 −Se) + (1 −π)Sp(1 −fem)λ. (16)
Their reading rates are given by Equations 3 and 4. However,
because µD=µND , the reading rates for the AI-positive
and AI-negative subgroups are the same; µ+=µ−=µD=
µND . Similar to Model A in with-CADt scenario, we ap-
ply the RDR method and solve for the AI-positive and AI-
negative systems separately.
For the AI-positive subgroup, it is noted that an AI-
positive patient image can only be interrupted by emergent
patient images and will not be impacted by any AI-negative
patient images. Therefore, the emergent and AI-positive sub-
groups form a two-priority-class queueing system which
can be solved using the framework developed for the non-
emergent subgroup in the without-CADt scenario. Figure 6
(and Equation 21 in Electronic Companions) can be reused
by replacing λ+with λem,λ−with λ+,µ+with µem , and
µ−with µ+. The state probability pn+for the AI-positive
subgroup is calculated, from which the average waiting time
per AI-positive patient image is given by Equation 10.
The calculation for the AI-negative, lowest-priority sub-
group involves states (nem, n+, n−)defined by the number
of emergent, AI-positive, and AI-negative patient images
in the system. An AI-negative patient image can be inter-
rupted by either an emergent or an AI-positive patient image.
The arrival time of the interrupting case denotes the start of
a busy period, which is defined as the time period during
which a radiologist is too busy for AI-negative cases. While
the radiologist is reading the interrupting case, new emer-
gent and/or AI positive images may enter the system, which
5
Figure 5: An Erlang-Coxian (EC) distribution defined by six
parameters (pEC , λYEC , NEC , pXEC , λX1EC , λX2EC ). Each arrow
represents an exponential transition time distribution. Cal-
culations of these parameters depend on the normalized mo-
ments of the original distribution with an unknown shape
(Osogami and Harchol-Balter 2006).
further delays the review of the interrupted AI-negative case.
Once all the higher-priority images are reviewed, the radiol-
ogist then resumes the reading of the interrupted AI-negative
patient image, and the busy period ends.
Due to the different arrival and reading rates between the
emergent and AI-positive patient images, the dependence of
AI-negative busy period on the two subgroups are different.
As (Harchol-Balter et al. 2005) discussed, one must keep
track of the state at which the busy period starts and the state
at which the busy period ends. With only one radiologist,
Model B has only two distinct busy periods:
•B1: (0, 1+,n−)→(0, 0, n−)
•B2: (1+, 0, n−)→(0, 0, n−)
Here, B1and B2are the rates of the two busy periods and
are explicitly shown as two non-exponential transitions in
Figure 7.
Just like the AI-negative system in Model A, one must
first calculate the first three moments for each busy period
and approximate each distribution using a two-phase Coxian
distribution. With three priority classes and two busy peri-
ods, the approximation involves the inter-level passage times
from the AI-positive transition diagram, from which a tran-
sition probability matrix as well as the transition rate matrix
are determined (see MBin Section ). From transition rate
matrix, the state probability pn−can be solved via conven-
tional matrix geometric method. Once pn−is determined,
the average waiting time per AI-negative patient image Wq−
can be calculated via Equation 12.
Model C: Model B with two radiologists
Model C extends Model B by adding one extra radiologist
on-site Nrad = 2. The arrival rates for the emergent, non-
emergent, AI-positive, and AI-negative classes remain the
same (Equations 13 - 16). Because µD=µND, the read-
ing rates for the non-emergent, AI-positive, and AI-negative
subgroups are the same; µ+=µ−=µnonEm. Because of the
extra radiologist, the traffic intensity ρhas a factor of two;
ρ=λ/2µ. It should be noted that Model C has the same
settings as the example in (Harchol-Balter et al. 2005).
Model C in without-CADt scenario With no CADt de-
vices, the RDR-truncated transition diagram for the non-
emergent, lower-priority class is given by Figure 8. Given
two radiologists on-site, a non-emergent image can depart
the system only when nem <2, and hence the truncation of
states starts when nem = 2. Moreover, when nem = 0, both
radiologists are available for non-emergent patient images.
Thus, the first row has a leaving rate 2µnonEm, except the
transition from (0,1) to (0,0) when only one radiologist has
work to do. When nem = 1 (the second row), only one of the
two radiologists is available to review a non-emergent case,
resulting in a leaving rate of 1µnonEm. When nem ≥2, both
radiologists are busy handling emergent cases. Since no ra-
diologist is available for non-emergent images, their leaving
rate is 0, and no non-emergent images can leave the sys-
tem. To approximate the transition rate Bin Figure 8, the
same two-phase Coxian approximation described in Models
A and B is applied.
The transition rate matrix MCnoCADt for Figure 8 can be
found in Section . From MCnoCADt , the state probability
pnnonEm is determined, and the average waiting time per non-
emergent patient image is given by Equation 6.
Model C in with-CADt scenario In the with-CADt sce-
nario, the calculations for AI-positive (middle-priority) and
AI-negative (lowest-priority) subgroups are separated.
The queueing system for the AI-positive subgroup con-
sists of two priority classes: the emergent and AI-positive
classes, and the framework developed for the non-emergent
subgroup in the without-CADt scenario can be reused. By
replacing λnonEm with λ+and µnonEm with µ+in Figure 8
and Equation 27, the state probability for the AI-positive
subgroup pn+can be computed. And the average waiting
time per AI-positive patient image Wq+is given by Equa-
tion 10.
The approach to analyze the AI-negative, lowest-priority
subgroup is similar to the analysis of the AI-negative cases
in Model B. Recall that a state is defined as (nem, n+, n−)
and that a busy period is defined by the time duration
in which all the radiologists on-site are too busy for AI-
negative patient images. With two radiologists, a busy period
may start from one of the three situations: when there are
two emergent cases, when there are one emergent and one
AI-positive case, or when there are two AI-positive cases.
On the other hand, the busy period ends when one radiol-
ogist is handling either an emergent case or an AI-positive
case such that the other radiologist is available for the AI-
negative case. Therefore, instead of two busy periods in
Model B, adding one extra radiologist increases the total
number of busy periods to six:
•B1: (0, 2+,n−)→(0, 1+,n−)
•B2: (0, 2+,n−)→(1+, 0, n−)
•B3: (1+,1+,n−)→(0, 1+,n−)
•B4: (1+,1+,n−)→(1+, 0, n−)
•B5: (2+, 0, n−)→(0, 1+,n−)
•B6: (2+, 0, n−)→(1+, 0, n−)
Figure 9 shows the RDR-truncated transition diagram
for AI-negative subgroup. Note that states (0,2+, n−),
(1+,1+, n−), and (2+,0, n−)are duplicated because their
6
Figure 6: The RDR-truncated, EC-approximated transition diagram for AI-negative, low-priority patient images in Model A
assuming a with-CADt scenario, where the set of t’s correspond to the three Coxian rates (in red) in Figure 5.
Figure 7: The RDR-truncated transition diagram for AI-negative class in Model B in the with-CADt scenario. The state
(nem, n+, n−)is defined by the numbers of emergent, AI-positive, AI-negative patient images. Thick arrows represent the
non-exponential transition rates B1and B2of the two busy periods.
corresponding arrival rates also depends on the probabili-
ties that the busy period ends at a particular state i.e. either
(0,1+, n−)or (1+,0, n−). For example, p1denotes the con-
ditional probability that the busy period ends at (0,1+, n−)
given that it starts at (0,2+, n−).
Before solving for Figure 9, one must compute the con-
ditional probability and the first three moments of each
busy period, from which the transition rates can be approx-
imated. The calculation is discussed in Section , where the
AI-positive transition diagram for inter-level passage times
is presented, and the transition probability matrix is con-
structed.
Each busy period is approximated using the EC distribu-
tion (Figure 5). However, unlike Model B in which two-
phase Coxian is sufficient for all busy periods, B2and B5
in Model C require an extra Erlang phase, as shown in Fig-
ure 10. With an extra phase, two extra parameters t0and t01
are needed to approximate B2and B5.
t0= (1 −pEC )λYEC ;t01 =pEC λYEC ;
t1= (1 −pXEC )λX1EC ;t12 =pXEC λX1EC ;t2=λX2.
(17)
Once all six busy periods are approximated, the transition
rate matrix for the AI-negative, lowest-priority class can be
constructed from Figure 9. (See Section .) Like before, the
corresponding state probability pn−can be solved by the
matrix geometric method. And, the average waiting time per
AI-negative patient image Wq−can be calculated via Equa-
tion 12.
For Nrad ≥3, the same approach can be applied. How-
ever, as the number of busy periods increases, the transi-
tion rate matrix will grow in size drastically, especially when
more Erlang phases are required for the busy period approx-
imation.
Model D: Model B with different reading rates
Model D extends Model B by differentiating the radiolo-
gist’s reading rate between the diseased and non-diseased
subgroups (Nrad = 1,fem >0, and µD6=µND ). The ar-
rival rates for the emergent, non-emergent, AI-positive, and
AI-negative classes remain the same (Equations 13 - 16).
However, because µD6=µN D , the reading rates for non-
emergent, AI-positive, and AI-negative subgroups depend
on disease prevalence π, positive predictive value PPV, and
negative predictive value NPV (Equations 2-4).
7
Figure 8: The RDR-truncated transition diagram for non-emergent patient images in Model C in the without-CADt scenario.
The states (nem, nnonEm )are defined by the numbers of emergent and non-emergent cases in the system.
Model D in without-CADt scenario The without-CADt
scenario has two priority classes: emergent and non-
emergent patient images. Within the non-emergent class,
two groups of patient images (diseased and non-diseased)
are reviewed in a first-in-first-out (FIFO) basis. The cor-
responding transition diagram is shown in Figure 11. As
usual, the state keeps track of nem and nnonEm. In addition,
because of the different reading rates between the diseased
and non-diseased subgroups, the state must also keep track
of the disease status of the image that the radiologist is re-
viewing. Therefore, the state is defined as (nem, nnonEm, i),
where iis either D(i.e. the radiologist is working on a dis-
eased image) or ND (i.e. the radiologist is working on a
non-diseased image). Furthermore, one must pay attention
to how the busy period starts and ends. For example, if the
radiologist reading a diseased image is interrupted by the
arrival of an emergent image i.e. (0, n, D)→(1+, n, D),
the state must go back to (0, n, D)and not to (0, n, N D)
when the busy period is over. This property is guaranteed
by having two sets of truncated states: (1+, n)→Dthat can
only interact with (0, n, D)and (1+, n)→N D that can only
interact with (0, n, N D).
The corresponding transition rate matrix of Figure 11 is
given in Section . Note that, although Figure 11 has two busy
periods per column (one for “→D” and the other for “→
ND”), they both describe the same transition time when at
least one emergent image is in the system. Therefore, only
one unique set of t-parameters is calculated to approximate
both busy periods.
Model D in with-CADt scenario The calculation for AI-
positive (middle-priority) and AI-negative (lowest-priority)
subgroups are separated.
Because AI-positive patient images are not impacted by
AI-negative cases, the emergent and AI-positive subgroups
form a two-priority-class queueing system. The transition
rate matrix MDnoCADt from Figure 11 can be reused to ana-
lyze the queueing of AI-positive patient images. By replac-
ing λnonEm by λ+(Equation 15), µnonEm by µ+(Equation
3), and πby PPV, the state probability for the AI-positive
subgroup pn+is calculated via standard matrix geometric
method. The average waiting time per AI-positive patient
image Wq+is then given by Equation 10.
For the AI-negative, lowest-priority class, the full defini-
tion of state (nem, n+, i, n−, j ).iis either Dor ND, in-
dicating whether the radiologist is working on a diseased,
AI-positive case or a non-diseased, AI-positive case respec-
tively. The disease status of an AI-negative case that the ra-
diologist is reading is represented by the jwhich is either
Dor ND. Because we only have one radiologist, iand j
cannot appear simultaneously; the one radiologist can only
handle an AI-positive or AI-negative case but not both at the
same time.
Figure 12 shows the RDR-truncated transition diagram
for the AI-negative subgroup. There are three unique busy
periods with the corresponding transition rates Bi:
•B1: (1+, 0, n−)→(0, 0, n−,j)
•B2: (0, 1+,D,n−)→(0, 0, n−,j)
•B3: (0, 1+,ND,n−)→(0, 0, n−,j)
For each busy period, the truncated state is duplicated with
either “→D” or “→ND” such that the system can return
to the state with the correct disease status jwhen the busy
period is over.
Like before, for each unique busy period, its conditional
probability and first three moments are determined from
the transition probability matrix (see Section ). And, each
unique busy period has a set of t-parameters (Equation 11)
approximated from a two-phase Coxian distribution. With
the approximated busy period transitions, a transition rate
matrix can be constructed for the AI-negative subgroup from
Figure 12 (see Section ). The state probability pn−is then
solved, and the average waiting time per AI-negative patient
image Wq−can be calculated via Equation 12.
Simulation
To verify the analytical results from our theoretical queue-
ing approach, a Monte Carlo software was developed using
Python to simulate the flow of patient images in a clinic with
8
Figure 9: The RDR-truncated transition diagram for AI-negative patient images in Model C in the with-CADt scenario. The
state is defined as (nem, n+, n−), and states with nem +n+≥2are truncated. A total of 6 busy periods are identified. Each
busy period ihas a transition rate Bialong with a probability that it ends at a certain state given that it starts with a particular
state.
Figure 10: An Erlang-Coxian (EC) distribution with one
Erlang phase and two Coxian phases. See (Osogami and
Harchol-Balter 2006) for the closed form solutions to cal-
culate these parameters.
and without a CADt device. A workflow model is defined by
a set of input parameters {fem,π,ρ,µ,Nrad , Se, and Sp}.
During the simulation, a new patient image entry is ran-
domly generated with a timestamp that follows a Poisson
distribution at an overall arrival rate of λ, which is computed
from the user-inputs (traffic ρand radiologist’s reading rates
µ). Each patient image is randomly assigned with an emer-
gency status (emergent or non-emergent) based on the input
emergency fraction fem. If the patient image is emergent,
a reading time is randomly generated from an exponential
distribution with a reading rate of µem. If the patient image
is non-emergent, a disease status (diseased or non-diseased)
is randomly assigned based on the input disease prevalence
π. The reading time for this non-emergent patient image is
also randomly drawn from an exponential distribution with
a reading rate of either µDif it is diseased or µND if it is
non-diseased. Each non-emergent patient image is also as-
signed with an AI-call status (positive or negative) based on
its disease status and the input AI accuracy (Se and Sp). The
patient image is then simultaneously placed into two worlds:
one with a CADt device and one without.
In a without-CADt world, the incoming patient image is
either a higher-priority case (if it is emergent) or a lower-
priority case (if non-emergent). If the patient image is emer-
gent, the case is prioritized over all non-emergent patient im-
ages in the system and is placed at the end of the emergent-
only queue. Otherwise, the patient image is non-emergent
and is placed at the end of the current reading queue. In
time, when its turn comes, this patient image is read by one
of the radiologists and is then removed from the queue. Two
pieces of information are recorded for this simulated patient
image. One is its waiting time defined as the difference be-
tween the time when the image enters the queue and when
it leaves the queue. In addition, the number of emergent and
non-emergent patient images in the queue right before the
arrival of the new patient image are also recorded to study
the state probability distribution.
Alternatively, this very same patient image is placed in the
with-CADt world. This image has either a high priority (if
emergent), a middle priority (if AI-positive), or a low prior-
ity (if AI-negative). If the patient image is emergent, the case
is prioritized over all AI-positive and AI-negative patient im-
ages in the system and is placed at the end of the emergent-
only queue. If the patient image is AI-positive, the case is
9
Figure 11: The RDR-truncated transition diagram for non-emergent patient images in Model D in a without-CADt scenario.
The state is defined by the number of emergent patient images (either 0 or 1+), number of non-emergent patient images n, and
the disease status of case that the one radiologist is reviewing (either Dfor diseased or ND for non-diseased). The “→D” and
“→ND” in the truncated states keep track of the disease status of the interrupted, lower-priority case.
prioritized over all AI-negative images and is placed at the
end of the queue consisting of only emergent and AI-positive
patient images. Otherwise, the patient image is AI-negative
and is placed at the end of the current reading queue. The
reading time for this patient image in the with-CADt world
is identical to its reading time in the without-CADt world.
However, due to the re-ordering by the CADt device, its
waiting time in the with-CADt world may be different from
that in the without-CADt world. For every patient image, the
difference between the two waiting times in the two worlds
can be calculated to determine whether the use of the CADt
device results in a time-saving or time delay for this image.
In addition to its waiting time, the number of emergent, AI-
positive, and AI-negative patient images right before the ar-
rival of the new patient image are also recorded.
To simulate a big enough sample size, a full simulation
includes 200 simulations, each of which contains roughly
2,000 patients. From all simulations, the waiting times from
all diseased patient images are histogrammed from which
the mean value and the 95% confidence intervals are deter-
mined.
Time-saving effectiveness evaluation metric
We define a metric to quantitatively assess the time-saving
effectiveness of a given CADt device. Both theoretical and
simulation approaches output the mean waiting time per dis-
eased patient image WDin both with- and without-CADt
scenarios.
Without a CADt device, since the arrival process is ran-
dom, the average waiting time per non-emergent patient im-
age Wno-CADt
nonEm is the same as Wno-CADt
Di.e.
Wno-CADt
D=Wno-CADt
nonEm =WqnonEm .(18)
When a CADt device is included in the workflow, the aver-
age waiting time per diseased and non-diseased patient im-
ages are no longer the same because the diseased images
are more likely to be prioritized by the CADt. To calculate
WCADt
D, we first compute the average waiting time per AI-
positive (WCADt
+=Wq+) and per AI-negative (WCADt
−=
Wq−) patient image based on the mathematical frameworks
discussed in Section . By definition, the average waiting time
for the diseased subgroup WCADt
Dis
WCADt
D≡Total waiting time from all diseased patient images
Number of diseased patient images .
Note that the total waiting time from all diseased patients
is the sum of the total waiting time from the true-positive
(TP) subgroup and that from the false-negative (FN) sub-
group. Let NTP,NFN , and NDbe the number of TP patient
images, that of FN patient images, and that of diseased im-
ages. WCADt
Dcan be rewritten as
WCADt
D=WCADt
+×NTP +WCADt
−×NFN
ND
.
Because NTP/NDand NFN/NDare, by definition, Se and
1−Se, we have
WCADt
D=WCADt
+×Se +WCADt
−×(1 −Se).(19)
To quantify the time-saving effectiveness of a CADt de-
vice for diseased patient images, we define a time perfor-
mance metric δWDas the difference in mean waiting time
10
Figure 12: The RDR-truncated transition diagram for AI-negative subgroup in Model E in the with-CADt scenario. State is
defined as (nem, n+, i, n−, j ), where i(or j) indicate the disease status of the AI-positive (or AI-negative) case the radiologist
is reading.
per diseased image in the with-CADt and that in the without-
CADt scenario:
δWD≡WCADt
D−Wno-CADt
D.(20)
It should be noted that, besides the explicit dependence on
AI sensitivity in Equation 19, δWDalso depends on AI
specificity and all the clinical factors in the calculation of
WCADt
+,WCADt
−, and Wno-CADt
nonEm .
Based on its definition, a negative δWDimplies that, on
average, a diseased patient image is reviewed earlier when
the CADt device is included in the workflow than when it
is not. The more negative δWDis, the more time is saved,
and the more effective the CADt device is. If δWD= 0,
the presence of CADt device does not bring any benefit for
the diseased patient images. If δWDis positive, the review
of a diseased patient image is delayed on average, and the
CADt device brings more risks than benefits to the diseased
subgroup.
It should also be noted that the amount of time savings
for other subgroups can be defined similarly. For example,
for the non-diseased subgroup, the average waiting time per
non-diseased patient image in the without-CADt scenario,
Wno-CADt
ND , is
Wno-CADt
ND =Wno-CADt
nonEm .
When the CADt device is included in the workflow, the av-
erage waiting time per non-diseased patient image, WCADt
ND ,
becomes
WCADt
ND =WCADt
+×(1 −Sp) + WCADt
−×Sp,
where the first and second terms correspond to the false-
positive and true-negative subgroups respectively. δWND can
then be defined to describe the average wait-time difference
between the with-CADt and without-CADt scenarios for the
non-diseased subgroup.
Results and Discussion
Top plot in Figure 13 shows the time saved per diseased pa-
tient images as a function of traffic intensity ρfor one and
two radiologists on-site without any emergent patient im-
ages. Assuming a disease prevalence πof 10%, an AI sensi-
tivity of 95%, a specificity of 89%, an average image read-
ing time of 10 minutes for both diseased and non-diseased
subgroups, and one radiologist on-site, the time saving is
significantly improved from about 2 minutes in a quiet, low-
volume clinic (radiology traffic intensity of 0.3) to about an
hour in a relatively busy clinic (radiology traffic intensity of
0.8). At a traffic intensity ρof 0.8, the impact due to disease
prevalence is found to be small (see middle plot in Figure
13). Overall, the time-saving effectiveness of the device is
also found to be more evident with only one radiologist on-
site compared to two. Bottom plot in Figure 13 shows the
impact on the time-saving effectiveness due to the presence
of emergent patient images with the highest priority that
overrides any AI prioritization. The amount of time saved
per diseased image without any emergent patients (fem = 0)
is more-or-less the same as that with fem = 50%. This is
likely because the amount of delay caused by emergent pa-
tient images in a without-CADt scenario is similar to that in
the with-CADt scenario.
The effect of having different radiologist’s reading rates
for diseased and non-diseased subgroups are shown in Fig-
ure 14. The overall dependence on traffic intensity, disease
prevalence, and emergency fraction is similar to that in Fig-
ure 13. However, more time is saved for diseased patient
11
Figure 13: Amount of time saved per diseased patient im-
age as a function of (top) traffic intensity, (middle) dis-
ease prevalence, and (bottom) emergency fraction. Green
and blue lines represent scenarios with one and two radi-
ologists respectively. Dashed lines are theoretical δWD, and
the solid lines represent the mean time-saving effectiveness
from simulation. Shaded areas are the 95% confidence inter-
vals (C.I.s) from simulation. The average reading time for
an emergent image is set at 5 minutes, whereas the average
reading time for the diseased and non-diseased subgroups
are both 10 minutes.
images when µD< µND i.e. when a radiologist takes more
time on average to read a non-diseased image than a diseased
image.
For the purpose of evaluating a CADt device, we propose
a summary plot as shown in Figure 15 based on Model B,
describing both the diagnostic and time-saving effectiveness
of a CADt device. This plot is built upon a traditional re-
ceiver operating characteristic (ROC) analysis (Metz 1978),
in which the ROC curve characterizes the diagnostic perfor-
mance of the CADt device. For a given radiologist workflow
defined by a set of parameters, every point of False-Positive
Rate (FPR) and True-Positive Rate (TPR) in the ROC space
has an expected mean time savings per diseased patient im-
age, δWD, which is presented by the color map. The device
diagnostic performance is near ideal in the top left corner of
the ROC space, where δWDis the most negative.
To show the time-saving effectiveness of a CADt device,
δWDalong the ROC curve is plotted as a function of FPR
(top) and TPR (left). At (FPR, TPR) = (0, 0), δWDis 0
minute because all images are classified as AI-negative i.e.
Figure 14: Amount of time saved per diseased patient as a
function of (top) traffic intensity, (middle) disease preva-
lence, and (bottom) emergency fraction. Only one radiolo-
gist is on-site, and its average reading times for emergent and
diseased patient images are set at 5 minutes and 10 minutes
respectively. The average reading time for non-diseased pa-
tient images varies between 5 minutes (orange), 10 minutes
(green), and 15 minutes (red). Dashed lines are theoretical
δWD, and the solid lines represent the mean time-saving ef-
fectiveness from simulation. Shaded areas are the 95% con-
fidence intervals (C.I.s) from simulation. Note that the green
set of lines here is identical to that in Figure 13.
no images are prioritized. As both FPR and TPR increase
along the ROC curve, the amount of time savings |δWD|in-
creases since most AI-positive cases are truly diseased pa-
tient images. As FPR and TPR continue to increase, the
number of false-positive cases becomes dominant, reducing
the device’s time-saving effectiveness. When (FPR, TPR) =
(1, 1), δWDgoes back to 0 because all images are classi-
fied as AI-positive, and the system essentially has no priority
classes.
The mean time-savings for diseased patient images δWD
can be directly linked to potential patient outcome. For ex-
ample, if our disease of interest is large vessel occlusion
(LVO) stroke, δWDcolor axis on the right side of Figure
15 can be translated to three stroke patient outcome met-
rics. According to Table 12 in Supplementary Content (Sup-
plementary 2) of (Saver et al. 2016), for every 15 minutes
sooner that a patient is treated, 3.9% of stroke patients re-
sulted in less disability. This can be translated to the two
other common LVO stroke patient outcome metrics - the
number of patients needed to treat for benefit (NNTB) and
12
Figure 15: A summary ROC plot for evaluating both the diagnostic and time-saving effectiveness of a CADt device. The middle
rainbow plot is an ROC space with an ROC curve (dashed dark gray) of a theoretical CADt device. Color map represents
theoretical mean time savings (δWD) per diseased patient image, assuming a disease prevalence of 10% in a relatively busy
hospital (traffic intensity of 0.8) with only one radiologist and no emergent patient images. The radiologist’s average reading
times for diseased and non-diseased patient images are both set at 10 minutes. Positive δWD(blue region) means an overall time
delay for diseased patient images, and negative δWD(red region) means an overall time savings. The values printed on the color
map are the δWD’s at the corresponding points of false-positive and true-positive rates. The dot represents the pre-determined
AI operating point (Se = 95%,Sp = 88%). Top plot represents the theoretical δWD(dashed gray) along the ROC curve as a
function of false-positive rate, and left plot represents the same theoretical δWD(dashed gray) along the curve but as a function
of true-positive rate. The green solid line represents the mean time savings along the ROC curve obtained from simulation. The
darker and lighter shaded areas indicate the 68% and 95% ranges from simulation around the mean time savings. The black
dotted vertical and horizontal lines indicate that the theoretical mean time-saving for diseased patients is roughly 36 minutes at
the given operating point. The color axis is translated to stroke patient outcome metrics based on Table 12 in Supplementary
Content (Supplementary 2) of (Saver et al. 2016).
the number of minutes faster needed to treat (MNT). The re-
lationships between δWDand LVO stroke patient outcome
metrics are extrapolated linearly and shown in the three axes
on the right side of Figure 15. As a result, the optimal δWD
along the ROC curve is roughly -40 minutes, which corre-
sponds to approximately 11% increase in LVO stroke pa-
tients with less disability, more than 9 NNTB, and more than
1.4 MNT. Remember that these results depend on our as-
sumed reading rates and traffic intensity. In the future we ex-
pect to gather clinical data to make more accurate estimates
of reading rates, traffic intensity, and wait-time savings.
Based on our queueing approach, the time-saving effec-
tiveness of a CADt device depends largely on the clinical
settings. Our model suggests that CADt devices with a typ-
ical AI diagnostic performance (95% sensitivity and 89%
specificity) are most effective in a busy, short-staffed clinic.
All theoretical predictions agree with simulation results well
within the 95% confidence intervals. All software used in
making the theoretical calculations and the simulations in
this paper will be made available on the Github site for the
FDA’s Division of Imaging, Diagnostics, and Software Re-
liability, https://github.com/DIDSR/QuCAD.
In this work where only one disease is considered, the
CADt device is trained to identify the disease, and a pa-
tient image can either be diseased or non-diseased. Under
this consideration, when evaluating the time-saving effec-
tiveness of the CADt, δWDis used as the performance met-
ric because the CADt device is intended to benefit diseased
patients with time critical conditions. In the future, when we
expand our work to a reading queue that consists of patient
images with two or more diseases, a new performance ma-
tric will be defined to take into account other time-critical
diseases that the CADt does not look for.
13
Conclusion
We present a mathematical framework based on queueing
theory and the Recursive Dimensionality Reduction method
to quantify the time-saving effectiveness of an AI-based
medical device that prioritizes patient images based on its
binary classification outputs. Several models are developed
to theoretically predict the wait-time-saving effectiveness of
such a device as a function of various parameters, includ-
ing disease prevalence, patient arrival rate, radiologist read-
ing rate, number of radiologists on-site, AI sensitivity and
specificity, as well as the presence of emergent patient im-
ages with the highest priority that overrides any AI prioriti-
zation. The methodology proposed in this paper helps evalu-
ate the time-saving performance of a CADt or any prioritiza-
tion device. The models presented here could also be used to
evaluate discrimination algorithms in many other queueing
contexts, such as serving customers or computer job queue-
ing. In the near future, we plan on expanding our model to
clinical scenarios of multiple disease conditions, modalities,
and anatomies with several CADt devices being used simul-
taneously.
Acknowledgments
The authors would like to thank Dr. Mor Harchol-Balter
(harchol@cs.cmu.edu) and Dr. Takayuki Osogami (OS-
OGAMI@jp.ibm.com) for helping us understand their Re-
cursive Dimensionality Reduction (RDR) method for com-
plex queueing systems. In addition, the authors acknowledge
funding from the Critical Path Program of the Center for De-
vices and Radiological Health. The authors also acknowl-
edge funding by appointments to the Research Participation
Program at the Center for Devices and Radiological Health
administered by the Oak Ridge Institute for Science and Ed-
ucation through an interagency agreement between the U.S.
Department of Energy and the U.S. Food and Drug Admin-
istration (FDA).
References
Barreira, C. M.; Bouslama, M.; Haussen, D. C.; Grossberg,
J. A.; Baxter, B.; Devlin, T.; Frankel, M.; and Nogueira,
R. G. 2018. Abstract WP61: Automated large artery occlu-
sion detection IN stroke imaging - ALADIN study. Stroke,
49(Suppl 1).
Harchol-Balter, M.; Osogami, T.; Scheller-Wolf, A.; and
Wierman, A. 2005. Multi-server queueing systems with
multiple priority classes. Queueing Syst., 51(3-4): 331–360.
Hassan, A. E.; Ringheanu, V. M.; Preston, L.; and Tekle,
W. 2021. Abstract P248: CSC implementation of artifi-
cial intelligence software significantly improves door-in to
groin puncture time interval and recanalization rates. Stroke,
52(Suppl 1).
Hassan, A. E.; Ringheanu, V. M.; Rabah, R. R.; Preston, L.;
Tekle, W. G.; and Qureshi, A. I. 2020. Early experience
utilizing artificial intelligence shows significant reduction in
transfer times and length of stay in a hub and spoke model.
Interv. Neuroradiol., 26(5): 615–622.
Metz, C. E. 1978. Basic principles of ROC analysis. Semin.
Nucl. Med., 8(4): 283–298.
Osogami, T.; and Harchol-Balter, M. 2006. Closed form so-
lutions for mapping general distributions to quasi-minimal
PH distributions. Perform. Eval., 63(6): 524–552.
Saver, J. L.; Goyal, M.; van der Lugt, A.; Menon, B. K.; Ma-
joie, C. B. L. M.; Dippel, D. W.; Campbell, B. C.; Nogueira,
R. G.; Demchuk, A. M.; Tomasello, A.; Cardona, P.; Devlin,
T. G.; Frei, D. F.; du Mesnil de Rochemont, R.; Berkhemer,
O. A.; Jovin, T. G.; Siddiqui, A. H.; van Zwam, W. H.; Davis,
S. M.; Casta˜
no, C.; Sapkota, B. L.; Fransen, P. S.; Molina,
C.; van Oostenbrugge, R. J.; Chamorro, ´
A.; Lingsma, H.;
Silver, F. L.; Donnan, G. A.; Shuaib, A.; Brown, S.; Stouch,
B.; Mitchell, P. J.; Davalos, A.; Roos, Y. B. W. E. M.; Hill,
M. D.; and for the HERMES Collaborators. 2016. Time to
treatment with endovascular thrombectomy and outcomes
from ischemic stroke: A meta-analysis. JAMA, 316(12):
1279.
Stewart, W. J. 2009. Probability, Markov chains, queues,
and simulation. Princeton, NJ: Princeton University Press.
The US Food and Drug Administration. 2018. FDA permits
marketing of Clinical Decision Support Software for alerting
providers of a potential stroke in patients.
van Leeuwen, K. G.; de Rooij, M.; Schalekamp, S.; van Gin-
neken, B.; and Rutten, M. J. C. M. 2022. How does artificial
intelligence in radiology improve efficiency and health out-
comes? Pediatr. Radiol., 52(11): 2087–2093.
Yahav-Dovrat, A.; Saban, M.; Merhav, G.; Lankri, I.;
Abergel, E.; Eran, A.; Tanne, D.; Nogueira, R. G.;
and Sivan-Hoffmann, R. 2021. Evaluation of artificial
intelligence-powered identification of large-vessel occlu-
sions in a comprehensive stroke center. AJNR Am. J. Neuro-
radiol., 42(2): 247–254.
Markov Chain Matrices
This appendix section provides the matrices involved for
each of the four radiologist workflow models discussed in
Section .
Model A in with-CADt scenario
Markov chain transition rate matrix MAis built upon Figure
6.
MA=
B00 B01
B10 A1A2
A0A1A2
A0A1
...
......
,(21)
14
where
A0= µ−0 0
0 0 0
0 0 0!, A1= ∗λ+0
t1∗t12
t20∗!,
A2= λ−0 0
0λ−0
0 0 λ−!,
B01 = λ−0 0
0λ−0
0 0 λ−!, B00 = ∗λ+0
t1∗t12
t20∗!,
B10 = µ−0 0
0 0 0
0 0 0!.(22)
MAhad a tri-diagonal block structure defined by sub-
matrices As and Bs, in which ∗’s are the negative of the
sum of all elements in the corresponding row. B00,B01 , and
B10 are block matrices representing the boundary condition
at the state of n−= 0; states with n−<0are forbidden
because the reading queue cannot have a negative number
for AI-negative patient images. A0,A1, and A2are repeti-
tive block structures that iterate along the diagonal axis of
the matrix.
Model B in with-CADt scenario
This scenario has two busy periods (B1and B2). For each
busy period, we first calculate its first three moments of
the inter-level passage times using Figure 16. The states at
which the two AI-negative busy periods start and end are
highlighted. For instance, B1is the time period starting from
(0,1) in red and ending at (0,0) in blue, regardless of any in-
termediate states that the system may go through. The steps
involved to calculate the first three moments are documented
in Appendix A of (Harchol-Balter et al. 2005).
Based on Figure 16, the transition probability matrix PB
is given below.
PB=
L1F1
B2L2F2
B3L3F3
B4L4
...
......
,(23)
where
B`=2 =
µ+
λem+λ++µ+
t1
λ++t1+t12
t2
λ++t2
,
B`≥3=
µ+
λem+λ++µ+0 0
t1
λ++t1+t12 0 0
t2
λ++t20 0
,
L`=1 = (0),L`≥2=
0 0 0
0 0 t12
λ++t1+t12
0 0 0
,
F`=1 =λ+
λ++λem
λem
λ++λem 0,
F`≥2=
λ+
λ++λem+µ+
λem
λ++λem+µ+0
0λ+
λ++t1+t12 0
0 0 λ+
λ++t2
.(24)
Here, the t-parameters are the approximated exponential
rates from the transition Bin Figure 16. With F,L, and B,
(Harchol-Balter et al. 2005) provide the framework to obtain
the Gmatrix, which contains the probabilities of the busy
periods involved, and the Zrmatrices, which have the r-th
moments of the busy periods. For the AI-negative priority
class in Model B, the Zrmatrix has a dimension of 3×1,
where the first and second elements are the r-th moments of
B1and B2respectively. For each of the two busy periods,
a two-phase Coxian distribution can be used to approximate
the distribution shape using Equation 11.
Let t(1)
1,t(1)
12 , and t(1)
2be the approximated rates for B1,
and t(2)
1,t(2)
12 , and t(2)
2be the approximated rates for B2. The
transition rate matrix MBfor Figure 7 is given below.
MB=
B00 B01
B10 A1A2
A0A1
...
A0
...
...
,(25)
15
Figure 16: The transition diagram to calculate inter-level passage times within the AI-positive (middle priority) in Model B in
a with-CADt scenario. The state is defined as (nem, n+), and each column keeps track of `=n++Min(Nrad , nem) + 1. Red
and blue boxes represent the states at which the busy periods for the AI-negative patient images start and end respectively.
where
A0=
µ−0000
0 0000
0 0000
0 0000
,
A1=
∗λ+0λem 0
t(1)
1∗t(1)
12 0 0
t(1)
20∗0 0
t(2)
10 0 ∗t(2)
12
t(2)
20 0 0 ∗
,
A2=
λ−0 0 0 0
0λ−0 0 0
0 0 λ−0 0
0 0 0 λ−0
0 0 0 0 λ−
,
B00 =
∗λ+0λem 0
t(1)
1∗t(1)
12 0 0
t(1)
20∗0 0
t(2)
10 0 ∗t(2)
12
t(2)
20 0 0 ∗
,
B01 =
λ−0 0 0 0
0λ−0 0 0
0 0 λ−0 0
0 0 0 λ−0
0 0 0 0 λ−
,
B10 =
µ−0000
0 0000
0 0000
0 0000
.(26)
The sub-matrices in MBare very similar to that in MA
(Equations 22 and ??). The one difference is that these sub-
matrices are now 5×5instead of 3×3due to the extra row
of truncated states in Figure 7 compared to Figure 4.
Model C in without-CADt scenario
The transition rate matrix MCnoCADt is built upon Figure 8.
MCnoCADt =
B00 B01
B10 A1A2
A0A1A2
A0A1
...
......
,(27)
where
A0=
2µnonEm 0 0 0
0µnonEm 0 0
0 0 0 0
0 0 0 0
,
A1=
∗λem 0 0
µem ∗λem 0
0t1∗t12
0t20∗
,
A2=
λnonEm 0 0 0
0λnonEm 0 0
0 0 λnonEm 0
0 0 0 λnonEm
,
16
B00 =
∗λem 0 0 λnonEm 0 0 0
µ+∗λem 0 0 λnonEm 0 0
0t1∗t12 0 0 λnonEm 0
0t20∗0 0 0 λnonEm
µnonEm 0 0 0 ∗λem 0 0
0µnonEm 0 0 µ∗λem 0
0 0 0 0 0 t1∗t12
0 0 0 0 0 t20∗
,
B01 =
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
λnonEm 0 0 0
0λnonEm 0 0
0 0 λnonEm 0
0 0 0 λnonEm
,
B10 =
00002µnonEm 0 0 0
0000 0 µnonEm 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
.(28)
Model C in with-CADt scenario
This scenario has six busy periods (B1to B6). For each busy
period, we first calculate its conditional probability and the
first three moments of the inter-level passage times using
Figure 17. And the corresponding transition probability ma-
trix PCis given below.
PC=
L1F1
B2L2F2
B3L3F3
B4L4
...
......
,(29)
where
B`=2 = µ+
λem+λ++µ+
µem
λem+λ++µem !,
B`=3 =
2µ+
λem+λ++2µ+0
µem
λem+λ++µ++µem
µ+
λem+λ++µ++µem
0t1
λ++t1+t12
0t2
λ++t2
B`≥4=
2µ+
λem+λ++2µ+0 0 0
µem
λem+λ++µ++µem
µ+
λem+λ++µ++µem 0 0
0t1
λ++t1+t12 0 0
0t2
λ++t20 0
,
L`=1 = (0),L`=2 =0 0
0 0,
L`≥3=
0 0 0 0
0 0 0 0
000 t12
λ++t1+t12
0 0 0 0
,
F`=1 =λ+
λem+λ+
λem
λem+λ+,
F`=2 = λ+
λem+λ++µ+
λem
λem+λ++µ+0 0
0λ+
λem+λ++µem
λem
λem+λ++µem 0!,
F`≥3=
λ+
λem+λ++2µ+
λem
λem+λ++2µ+0 0
0λ+
λem+λ++µ++µem
λem
λem+λ++µ++µem 0
0 0 λ+
λ++t1+t12 0
0 0 0 λ+
λ++t2
.
(30)
With PC, the conditional probabilities and first three mo-
ments of inter-level passage times for all six busy periods are
computed according to Appendix A of (Harchol-Balter et al.
2005). For most busy periods, three t-parameters are suffi-
cient for the approximation. However, for B2and B5, due
to the two extra Erlang phases, two additional parameters t0
and t01 are required.
Let t(i)
jdenote the tj-parameter for a busy period Bi. The
transition rate matrix MCCADt for the AI-negative, lowest-
priority class from Figure 9 is given by
MCCADt =
B00 B01
B10 A1A2
A0A1
...
A0
...
...
.(31)
All Asub-matrices are 17 ×17. Here, 014 denotes a 14 ×14
zero matrix, and I17 is a 17 ×17 identity matrix.
A0=
2µ−
µ−
µ−
014
,
A2=λ−I17,
17
Figure 17: The transition diagram to calculate inter-level passage times within the AI-positive (middle-priority) class in Model
C in a with-CADt scenario.
A1=
∗λ+λem
µ+∗p1λ+p2λ+p3λem p4λem
µem ∗p3λ+p4λ+p5λem p6λem
t(1) T(1)
4
t(2) T(2)
5
t(3) T(3)
6
t(4) T(4)
7
t(5) T(5)
8
t(6) T(6)
9
,
(32)
where, for i= 1,3,4,6,
pi= (pi0),t(i)=
t(i)
1
t(i)
2
,T(i)
k= ∗t(i)
12
∗!.
For i= 2,5, because of the extra Erlang phase, the sub-
matrices have an extra row and/or column.
pi= (pi0 0),t(i)=
t(i)
0
t(i)
1
t(i)
2
,T(i)
k=
∗t(i)
01
∗t(i)
12
∗
.
Similarly, the boundary Bsub-matrices are given below.
B01 = 017
λ−I17!, B10 =
017
2µ+
µ+
µ+
014
,
B00 =
A1λ−I17
µ+
µ+
µ+
014
A1
.(33)
Model D in without-CADt scenario
The transition rate matrix MDnoCADt for Figure 11 is given
below.
MDnoCADt =
B00 B01
B10 A1A2
A0A1A2
A0A1
...
......
,(34)
where
A0=
πµD(1 −π)µD0000
πµN D (1 −π)µN D 0000
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
,
A1=
∗0λem 0 0 0
0∗0 0 λem 0
t10∗t12 0 0
t20 0 ∗0 0
0t10 0 ∗t12
0t20 0 0 ∗
,
A2=λnonEmI6,
18
B00 = ∗λem 0
t1∗t12
t20∗!, B10 =
µD0 0
µND 0 0
0 0 0
0 0 0
0 0 0
0 0 0
,
B01 = πλnE (1 −π)λnE 0 0 0 0
0 0 πλnE 0 (1 −π)λnE 0
000πλnE 0 (1 −π)λnE !,
(35)
where λnE refers to the arrival rate of non-emergent sub-
group. Note that both MDnoCADt and MA(Equation 21) de-
scribe a queueing system with two priority classes and one
radiologist. However, because µD6=µN D , the size of A
sub-matrices grow from 3×3to 6×6due to the extra i
in the state definition and the extra set of truncated states to
keep track of disease status of the interrupted case.
Model D in with-CADt scenario
This scenario has three busy periods (B1to B3). For each
busy period, we calculate its conditional probability and the
first three moments of the inter-level passage times using
Figure 18 and the corresponding transition probability ma-
trix PD.
PE=
L1F1
B2L2F2
B3L3F3
B4L4
...
......
,(36)
where
B`=2 =
µD
λem+λ++µD
µND
λem+λ++µN D
t1
λ++t1+t12
t2
λ++t2
,
B`=3 =
P P V µD
λem+λ++µD
(1−PPV)µD
λem+λ++µD0 0
P P V µN D
λem+λ++µN D
(1−PPV)µN D
λem+λ++µN D 0 0
t1
λ++t1+t12 0 0 0
t2
λ++t20 0 0
0t1
λ++t1+t12 0 0
0t2
λ++t20 0
B`≥4=
P P V µD
λem+λ++µD
(1−PPV)µD
λem+λ++µD0000
P P V µN D
λem+λ++µN D
(1−PPV)µN D
λem+λ++µN D 0000
t1
λ++t1+t12 0 0000
t2
λ++t20 0000
0t1
λ++t1+t12 0000
0t2
λ++t20000
,
L`=1 = (0),L`=2 =
0000
0000
0000
0000
,
L`≥3=
0 0 0 0 0 0
0 0 0 0 0 0
000 t12
λ++t1+t12 0 0
0 0 0 0 0 0
0 0 0 0 0 t12
λ++t1+t12
0 0 0 0 0 0
,
F`=1 =P P V λ+
λem+λ+
(1−PPV)λ+
λem+λ+
λem
λem+λ+0,
F`=2 =
λ+
λem+λ++µD0λem
λem+λ++µD0 0 0
0λ+
λem+λ++µN D 0 0 λem
λem+λ++µN D 0
0 0 P P V λ+
λ++t1+t12 0(1−PPV)λ+
λ++t1+t12 0
000P P V λ+
λ++t20(1−PPV)λ+
λ++t2
,
F`≥3=
λ+
λem+λ++µD0λem
λem+λ++µD0 0 0
0λ+
λem+λ++µN D 0 0 λem
λem+λ++µN D 0
0 0 λ+
λ++t1+t12 0 0 0
000λ+
λ++t20 0
0 0 0 0 λ+
λ++t1+t12 0
0 0 0 0 0 λ+
λ++t2
.(37)
19
Figure 18: The transition diagram to calculate inter-level passage time within the AI-positive (middle-priority) class in Model
D in a with-CADt scenario.
Each of the three busy periods has a set of t-parameters
(Equation 11) approximated from a two-phase Coxian dis-
tribution.
Let t(i)
jdenote the tjparameter for the busy period Bi.
The transition rate matrix MDCADt for the AI-negative sub-
group (Figure 12) is given by
MDCADt =
B00 B01
B10 A1A2
A0A1
...
A0
...
...
(38)
The 14 ×14 Asub-matrices are defined below, where 012
denotes a 12 ×12 zero matrix, and I14 is a 14 ×14 identity
matrix.
A0= (1 −N P V )µDN P V µD
(1 −N P V )µN D N P V µN D
012!,
A2=λ−I14,
A1=
∗pλem pP P V λ+p(1 −PPV )λ+
∗pλem pP P V λ+p(1 −PPV )λ+
t(1) T(1)
3
t(1) T(1)
4
t(2) T(2)
5
t(2) T(2)
6
t(3) T(3)
7
t(3) T(3)
8
,(39)
where, for a busy period Bi,
p= (1 0),t(i)=
t(i)
1
t(i)
2
,T(i)
k= ∗t(i)
12
0∗!.(40)
20
The boundary Bmatrices, on the other hand, are
B00 =
−σ1pλem pP P V λ+p(1 −PPV )λ+
t(1) T(1)
2
t(2) T(2)
3
t(3) T(3)
4
,
B01 =
(1 −N P V )λ−N P V λ−
Q
Q
Q
,
B10 = µD
µND
012×6!,(41)
where
Q= (1 −N P V )λ−0N P V λ−0
0 (1 −N P V )λ−0N P V λ−!.
(42)
21
